Gradient Accumulation

[ademyanchuk]

Hi @rwightman. Thank you for the great work.

While skimming the train.py script I didn't see any gradient accumulation paths. Have you use it in your practice.
If so, does it help or not to get a better results lets say using only one GPU but bigger effective batch size? What are your thoughts about why it does or doesn't help?

Thanks in advance!

Cheers,
Alexey

[rwightman]

@ademyanchuk I don't use it because these models all use BatchNorm by default. Gradient accumulation isn't a clear win with BatchNorm as it does not improve the effective batch size for the BN running stats calc and that can cause instability or poor results with small batches. One could use GroupNorm and several models do support switching the norm layer quite easily, but not something I've experimented with.

[ademyanchuk]

Thank you for the thorough explanation. I was trying accumulation a couple of times for Kaggle and didn't find it improving results as well.