All benefits of using a larger batch size assume the training throughput increases?

[SimLif]

• All benefits of using a larger batch size assume the training throughput increases. If it doesn't, fix the bottleneck or use the smaller batch size.
• Gradient accumulation simulates a larger batch size than the hardware can support and therefore does not provide any throughput benefits. It should generally be avoided in applied work.

Is a more stable gradient descent guaranteed by adding batch size?
In which scenarios should the gradient accumulation method be used?

[georgedahl]

What do you mean by "more stable gradient descent"?

I would use gradient accumulation in two scenarios:

1. When I need to exactly reproduce a result that uses a specific batch size that is larger than what I can fit in memory. This will happen when doing certain comparisons in research projects or in debugging. However, when trying to get the best results in an application, I will restrict myself to batch sizes that fit in memory and not use any gradient accumulation.
2. When I'm using an optimizer where it makes sense to amortize update-specific costs somehow. This is rare, but this might happen when using the Shampoo optimizer in some cases. It generally will not happen when using plain SGD with momentum or Adam.