Does gradient accumulation exactly emulate a larger batch? #22
Answered
by
martex-dev
martex-dev
asked this question in
Q&A
|
Does gradient accumulation exactly emulate a larger batch? Notes and answer below. |
Answered by
martex-dev
Aug 25, 2026
Replies: 1 comment
|
For the gradient itself, yes. For normalisation statistics, no. Batch normalisation computes its mean and variance from the microbatch, so a model using batch-norm does not get a result equivalent to the larger batch. Layer normalisation is computed per sample and is unaffected, which is one of the practical reasons transformers standardised on it. |
0 replies
Answer selected by
martex-dev
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
For the gradient itself, yes. For normalisation statistics, no. Batch normalisation computes its mean and variance from the microbatch, so a model using batch-norm does not get a result equivalent to the larger batch. Layer normalisation is computed per sample and is unaffected, which is one of the practical reasons transformers standardised on it.