May I ask you a question about precise-bn ?

[CoinCheung]

Hi,

Thanks for releasing this helpful codebase. After reading the part of precise-bn, a question comes to me:
Do I need to enlarge the batch size when I do precise bn?
I noticed that the performance of bn becomes bad when the batch size is small. People say that the bad performance is due to the noisy estimation of running mean/var of the bn layers. When batch size is small, there will be more noise in each batch's mean/var which brings bad estimation for the running mean/var. So in order to cope with the problem of small batch, do I need to enlarge the batch size when I use precise bn, or is there other explanations of the bn performance drop?

[ppwwyyxx]

Very small batch size is in general not good for BatchNorm, regardless of whether percise-BN is used or not.

[CoinCheung]

Do I have a chance to achieve a good result if I use small batch size to train the model and then use large batch size to do precise bn ?

[haooooooqi]

That probably would not work. Precise BN is computing the “precise” running mean and std and it depends on the current state of weights.

[haooooooqi]

A small BN would leads to a non-optimal set of weights.

[BluebirdStory]

Actually I think that you can never get a precise estimation of running_mean and running_var through precise BN unless you use all data in a single forward computation, do you guys understand what I am talking about?

[ppwwyyxx]

It is precise enough to give accurate final results and computing the "perfectly precise" stats do not improve results further. This is discussed in the end of Sec 3.2 as well as Appendix A3 of https://arxiv.org/pdf/2105.07576.pdf.