batchnorm momentum convention

[Peter554]

Hi,

Is the batchnorm momentum convention (default=0.1) correct as in other libraries e.g. Tensorflow it seems to usually be 0.9 or 0.99 by default? Or maybe we are just using a different convention?

[jaesuny]

Note on http://pytorch.org/docs/master/nn.html#torch.nn.BatchNorm2d
says you're right.

[dashesy]

Does this mean 0.1 in PyTorch's BatchNorm2d is same as 1 - 0.1 = 0.9 in TF/caffe? So if a TF model for example uses 0.2 I should use 0.8 here in PyTorch?

[soumith]

@dashesy yes

[daeyun]

Isn't 10% momentum as a default value really big?

Tensorflow's default is 0.99 which would be equivalent to 0.01 in pytorch, or 1% momentum.

[jxcodetw]

So is 0.1 enough for most cases?