BN after Activation, Have you tried it?

[ademyanchuk]

Hello, Ross,

Thanks for the great work you are doing! As you have such an impressive experience with training so many models with so many more hyperparameters, I am curious, have you ever tried to change only one thing (for models there it is appropriate, e.g. resnet), to put BatchNorm layer after activation. If yes, what was your results?

Thank you in advance and all the best :)

Alexey

[rwightman]

@ademyanchuk ResNetV2 nets are 'pre-act' and so are NfNets (but they don't have normalization layers). I've trained NFNets well, haven't really tried training the ResNetV2 but the weights in that file are pretty interesting as they were trained by google researchers investigating transfer learning and in21k pretraining.

https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/resnetv2.py
https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/nfnet.py

[ademyanchuk]

Thank you :)