This is quick evaluation of BatchNorm layer (https://github.com/BVLC/caffe/pull/3229) performance on ImageNet-2012.
Other on-going evaluations:
- [architectures] (https://github.com/ducha-aiki/caffenet-benchmark/blob/master/Architectures.md)
- [augmentation] (https://github.com/ducha-aiki/caffenet-benchmark/blob/master/Augmentation.md)
The architecture is similar to CaffeNet, but has differences:
- Images are resized to small side = 128 for speed reasons.
- fc6 and fc7 layers have 2048 neurons instead of 4096.
- Networks are initialized with LSUV-init
Because LRN layers add nothing to accuracy, they were removed for speed reasons in further experiments.
BN -- before or after ReLU?
|Before||0.474||2.35||As in paper|
|Before + scale&bias layer||0.478||2.33||As in paper|
|After + scale&bias layer||0.493||2.24|
So in all next experiments, BN is put after non-linearity
BN and activations
BN and dropout
ReLU non-linearity, fc6 and fc7 layer only
|Dropout = 0.5||0.499||2.21|
|Dropout = 0.2||0.527||2.09|
|Dropout = 0||0.513||2.19|
|Caffenet BN Before + scale&bias layer LSUV||0.478||2.33|
|Caffenet BN Before + scale&bias layer Ortho||0.482||2.31|
|Caffenet BN After LSUV||0.499||2.21|
|Caffenet BN After Ortho||0.500||2.20|
|GoogLeNet BN Before + scale&bias layer LSUV||0.603||1.68|
|GoogLeNet BN Before + scale&bias layer Ortho||0.607||1.67|
|GoogLeNet BN After LSUV||0.596||1.70|
|GoogLeNet BN After Ortho||0.584||1.77|
|[GoogLeNet128_BN_lim0606][https://github.com/lim0606/caffe-googlenet-bn]||0.645||1.54||BN before ReLU + scale bias, linear LR, batch_size = 128, base_lr = 0.005, 640K iter, LSUV init. 5x5 replaced with 3x3 + 3x3. 3x3 replaced with 3x1+1x3|
BatchNorm evaluation ReLU
Different activations plus BN
As one can see, BN makes difference between ReLU, ELU and PReLU negligable. It may confirm that main source of VLReLU and ELU advantages is that their output is closer to mean=0, var=1, than standard ReLU.
Batch Normalization and Dropout
Do we need EltwiseAffine layer?
P.S. Logs are merged from lots of "save-resume", because were trained at nights, so plot "Anything vs. seconds" will give weird results.