Bad mIoU when using many GPUs

[xpngzhng]

I use the default deeplabv3plus config to train, and only modify the number of GPUs used. I noticed that the mIoU in the validation set drops significantly when the number of GPUs exceeds 4, as follows:

1 gpu: 0.7729
2 gpus: 0.7750
4 gpus: 0.7478
8 gpus: 0.5373

I guess it is caused by the batch normalization. Maybe sync BN will make a difference.
Things are quite different in object detection, e.g. mmdetection, where basic BN is used. The performance does not vary too much when I change the number of GPUs.

[DarthThomas]

Hi，

Yes, it is the problem with BN. Sync BN would reduce the difference between the different # of GPUs.

[xpngzhng]

yes, sync bn dramatically improves the mIoU metric