multi-node multi-cpu training do not converge

[waynezhang2018]

hi, guys

sorry for bothering!

I have successfully run the example, and also convert my own code from single node DataParallel to distributedDataParallel mode, the train runs without any error, but the loss is not descending and the accuracy looks wrong (finally to be 0)

the surprising point is, if I run the code with single GPU with batch size 128 and LR=0.001, it can work, everything is fine and converging, but if I take mpirun for 2 node X 8 opus =16gpus, loss is not descending.

what should I do?

I have changed the LR to be 10 or 1/10 times of the original LR, no help.

[waynezhang2018]

btw, I am using pytorch 0.4.0 and horovod 14

[alsrgv]

@waynezhang2018, could you review this example and check that you have corresponding code in your model for every part that says # ... Horovod ...?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.