How to adjust learning rate when the number of workers incresing?

[Lihengwannafly]

For example, I set original learning rate 0.001 when standalone mode, and 0.001 / sqrt(10) performs well when 10 workers running. But when 20 workers running, 0.001 / sqrt(20) performs very bad. So, is there any suggestion to adjust when the number of workers incresing?

[liutongxuan]

There's list of topics about how to adjust the learning rate when increase total batch size in both sync mode or async mode.

This is an invalid, closed.