When doing the distributed training, the bandwidth becomes the bottleneck.

[GoodJoey]

is there any experience/magic solutions when using mxnet to do distribute training?
say, is there any method, force the worker communicates with the parameter server on the same node to save the bandwidth? or something like ring-allreduce(horovod)?

[huyangc]

You could try the gradient compression first. It is more easier.

[sandeep-krishnamurthy]

@rahul003

[rahul003]

What kind of a network are you training?

[GoodJoey]

such as ResNet-101.
when you got a lot of data to process, you need to do data parallel using distributed training to save time.