Double gradient reduction in examples?

[TimZaman]

First, I'm confused why the example models implement their own DistributedDataParallel module. Why not use the torch one?

DistributedDataParallel is used in the examples:
https://github.com/kubeflow/pytorch-operator/blob/master/examples/ddp/mnist/cpu/mnist_ddp_cpu.py#L154

The DistributedDataParallel documentation states:

During the backwards pass, gradients from each node are averaged.

However, a DIY average_gradients function is used in the same example as well: https://github.com/kubeflow/pytorch-operator/blob/master/examples/ddp/mnist/cpu/mnist_ddp_cpu.py#L168

Double trouble?

[johnugeorge]

/assign @andreyvelich

[andreyvelich]

Thank you for your issue.
Yes, we can use DistributedDataParallel from pytorch library.
Will change it.