DataParallel copies the model onto GPUs sequentially

[vadimkantorov]

I have 8 GPUs and can see it graphically with watch -n 0.1 nvidia-smi. It could save some time by doing the parallel async copy?

Same seems for distributing the batch size, but less sure

P.S. I know that DP is not recommended in favor of DDP, but for legacy code and simplicity reasons (also for easier recovery from OOMs, exceptions and easier logging), DP still remains important

cc @VitalyFedyunin @ngimel

[ezyang]

This sounds pretty reasonable

[vadimkantorov]

Maybe even scatter_all somehow could be used (e.g. we don't have bandwidth to copy to all devices at once, once the first device has received the copy, it could also transfer the model copy to other devices, halving the copy time)

[ngimel]

Are you talking about initial parameter copy? Because during training broadcast/reduce should not copy data to the cpu.

[vadimkantorov]

Yes, initial parameter copy. But the model is already on the first GPU, since I'm doing:

model = Model().cuda()
model = nn.DataParallel(model)

So it should already be amenable to inter-GPU copies (if it's enabled)

[ngimel]

We will accept a PR implementing that.