Investigate and update DDP tutorials

[rohan-varma]

📚 The doc issue

https://discuss.pytorch.org/t/properly-implementing-ddp-in-training-loop-with-cleanup-barrier-and-its-expected-output/146465 mentions a couple of issues with DDP tutorial, when the framework is repurposed to train a different model (in this case a CIFAR classifier):

1. dist.barrier() causes a hang in demo with checkpoint: https://github.com/pytorch/tutorials/blob/a1ad9ed50305e96597a1a5d3c3d3d565e881e27e/intermediate_source/ddp_tutorial.rst
2. model parallel demo causes duplicate GPU issue, fixed by set_device.

The mentioned forum post has a repro script for the above 2 issues.

Suggest a potential alternative/fix

No response

cc @ezyang @gchanan @zou3519 @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @brianjo @mruberry

[awgu]

There was some misunderstanding of the tutorial. I have provided responses to the forum user.