Multi-node+multi-GPU papers100m+GCN example

[puririshi98]

working w/ nvidia pyg container

[rusty1s]

Wondering if it is possible to merge this with the initial multi-GPU example, or whether there exists a strong reason not do this.

[puririshi98]

Wondering if it is possible to merge this with the initial multi-GPU example, or whether there exists a strong reason not do this.

The set up is relatively different. The old one uses mp.spawn for single node multigpu, for multinode multigpu we are using torch distributed w/ nccl backend, i think for now we should keep them seperate. additionally for the single node multigpu its very simple to run, just python3 scriptname.py and it handles the rest, for multinode you need to run several slurm commands to get it to work (see comments on file). i think a separate multi-node-multi-gpu example and tutorial (#8071) is more appropriate given the extra complexity

[akihironitta]

This will be a very nice addition! 🚀

Haven't tried running this example in a multi-node multi-gpu environment myself, but the code looks good to me. Just like Matthias minimised the other example in #7954, I think we should keep this example minimal, too, where it makes sense.

[puririshi98]

@akihironitta @rusty1s lmk if anything else needed to merge

[rusty1s]

Thanks @puririshi98. I cleaned it up a bit. I really like the example. One thing we should definitely improve on would be to allow distributed evaluation as well, but it is not blocking this PR.