Improve multi-node multi-gpu tutorial

[flxmr]

So, in response to my remarks in #8071 I now prepared this PR for updating the multi-node documentation (sadly no student contrib, they can prep multi-gpu metrics).

Reasoning:

• if pyg has a tutorial on DDP it's for people who essentially grow within pyg to this usecase. This should be reflected in telling them more about this and be clear on what is done (like torch.multiprocessing injecting the rank. it seems very random otherwise).
• in the same vein, the multi-node tutorial should evolve from the single-node one imho.
• embedding the pyxis-container as the only way to do things is good for Nvidia, but bad for people who arrive on a system without that.

Things I skipped:

• setting up the worker-count. I would say people should just figure this out on their own. the os.sched_getaffinity is cool, but this is a very general problem too and I essentially do it manually now.
• anything GRES-related. I doubt that anyone growing into this and installing slurm on their computers will not grok this. Every multi-user, public research system has GRES I'd say (and everyone has their internal docs).

[akihironitta]

I'm not a slurm expert, but LGTM. Just pushed a few commits, but feel free to revert them if you think otherwise.

[flxmr]

So, now I tried the pyxis-example on a cluster I got access too (and also to check whether this is worth implementing for our own cluster)... and it doesn't work (I built my own container, but I don't know where the master-adress would come from even in the early-access-NGVC one...). Will wrap this into an sbatch-file too and then do a final version. Maybe @puririshi98 can tell me how this is working (I installed pyg into this)

[flxmr]

I didn't find how to make the srun-only example work with the containers. The current example which I just copied doesn't work: there is no environment variables in the pytorch-only NGC-container and I wonder how those would be injected anyway. Also the mounting doesn't make sense. So now it is sbatch only, but verified to work on our local HPC (which also has a tutorial on modifying the enroot-container).

In addition I noticed by trying this, that having multiple processes download data and then trying to unzip it is not working well (it worked previously, because the data was downloaded already). Fixed that too.

[puririshi98]

LGTM, thanks for the improvements :)

[puririshi98]

Although it looks good to me we should definitely make sure containers work as my original PR was tested working on a pyxis enabled nvidia cluster using our NVIDIA early access PyG container.

[flxmr]

So, I checked again and it seems pyxis/enroot is indeed injecting the LOCAL_RANK, RANK and MASTER_ADRESS into the container: https://github.com/NVIDIA/enroot/blob/master/conf/hooks/extra/50-slurm-pytorch.sh#L32-L37

it seems to be quite a mess though (user→slurm→pyxis→enroot): NVIDIA/pyxis#46 (comment)

Maybe this is site-configuration specific, they have it our HPC centers DGX don't have it? Would be nice if you check this, then you can revert the doc-rewrite (it still needs a single process downloading though!)

[puririshi98]

@flxmr I will investigate and get back to you as soon as I can

[puririshi98]

@flxmr i asked around internally with our enroot team

Pytorch is supported through this enroot hook: https://github.com/NVIDIA/enroot/blob/master/conf/hooks/extra/50-slurm-pytorch.sh
It is shipped by default and just need to be enabled by the administrator

does this help? I can follow back up if not.

[flxmr]

So, hope this works now for everyone. I added a link to this issue because I suppose if our HPC didn't like the hook, others might not like it too.

[puririshi98]

LGTM now, thanks @flxmr for this great PR