multi-gpu training hangs

[ajtao]

Hmm, training with --gpus 0, works fine but training with --gpus 0,1 hangs right at initializing ddp ...

[ajtao]

Hmm, maybe this is related: Lightning-AI/pytorch-lightning#4612

[MicPie]

I seem to have the same problem. How did you solved it?

[ajtao]

In our environment we were setting NODE_RANK. pytorch lightning didn't like this to be set, so that fixed it for us.

[sanersbug]

@MicPie @ajtao I have the same problem when i training with --gpu 0, how did you solved it? and where to setting the NODE_RANK.
sorry, i even don't kown what's the NODE_RANK mean

[ajtao]

We had an environment variable called NODE_RANK set. We unset it, and it fixed things. Outside of that, i can't help you.