Error step_pretrain on Rank

[lileiooo]

    Hello, I run the pre-training model in the environment of 4 gpus, and Error step_pretrain on Rank 1, 3, 2 0 is displayed, but the pre-training is not successful。

[tsujuifu]

The message is from here, which is a try-except block.
You can remove that block and see where exactly it gets an error.

I have this try-except because the distributed training in torch.distributed may sometimes stuck on our system.
If you get this error every time/iteration, it seems somewhere is not running well.