loss is nan, stopping training!

[Haoqing-Wang]

Wonderful job! When I try to train the Swin-B for 800 epochs, I meet this problem, 'loss is nan, stopping training'. But I find the loss value has no question. If I skip this error, the loss will keep as nan forever. Would you have some suggestions for this problem? Thanks very much!

[LayneH]

Hi, there might be multiple reasons for loss becoming NaN, e.g., AMP.
I did not encounter this problem in my pre-training experiments, so I am not sure what the problem is.
There are similar issues raised in the repo of MAE, you might want to check if they help (facebookresearch/mae#42).

[Haoqing-Wang]

Thanks a lot!