`nan` error in Pytorch ImageNet-ResNet50

[dqgdqg]

OS: Ubuntu 18.04 LTS
GPU: NVIDIA TESLA V100 32G * 8
Docker: pytorch-18.09-py3

I ran these script files RN50_FP16_8GPU.sh, RN50_FP16_4GPU.sh, RN50_FP32_8GPU.sh, RN50_FP32_4GPU.sh, and all got nan loss after several epochs (<=6). After I replaced ResNet50 to ResNet18, there was also got nan loss after ~20 epochs.

I have tried to decrease lr and batch_size, but it not works.

PS: In addition, I could successfully run the mxnet version of ImageNet ResNet50v1.5.

[itsliupeng]

GPU: NVIDIA TESLA V100 32G * 8
Docker: tensorflow:19.04-py3
I meet the same problem when training tensorflow resnet50. It is caused by l2_loss nan.

[asulecki]

@dqgdqg what dataset are you using? If it is not ImageNet, can you try running on ImageNet?

@itsliupeng please file an issue regarding TF problems.

[dqgdqg]

@dqgdqg what dataset are you using? If it is not ImageNet, can you try running on ImageNet?

@itsliupeng please file an issue regarding TF problems.

@asulecki Well, I'm using the official ImageNet-2012 dataset indeed.

@itsliupeng BTW, I could train tensorflow resnet50 successfully via tensorflow:19.04-py3. In fact, I noticed that nan loss could appear in all three versions (MXNet, PyTorch and Tensorflow). But in my empirical experiment, the nan loss with PyTorch version is almost 100% occurs and the nan loss failure probabilities of MxNet and TF dockers are acceptable. You may try to run it several times more.

[asulecki]

@dqgdqg can you share the exact command you run? Did you try upgrading the container to 19.04/19.05 ?

[dqgdqg]

@dqgdqg can you share the exact command you run? Did you try upgrading the container to 19.04/19.05 ?

@asulecki
I just ran the RN50_FP16_8GPU.sh file in example folder. In addition, I have tried RN50_FP16_8GPU.sh, RN50_FP16_4GPU.sh, RN50_FP32_8GPU.sh, RN50_FP32_4GPU.sh, RN50_FP16_1GPU.sh, RN50_FP32_1GPU.sh. All of these scripts could cause nan loss after 5~20 epochs.

I have tried upgrading the container to 19.04, but it still didn't work.

[asulecki]

Due to merge problems the ./example directory that should be deleted, was not. You should use ./resnet50v1.5/training/* scripts as mentioned in readme.