can you help me to solve it ?WARNING: non-finite loss, ending training tensor([ nan, 0.22070, 0.00000, nan], device='cuda:0')

[shenghsiaowong]

WARNING: non-finite loss, ending training tensor([ nan, 0.22070, 0.00000, nan], device='cuda:0')

[tanluren]

Hi, it looks like gradient explosion, did you load pretrained model? did you prepare your data according to https://github.com/ultralytics/yolov3/wiki/Train-Custom-Data ?could you show the command you used to train?

[shenghsiaowong]

Hi, it looks like gradient explosion, did you load pretrained model? did you prepare your data according to https://github.com/ultralytics/yolov3/wiki/Train-Custom-Data ?could you show the command you used to train?

this is my train command ,python train.py --cfg cfg/yolov3-1cls.cfg --data data/voc.data --weights darknet53.conv.74

[tanluren]

you may try to lower the lr and use yolov3.weights as pretrained model

[chouxianyu]

@shenghsiaowong
Are you using focal loss?
If you are using it,you may try removing the focal loss.

A few days ago,i used focal loss and found an error(obj_loss is nan,other loss,other losses are occasionally nan) .
Accoarding to issue842, i have tried many ways:lower the lr,remove the tiny object ,lower the weight of
losses in the hyp and so on,but the error still exists.
Finally I removed focal loss on the obj_loss and the nan error didn't appear anymore.
So you may try removing the focal loss.For example,if your obj_loss is nan,just remove the focal loss on obj_loss).

[xiaozhangya]

you may try to lower the lr and use yolov3.weights as pretrained model
Hello, what if you change the network slightly and cannot use the pre-trained network? How can I start training a model from scratch? thank you for your reply!