getting nan value while training

[anju54]

when i'm trying to train my dataset on yolo it's giving me some nan value.
I have 200 images of dataset. and I want to train my network so that I could get weights for my model.

please help me resolving this error.
Region Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.469436, Avg Recall: -nan, count: 0
Region Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.461745, Avg Recall: -nan, count: 0
Region Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.463417, Avg Recall: -nan, count: 0
Region Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.463578, Avg Recall: -nan, count: 0
Region Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.464747, Avg Recall: -nan, count: 0
Region Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.465984, Avg Recall: -nan, count: 0
Region Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.463966, Avg Recall: -nan, count: 0
Region Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.463675, Avg Recall: -nan, count: 0
1: 217.337524, 217.337524 avg, 0.000500 rate, 1.763889 seconds, 16 images
Loaded: 0.000051 seconds

[AlexeyAB]

• What repo do you use this https://github.com/AlexeyAB/darknet or this https://github.com/pjreddie/darknet ?
• What cfg file do you use as base for your cfg-file: yolo-voc.2.0.cfg or yolo-voc.cfg?
• How did you get your training dataset, what software did you use to get labels?
• Do you have values of x,y,w,h <= 0 or >=1 in your leabel txt-files?

[anju54]

1.I have used this
https://github.com/pjreddie/darknet ?
2. yolo-voc.2.0.cfg
3. bbox label tool
4. my some of the text files are as follows: :
0 0.630078125 0.51875 0.041406250000000006 0.23958333333333334
0 0.5769531250000001 0.4791666666666667 0.03671875 0.16458333333333333

[AlexeyAB]

If you used bbox-label-tool instead of Yolo_mark, then you should check each your txt-file, and check that none of the objects are outside the screen, i.e. none values >= 1 or values =< 0

[anju54]

yeah, i have cross checked the same.
none of the txt file contain value lesser then 0 and greater then 1.

[AlexeyAB]

Try to add the parameter small_object=1 in the last layer [region] in your cfg-file: https://github.com/AlexeyAB/darknet#how-to-improve-object-detection
And try to train using this fork: https://github.com/AlexeyAB/darknet

[inders]

I am facing the same while the darknet repo on VOC dataset. What's the resolution here guys?

[Luxato]

@inders are you using the https://github.com/AlexeyAB/darknet fork and did you put small_object=1 into your cfg file as mentioned above?

[kpreeti096]

when i'm trying to train my dataset on yolo it's giving me some nan value.
I have 60 images of dataset. and I want to train my network so that I could get weights for my model.45: 14.083967, 18.795122 avg, 0.000000 rate, 177.142862 seconds, 360 images
Loaded: 0.000055 seconds
Region Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.500224, Avg Recall: -nan, count: 0
Region Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.498702, Avg Recall: -nan, count: 0
Region Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.501422, Avg Recall: -nan, count: 0
Region Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.501599, Avg Recall: -nan, count: 0

I am also getting the same issue.

1. I have used https://github.com/AlexeyAB/darknet
2. yolo-obj.cfg
3. bbox label tool
   4.I don't have values of x,y,w,h <= 0 or >=1 in my leabel txt-files

Suggest me if anyone have the idea to deal with this issue

[skyuuka]

I also have a similar issue. Is it normal that when count = 0?

Region 106 Avg IOU: 0.726446, Class: 0.989863, Obj: 0.500152, No Obj: 0.000140, .5R: 1.000000, .75R: 0.500000, count: 2
Region 82 Avg IOU: 0.905383, Class: 0.999315, Obj: 0.999289, No Obj: 0.007577, .5R: 1.000000, .75R: 1.000000, count: 5
Region 94 Avg IOU: 0.839787, Class: 0.999768, Obj: 0.994878, No Obj: 0.000413, .5R: 1.000000, .75R: 1.000000, count: 2
Region 82 Avg IOU: 0.893517, Class: 0.999831, Obj: 0.999987, No Obj: 0.010304, .5R: 1.000000, .75R: 1.000000, count: 6
Region 106 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.000000, .5R: -nan, .75R: -nan, count: 0
Region 94 Avg IOU: 0.727787, Class: 0.999884, Obj: 0.954851, No Obj: 0.000401, .5R: 1.000000, .75R: 0.000000, count: 1
Region 106 Avg IOU: 0.732136, Class: 0.999486, Obj: 0.839145, No Obj: 0.000030, .5R: 1.000000, .75R: 0.000000, count: 1
Region 82 Avg IOU: 0.859362, Class: 0.998684, Obj: 0.985568, No Obj: 0.015233, .5R: 1.000000, .75R: 1.000000, count: 11
Region 94 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.000231, .5R: -nan, .75R: -nan, count: 0
Region 106 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.000000, .5R: -nan, .75R: -nan, count: 0
Region 82 Avg IOU: 0.906254, Class: 0.982144, Obj: 0.999936, No Obj: 0.009709, .5R: 1.000000, .75R: 1.000000, count: 6
Region 94 Avg IOU: 0.681093, Class: 0.969315, Obj: 0.998132, No Obj: 0.000346, .5R: 1.000000, .75R: 0.000000, count: 1
Region 106 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.000000, .5R: -nan, .75R: -nan, count: 0
Region 82 Avg IOU: 0.861664, Class: 0.997199, Obj: 0.961002, No Obj: 0.010158, .5R: 1.000000, .75R: 0.800000, count: 10
Region 94 Avg IOU: 0.819717, Class: 0.999879, Obj: 0.990960, No Obj: 0.000866, .5R: 1.000000, .75R: 0.750000, count: 4
Region 82 Avg IOU: 0.851591, Class: 0.992718, Obj: 0.999991, No Obj: 0.008075, .5R: 1.000000, .75R: 0.800000, count: 5
Region 106 Avg IOU: 0.713969, Class: 0.999687, Obj: 0.983980, No Obj: 0.000036, .5R: 1.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: 0.810556, Class: 0.999998, Obj: 0.999999, No Obj: 0.000262, .5R: 1.000000, .75R: 1.000000, count: 1
Region 82 Avg IOU: 0.851022, Class: 0.971247, Obj: 0.999922, No Obj: 0.013315, .5R: 1.000000, .75R: 1.000000, count: 7
Region 106 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.000000, .5R: -nan, .75R: -nan, count: 0
Region 94 Avg IOU: 0.826256, Class: 0.998733, Obj: 0.992057, No Obj: 0.001231, .5R: 1.000000, .75R: 0.666667, count: 3
Region 106 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.000000, .5R: -nan, .75R: -nan, count: 0
Syncing... Done!
50208: 0.257869, 0.233150 avg, 0.000040 rate, 6.741087 seconds, 12853248 images
Saving weights to backup/yolov3-voc_final.weights

[AlexeyAB]

@skyuuka It is normal. Your training goes well.