New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
yolov3: "Avg IOU: -nan, Class: -nan, Obj: -nan" while Retraining on voc data to detect custom class along with existing 20 classes #622
Comments
Just do the training for atleat 5k iterations , these nans will decrease . They generally come when subdivision batch has zero annotation (see count=0) . It can be because the anchors are big or object dimensions are too small. |
When I have finished retraining, I only get below results
** But not like below, with detection probability and also Prediction Boxes are missing too.
gaurav@Desktop-192:~/yolov3/darknet$ |
change batch and subdivisions to 1 in the cfg and try |
It didnt help.Attaching snapshot
It doesnt add bounding box in prediction.png Also Training keeps showing NAN but initial iteration does had valid non NAN values
|
./darknet detector train cfg/voc.data cfg/yolov3-voc.cfg darknet53.conv.74 |
default config( |
I with setting mentioned above only I am unable to detect the object. it just shows till there is no bounding box or below kinda results shown |
if you are not getting annotated image at the end , the problem is with the opencv version. Which OPencv version do you have . Try to install 3.4.1 and re-test |
@ztilottama I got the same question. do you fix it and how? |
Same problem. I'm training on VOC 2012 like it described here: https://pjreddie.com/darknet/yolo/ |
Tested on different PC - yes, that was the source. I think issue can be closed: just use bigger batch. |
@justanotheruser : What is the root cause, did you figure it out? Can you provide me the solution which solved this problem. What was the batch size you have used |
@ztilottama : I used batch=64 and subdivisions=16, but it can be specific to your task. With single image SGD was diverging. |
@axzxc1236 it's seems that your loss is incredible high |
Note: If during training you see nan values for avg (loss) field - then training goes wrong, but if nan is in some other lines - then training goes well. |
Same problem. I have changed batch and subdivisions to 1 in the cfg and try When I have finished retraining, I only get below results: No detection probability and also Prediction Boxes are missing too |
I got the same question. do you fix ?? |
https://medium.com/@manivannan_data/how-to-train-yolov3-to-detect-custom-objects-ccbcafeb13d2
like this (when classes =2) |
Hi, can you please specify what "anchors" refer to? thnx! |
I am retraining yolov3 to detect custom class along with 20 existing classes using VOc 2007 and 2012 dataset. I followed instructions on
Training is happening on GPU,
Snapshot is as below
Region 82 Avg IOU: 0.236180, Class: 0.205503, Obj: 0.004179, No Obj: 0.009506, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: 0.150655, Class: 0.315263, Obj: 0.005658, No Obj: 0.004314, .5R: 0.000000, .75R: 0.000000, count: 4
Region 106 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.001968, .5R: -nan, .75R: -nan, count: 0
279: 33.403381, 14.190192 avg, 0.000006 rate, 29.993717 seconds, 279 images
Loaded: 0.000050 seconds
I already done changes to obj.names obj.data as mentioned. My doubt is whether its okay to get -nan in training or I am missing out something.
Also how should I debug the root cause if -nan are not welcome
The text was updated successfully, but these errors were encountered: