Low loss and low MAP for damage detection

[MarcPoulin1]

When I train a YOLOv3 to detect damage on images of cars I get a low average loss but also a really low MAP. The model is overfitting a lot (see below). I have trained the model on 500 images of damaged car.

Any suggestion on how to reduice overfitting for this type of detection? I can increase the number of photos and change the config file. Also since the damage doesn't have any aspect ratio, what are the optimal anchors for this problem? If I increase the number of anchors will it help on the validation set?

[AlexeyAB]

What mAP do you get on Training dataset?

[MarcPoulin1]

For the last weights I get this:

Thanks for your help!

[AlexeyAB]

Your model is trained very well.

https://github.com/AlexeyAB/darknet#how-to-improve-object-detection

for each object which you want to detect - there must be at least 1 similar object in the Training dataset with about the same: shape, side of object, relative size, angle of rotation, tilt, illumination. So desirable that your training dataset include images with objects at diffrent: scales, rotations, lightings, from different sides, on different backgrounds - you should preferably have 2000 different images for each class or more, and you should train 2000*classes iterations or more

[MarcPoulin1]

Yes I understand this. The problem is that the object that I am trying to identify has too many variations. Damage is not a well defined object by clear features like cars, animals and others. My question is what do you suggest in order to have a better object detector? Should I only increase the number of images or can I optimize some parameters in the config file?

[AlexeyAB]

You use bad images in your training dataset.

Did you get Training and Validation datasets by evenly randomly dividing a Single dataset in a Training 80% and Validation 20% of images? You should do this.

[MarcPoulin1]

Yes the training and validation sets come from a random split 80%/20% of the same dataset.

[MarcPoulin1]

I have used the same code for another project that worked perfectly. Here is the python code for the spit (I masked some parts of the code):

#split train/validation for early_stopping and create text files with relative paths

import os
import random
import numpy as np

image_folder = MASKED!!!
#Select only images
image_list = [f for f in os.listdir(image_folder) if f.split('.')[-1] in ["jpg","jpeg"]]

#Split 80% train/20% validation
random.seed(5)
np.random.shuffle(image_list)

split_index = int(0.8*len(image_list))

#We dont want the same claim number in both training and validation (image title starts with a 9 digit code)
while image_list[split_index-1][:9]==image_list[split_index][:9]:
split_index+=1

training,validation = image_list[:split_index],image_list[split_index:]

def write_file(data,name):
#train/test/validation folder
data_folder = MASKED!!!!
data = ['data/obj2/' + f for f in data]
f = open(os.path.join(data_folder,name),'w+')
f.write('\n'.join(data))
f.close()

write_file(training,'train2.txt')
write_file(validation,'valid2.txt')

[AlexeyAB]

So I think that your dataset is too small.

Also what mAP@0.05 can you get by using ./darknet detector map ... -iou_thresh 0.05 command?

[MarcPoulin1]

At 0.05 I get a better mAP! What does it mean?
Thank you very much for your help!

[AlexeyAB]

Accuracy on the validation dataset is very low even at low iou_thresholds. While the accuracy on the training dataset is very high. This means that your training set is not representative - there are very few objects in it that you want to detect. Just collect 4-8x more images and follow these rules: #4359 (comment)

[MarcPoulin1]

Thank you! I will increase the number of images and post if I need some help later.

[maria-mh07]

I'm training yolov3 with COCO dataset (only 5 categories):

• train2014: traffic light=2893 images; stop sign=1214 images; cell phone=3322 images; mouse=1290 images; keyboard=1471 images
• val2014: traffic light=1437 images; stop sign=589 images; cell phone=1665 images; mouse=674 images; keyboard=750 images
  I downloaded the images and annotations from the official website and modified the annotations to be compatible with darknet: I converted (x,y) to be the center of the box and converted all measures to relative. When run training with -show_imgs flag I see correct bounded boxes of objects.

Problem: In training the error decreases but mAP is always 0.0%

COCO dataset has already been used for this type of training, so I don't use bad images in my dataset training, right? Any ideas?
please help

[AlexeyAB]

@maria-mh07

• Show chart.png file
• How many iterations did you train?
• What pre-trained weights-file do you use?
• Do you use latest version of Darknet?
• What versions of CUDA, cuDNN and OpenCV do you use?
• What cfg-file do you use?

[Aleksei91]

Yes I understand this. The problem is that the object that I am trying to identify has too many variations. Damage is not a well defined object by clear features like cars, animals and others. My question is what do you suggest in order to have a better object detector? Should I only increase the number of images or can I optimize some parameters in the config file?

I do not think that damage is object. In COCO terms it is "stuff", I think it can be better use semantic segmentation networks like BiSeNet and others (but its much harder to annotate images). https://paperswithcode.com/sota/real-time-semantic-segmentation-on-cityscapes
But I'm not really sure, you should try and compare. If you have only 1% test result you definitely should increase your dataset.

[maria-mh07]

• Only 4267 iterations because because mAP did not increase
• darknet53.conv.74
• How do I check if I have the lastest version? I run git clone https://github.com/AlexeyAB/darknet.git on 28 October
• CUDA 10.1 with its associated cuDNN version and OpenCV 4.1.1
• yolov3.cfg
  @AlexeyAB Thanks for your answer

[AlexeyAB]

@maria-mh07

Show bad.list and bad_labels.list files

[maria-mh07]

• bad.list
  data/img/00006.jppg
  data/img/00006.jppg
  data/img/00006.jppg
  data/img/00006.jppg
  data/img/00006.jppg
  data/img/00006.jpp
  data/img/00006.jpp
  data/img/00006.jpp
  data/img/00006.jpp
• But but I can't find bad_labels.list

[AlexeyAB]

@maria-mh07

• Check that your class_id [0 - 4] in your txt-annotations
• What params do you use in Makefile?
• Attach your cfg-file
• Download new version of Darknet and recompile

[maria-mh07]

• class_id [0-4], but some txt-annotations are empty. I also think that in train.txt there are some repeated routes. I don't know if this affects.
• I did'nt modify Makefile
  GPU=0
  CUDNN=0
  CUDNN_HALF=0
  OPENCV=0
  AVX=0
  OPENMP=0
  LIBSO=0
  ZED_CAMERA=0
• train_yolov3.zip
• I had a hard time compiling darknet to make it work ... but if necessary I recompile

[AlexeyAB]

@maria-mh07

class_id [0-4], but some txt-annotations are empty. I also think that in train.txt there are some repeated routes. I don't know if this affects.

This is normal.

set
GPU=1
CUDNN=1
in the Makefile
and do

make clean
make

[maria-mh07]

Ok thank you very much @AlexeyAB Sorry for this silly question but how do I do it on windows?

[AlexeyAB]

https://github.com/AlexeyAB/darknet#how-to-compile-on-windows-using-cmake-gui

[maria-mh07]

Thanks!! =)

[maria-mh07]

I did a training with 128 images, I used yolo_mark for the annotations and it seems to be going well.

So I think there is somthing wrong with my .txt annotations or train.txt and valid.txt files.

• Does it affect if I have different numbers of decimals? For example some annotations
  0 0.14718 0.804426 0.294359 0.328337
  0 0.197656 0.700234 0.0 0.0
  0 0.55082 0.740858 0.065422 0.126893
  0 0.317961 0.702476 0.033266 0.087864
• I have repeated paths to the same imagen in valid.txt/train.txt files. For example in train.txt
  data/img/img1.jpg
  data/img/img1.jpg
  data/img/img2.jpg
  data/img/img2.jpg
  ...
• Does any of this affect?

[AlexeyAB]

Does it affect if I have different numbers of decimals?

No, it's normal

0 0.197656 0.700234 0.0 0.0

How can be object with zero size? It's bad.

I have repeated paths to the same imagen in valid.txt/train.txt files.

Its normal.

Check your dataet by using Yolo_mark and run training with flag -show_imgs

[maria-mh07]

• You are right that height = width = 0, but COCO annotations were like that.
• I show an example. The original image:
  
  classe=bicycle
  yolo_mark annotation: 0 0.071875 0.666667 0.101563 0.302778
  my_annotation: 0 0.071625 0.666903 0.098969 0.304972
  When I use flag -show_imags:
  
  I can see the bounding box but the image is shifted

[maria-mh07]

Would you look at my conversion scrip of the bounding boxes, please?
I would really appreciate it because I don't know what I'm doing wrong
filter_train.zip
Thank you very much for all the help

[AlexeyAB]

@maria-mh07

I can see the bounding box but the image is shifted

It's normal. This is due data augmentation. Your dataset is correct.

Train more using much more images. https://github.com/AlexeyAB/darknet#when-should-i-stop-training

[maria-mh07]

ok thank you very much for the help! =)