mAP by Darknet and mAP by Cartucho are different on detection same validation images.

[HanTnx]

Let's me explain shortly what I have done. Firstly, I trained my custom dataset on Darknet Ubuntu 18.04, the best weigtht on 840 validation images is mAP 87%. Secondly, I use this 87% best weight on OpenCV DNN Python code to detect the same 840 images and used the detection results to calculate mAP with Cartucho source code. But these two mAP are very different, you can see in include image

Can you explain or correct what I have done wrongly ? Thank you for your time and consideration

[AlexeyAB]

• Detections for mAP should be calculated with low confidence-threshold like -thresh 0.005 so you should use the same threshold in OpenCV-DNN to compare apples with apples

  darknet/src/detector.c

  Line 995 in 64efa72

  const float thresh = .005;

• OpenCV-DNN uses letter_box instead of resize by default, while Darknet uses resize by default (for training and detection, except yolov4-csp and yolov4x-mish models) Resizing : keeping aspect ratio, or not #232 (comment)

• Also what accuracy does calculate Cartucho code, is it mAP50, mAP75 or mAP50...95, how many points in Precision-Recall-curve are used?

[HanTnx]

To answer to your question

• Accuracy Cartucho code calculate, it is mAP 50.
  

• Points in Precision-Recal-curve are used, I am not sure from reading his code but he follows VOC challenge 2012 which is the same as Darknet instruction
  
  

• I follow your instruction to set confidence-threshold in OpenCV-DNN 4.4.0 as follows, and the mAP by Cartucho increase from 67% to 73% mAP50
  
  

• The final problem, I used Darknet yolov4 models (not yolov4-csp and yolov4x-mish models ), so OpenCV-DNN uses letter_box instead of resize by default. The OpenCV I used is 4.4.0, how about update to 4.5 (or latest one) solve this problem ? or Can you suggest me another framework with Python (pytorch, tensorflow) uses resize by default for detection ? Thank you

[poornimajd]

@HanTnx were you able to solve this difference in map issue?

[matt-sharp]

@HanTnx I'm also curious to know if you found a solution to get a better match between Darknet and Open-CV dnn results.