valid mAP much higher than the original darknet using the same weights trained by this repo

[yanxurui]

Hi @AlexeyAB

I have trained this repo on a custom dataset.

I want to compute mAP on the validation set.

I run this command:
./darknet detector valid cfg/bdd.data cfg/yolov3-bdd.cfg backup/yolov3-bdd_30000.weights
Then compute the mAP using cocoapi on the result json file.
The mAP is around 16%.

I run the same command using the original darknet repo with config and weights being replaced with the same files.
The mAP is around 10%.

I also use ultralytics/yolov3 to calculate mAP which is around 10%.

So, why do the weights trained by this repo perform worse when using the original repo to do validation?

Thank you very much.

[AlexeyAB]

@yanxurui Hi,

The latest version of this repo supports 2 approaches of resizing: #232 (comment)

• resize (by default)

• letter_box (during Detection and checking mAP use flag -letter_box, during Training set letter_box=1 in the [net] section in cfg-file): https://github.com/AlexeyAB/darknet/wiki/CFG-Parameters-in-the-%5Bnet%5D-section

While pjreddie/darknet and ultralytics/yolov3 supports only letter_box.

[yanxurui]

@AlexeyAB
Got it.
Thank you very much.
I am using the default resize function, which makes it incompatible with the pjreddie/darknet and ultralytics/yolov3.

[AlexeyAB]

@yanxurui Also if you use this repository, then download the latest version.

Only in the last commit was added letter_box=1 to the [net] section.

[yanxurui]

Great.

I still don't understand one thing.
If the official weights are trained with the original aspect ratio, how can this repo achieve the same performance (mAP) during valid when it resize the input image to square?

[yanxurui]

Another question.
The mAPs of detector valid command using this repo with and without -letter_box are the same.
Why is that?
Since the weights are trained without using letter box, I suppose the valid command with -letter_box should be lower.

[AlexeyAB]

I still don't understand one thing.
If the official weights are trained with the original aspect ratio, how can this repo achieve the same performance (mAP) during valid when it resize the input image to square?

I don't understand it too :) I got higher mAP on original weights-file (that is trained with letter_box) by using resizing than by using letter_box during detection.
Maybe this is the best practice in some cases - to train with letter_box, and detect with resizing?

---

The mAPs of detector valid command using this repo with and without -letter_box are the same.
Why is that?

Have you downloaded the latest version of Darknet?

[yanxurui]

No. I am not using the latest version But I do see this option in my detector.c.
Is -letter_box also added recently?
Thanks.

[AlexeyAB]

Is -letter_box also added recently?

For checking mAP - yes.

For detection - several monthes ago.

[yanxurui]

Great.
Thank you.