Mismatched mAP of TF2 object detection modelzoo #9127
Description
Prerequisites
Please answer the following question for yourself before submitting an issue.
- I checked to make sure that this issue has not been filed already.
1. The entire URL of the documentation with the issue
2. Describe the issue
I use TF-2.3.0 and follow your instruction to set up the environment for OD API. I try to test your models of TF2.X from the detection modelzoo, only to find my mAPs seem higher than the reported on the table. I use the 8059 images from your given list to create tfrecord for evaluation.
Could you check if the mAPs of the TF2-OD-Modelzoo are given with your default settings (score_thresh, nms parameters, etc.) in the pipeline.config ? My results are listed in the following table for comparison.
(Another problem: the input_size in your SSD-MobileNet-v2-320x320 pipeline.config is 300x300 instead of 320x320)
I'll appreciate it if you could help to check the issue.
| model | Claimed mAP | my results |
|---|---|---|
| SSD MobileNet v2 320x320 | 20.2 | Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.244 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.413 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.251 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.014 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.113 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.463 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.242 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.363 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.382 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.057 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.278 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.651 |
| SSD MobileNet V1 FPN 640x640 | 29.1 | Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.335 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.521 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.366 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.094 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.308 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.470 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.297 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.477 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.515 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.222 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.519 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.668 |
| SSD ResNet50 V1 FPN 640x640 (RetinaNet50) | 34.3 | Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.402 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.592 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.446 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.126 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.384 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.556 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.334 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.530 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.568 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.264 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.582 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.726 |