evaluation results

[ouceduxzk]

First of all, thanks for sharing the work. I quickly run a test of AP with following results, do you know why it is too low?

python3 models/COCO.res50.256x192.CPN/mptest.py -d 0-1 -r 350
loading annotations into memory...
Done (t=2.09s)
creating index...
index created!
loading the precalcuated json files
Loading and preparing results...
4581
4581
DONE (t=2.98s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type keypoints
there are 40504 unique images
DONE (t=14.41s).
Accumulating evaluation results...
DONE (t=0.53s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets= 20 ] = 0.093
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets= 20 ] = 0.116
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets= 20 ] = 0.102
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.089
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.099
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 20 ] = 0.097
Average Recall (AR) @[ IoU=0.50 | area= all | maxDets= 20 ] = 0.117
Average Recall (AR) @[ IoU=0.75 | area= all | maxDets= 20 ] = 0.104
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.092
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.103
AP50
ap50 is 0.141489
ap is 0.099431

I added the AP calculation and saved the json file already

[chenyilun95]

Your result is similar to #11 . I think there is something wrong while testing. I found your testing images are
40504 unique images. We test the results on the COCO minival dataset which contains 5000 images and so as the provided detection boxes. You might get the wrong human detection in your dataset.

[ouceduxzk]

Thanks for your quick reply, you are right that my val json files is not the same, I am using the person_keypoints_val2014.json , can you provide those json files, in coco dataset official website, they are not existing anymore

[chenyilun95]

COCO 2014 minival json and its detection result json is provided.

[ouceduxzk]

Thanks, now it looks normal

DONE (t=0.37s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets= 20 ] = 0.697
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets= 20 ] = 0.883
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets= 20 ] = 0.770
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.662
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.761
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 20 ] = 0.764
Average Recall (AR) @[ IoU=0.50 | area= all | maxDets= 20 ] = 0.927
Average Recall (AR) @[ IoU=0.75 | area= all | maxDets= 20 ] = 0.823
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.715
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.830