Python API : Incorrect AP score with preditction = ground truth

[lulud41]

Hi,
I'm testing the Python API. I wanted to check if I could get 1.0 AP when the prediction is strictly equal to the ground truth, wich seems obvious. However, I get 0.73 AP@0.5 and other weard stuff. I'm using a custom annotation file

Am I doing something wrong ? Thanks in advance,

Here is my code :

from pycocotools.coco import COCO
from pycocotools.cocoeval import COCOeval
import json
import numpy as np

d = json.load(open("annotations.json"))

# construct a prediction array from ground truth, with (id_image, x1, y1, w, h, confidence, cls ) format

pred = np.zeros((7,7))
for idx, ann in enumerate(d["annotations"]):
    pred[idx,1:5] = ann["bbox"]
    pred[idx,5] = 1

print(f"prediction {pred}\n\n ground truth : {d}")

outputs
prediction
[[ 0. 61. 123. 191. 453. 1. 0.]
[ 0. 165. 95. 187. 494. 1. 0.]
[ 0. 236. 104. 195. 493. 1. 0.]
[ 0. 452. 110. 169. 508. 1. 0.]
[ 0. 520. 95. 163. 381. 1. 0.]
[ 0. 546. 39. 202. 594. 1. 0.]
[ 0. 661. 132. 183. 510. 1. 0.]]

ground truth : {
'type': 'instances',
'annotations':
[{'id': 0, 'image_id': 0, 'category_id': 0, 'bbox': [61, 123, 191, 453], 'iscrowd': 0, 'area': 86523},
{'id': 1, 'image_id': 0, 'category_id': 0, 'bbox': [165, 95, 187, 494], 'iscrowd': 0, 'area': 92378},
{'id': 2, 'image_id': 0, 'category_id': 0, 'bbox': [236, 104, 195, 493], 'iscrowd': 0, 'area': 96135},
{'id': 3, 'image_id': 0, 'category_id': 0, 'bbox': [452, 110, 169, 508], 'iscrowd': 0, 'area': 85852},
{'id': 4, 'image_id': 0, 'category_id': 0, 'bbox': [520, 95, 163, 381], 'iscrowd': 0, 'area': 62103},
{'id': 5, 'image_id': 0, 'category_id': 0, 'bbox': [546, 39, 202, 594], 'iscrowd': 0, 'area': 119988},
{'id': 6, 'image_id': 0, 'category_id': 0, 'bbox': [661, 132, 183, 510], 'iscrowd': 0, 'area': 93330}],
'images': [{'id': 0, 'width': 1024, 'height': 683}],
'categories': [{'id': 0, 'name': 'person', 'supercategory': 'person'}]}

coco_gt = COCO("annotations.json")
cooc_det = coco_gt.loadRes(pred)
coco_eval = COCOeval(coco_gt, cooc_det, "bbox")

coco_eval.evaluate()
coco_eval.accumulate()
coco_eval.summarize()

outputs :
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
Loading and preparing results...
Converting ndarray to lists...
(7, 7)
0/7
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type bbox
DONE (t=0.05s).
Accumulating evaluation results...
DONE (t=0.01s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.730
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.730
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.730
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.730
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.857
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.857
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.857
"""

[soheilazangeneh]

I faced the same issue!

[soheilazangeneh]

I figured out the problem. When the annotation ids start from 0, it does not return the right metric values!! Just start from 1 and it is fixed for me.
#332