Question about AP definition

[0phoff]

I was wondering what the COCO api uses as definition for the AP metric (for object detection).
Seeing as this project is used by a lot of researchers to evaluate the performance of their object detection models, I would like to have more details about the chosen definition.
As I understand it, there are a few different ways to compute AP:

1. You can make the Precision monotonically decreasing, by taking the maximal precision of every
   R >= Rn for every point n on the curve. This removes the sawtooth-jiggles from the curve and is done in the Pascal VOC metric.
2. There are a few ways to perform the actual AP computation:
   - 11-point interpolated AP: compute the PR-values at [0, 0.1, ..., 1] and compute the AP as the arithmetic mean of these values (works because delta-r is constant).
   - Following the definition of AP according to scikit-learn and wikipedia (not the best sources, but all I could seem to find in terms of proper definition).
   This means taking the sum of Pn * (Rn - Rn-1) and is like approximating the Area under the curve with rectangles at each (P,R) point.
   - Approximating the AP by computing the Area under the curve (AUC). This is usually done by using the trapezium rule.

I would want to know this to be able to compare my results with other papers which use your framework, but I could not seem to find any explanation on the COCO website and the (python) code is quite difficult to get into. It could be nice to have this properly explained somewhere? (Or I completely missed it, in which case I'm sorry for opening this issue)

The differences between the different methods might not be that big, bug they are noticeable (mainly that the the first 2 methods are usually a small overestimation of the proper AUC).
Using the AUC seems the most correct metric to me, but then you would have called it AUC and not AP I guess? So I am not sure what you guys are using...

Thanks a lot in advance!

[harrisonford]

I would like to know more about the AP metric too, I wonder if "AP-head" is implemented internally in the cocotools?

[Menooker]

@0phoff @harrisonford Hi, I have found in the COCO Python implementation that it uses 11-point interpolated AP. You can follow the link:

cocoapi/PythonAPI/pycocotools/cocoeval.py

Line 507 in 636becd

self.recThrs = np.linspace(.0, 1.00, np.round((1.00 - .0) / .01) + 1, endpoint=True)

The variable recThrs should be "recall threshold" and it should be [0, 0.1,... 1.0] in the code

[thho]

@Menooker not sure how you interpret line 507 in 636becd as a 11 point interpolation,

cocoapi/PythonAPI/pycocotools/cocoeval.py

Line 26 in 636becd

# recThrs - [0:.01:1] R=101 recall thresholds for evaluation

providing the hint for a 101 point interpolation over the recall curve.

[Menooker]

@Menooker not sure how you interpret line 507 in 636becd as a 11 point interpolation,

cocoapi/PythonAPI/pycocotools/cocoeval.py

Line 26 in 636becd

# recThrs - [0:.01:1] R=101 recall thresholds for evaluation

providing the hint for a 101 point interpolation over the recall curve.

Oh... It's a typo. It should be 101-point as you pointed out, since np.round((1.00 - .0) / .01) + 1 = 101

[0phoff]

So they use yet another way...
Why is it so hard to have a single definition that is used by everyone ?

Any idea why one would use 101-point interpolation ?

I guess with that many points the differences between this AP and the scikit one would be minimal, but this one will probably ignore strong spikes more ?

I think I will just implement the scikit version in my own library, as that is the one with the definition I encounter the most when searching through the webs about AP.