Feature MeanAveragePrecision 🔥

[hardikdava]

Description

This pr is related to feature #206 and #232.

Type of change

• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

How has this change been tested, please provide a testcase or example of how you tested the change?

Google colab for testing:https://colab.research.google.com/drive/1MT5wc6H8WPshQfPoQITr6mLOcfd47x5E?usp=sharing

Any specific deployment considerations

For example, documentation changes, usability, usage/costs, secrets, etc.

Docs

• Docs updated at docs/supervision/metrics/detection

Basline Metrics for comparision:

• Ultralytics
• Torchmetrics (utilizes CocoApi)

[hardikdava]

@SkalskiP @mayankagarwals , this feature is not completed yet, but just to keep track. I am still facing some issue in result difference to other baselines method. I will let you guys know once it is ready to tests.

[SkalskiP]

No rush, @hardikdava! In the meantime, I'm putting some work into: #170.

[hardikdava]

@SkalskiP
Comparision between different baselines:

model=yolov8n.pt

Framework	map50	map75	map
Ultralytics	0.605	-	0.44
Torchmetrics	0.6145	0.4786	0.4473
Supervision	0.56066	0.48054	0.432033

@mayankagarwals there is a bug, but I am not able to find it. Let me know if you have previous expirience with mAP.

[mayankagarwals]

I do have some experience but it's been a while. Can you give me a colab to reproduce these numbers for supervision? I'll take a look

[hardikdava]

I do have some experience but it's been a while. Can you give me a colab to reproduce these numbers for supervision? I'll take a look

The link of colab is attached in the description.

[mayankagarwals]

The link of colab is attached in the description.

Amazing, thanks!

[hardikdava]

@SkalskiP @mayankagarwals the bug is fixed. We have a working code for mAP.

[hardikdava]

@SkalskiP Should we add functions of plotting curves? or just values are fine?

[hardikdava]

@SkalskiP Final report on values:

model=yolov8n.pt

Framework	map50	map75	map
Ultralytics	0.605	-	0.44
Torchmetrics	0.6145	0.4786	0.4473
Supervision	0.6129	0.4807	0.448

model=yolov8s.pt

Framework	map50	map75	map
Ultralytics	0.759	-	0.585
Torchmetrics	0.7573	0.6331	0.5820
Supervision	0.7577	0.63638	0.585

[hardikdava]

@SkalskiP I have fixed the changes you requested.

[hardikdava]

@SkalskiP everything good? merging it?

[SkalskiP]

@hardikdava I'm working on unit tests. We will merge it when I'm done.

[SkalskiP]

@hardikdava could you help me out here? What is stored under the average_precisions field?

[hardikdava]

Average precision holds the result of ap at different iou threshold starting from 0.5 to 0.95 at an interval of 0.5 also the results are for each class. Columns represents classes and rows represents iou threshold.

[SkalskiP]

So what would be the expected shape of this np array? (N, 10)? Where N is the amount of classes?

[hardikdava]

Yeah, numpy array of (num of class, 10).

[SkalskiP]

I Done some unit teats and it looks like the shape is different :/

[SkalskiP]

Test 1

>>> import supervision as sv
>>> import numpy as np

>>> predictions = [
...     np.empty((0, 6), dtype=np.float32),
... ]
>>> targets = [
...     np.empty((0, 5), dtype=np.float32),
... ]

>>> sv.MeanAveragePrecision.from_tensors(predictions=predictions, targets=targets)
# MeanAveragePrecision(map=0, map50=0, map75=0, average_precisions=[])

Test 2

>>> import supervision as sv
>>> import numpy as np

>>> predictions = [
...     np.array([
...         [0.0, 0.0, 1.0, 1.0, 1, 0.9],
...     ]),
... ]
>>> targets = [
...     np.array([
...         [0.0, 0.0, 1.0, 1.0, 1],
...     ]),
... ]

>>> sv.MeanAveragePrecision.from_tensors(predictions=predictions, targets=targets)
# MeanAveragePrecision(map=0.9949999999999999, map50=0.995, map75=0.995, average_precisions=array([      0.995]))

Test 3

>>> import supervision as sv
>>> import numpy as np

>>> predictions = [
...     np.array([
...         [0.0, 0.0, 1.0, 1.0, 1, 0.9],
...     ]),
...     np.array([
...         [0.0, 0.0, 1.0, 1.0, 1, 0.9],
...     ]),
...     np.array([
...         [0.0, 0.0, 1.0, 1.0, 2, 0.9],
...     ]),
... ]
>>> targets = [
...     np.array([
...         [0.0, 0.0, 1.0, 1.0, 1],
...     ]),
...     np.array([
...         [0.0, 0.0, 0.8, 0.8, 1, 0.9],
...     ]),
...     np.array([
...         [0.0, 0.0, 1.0, 1.0, 2, 0.9],
...     ]),
... ]

>>> result = sv.MeanAveragePrecision.from_tensors(predictions=predictions, targets=targets)
# MeanAveragePrecision(map=0.864625, map50=0.995, map75=0.80875, average_precisions=array([    0.73425,       0.995]))

[SkalskiP]

Those are simply average precisions per class. Not per threshold but already averaged.

@hardikdava is it what you intended?

[hardikdava]

Ohh yeah, average_precision is already per class map values. Sorry for miscommunication. This is intended.