why do Recall / F1Score use a different matching algorithm than ConfusionMatrix

[abhijithneilabraham]

Hi,

I've been reading through (and trying to learn) the metrics module to learn how it works, and I noticed something I can't fully explain. On the same input, the matcher used inside ConfusionMatrix.evaluate_detection_batch and _split_detections_by_outcome gives me a different TP count than the matcher used inside Recall._match_detection_batch, F1Score._match_detection_batch and MeanAverageRecall._match_detection_batch.

Wanted to ask the maintainers if this is intentional before I assume anything.

Minimal repro (uses the COCO val image of two cats just so the example sits on a real image, no model needed):

import urllib.request, ssl, cv2, numpy as np
import supervision as sv
from supervision.metrics import Recall, F1Score
from supervision.metrics.detection import _split_detections_by_outcome

ctx = ssl.create_default_context(); ctx.check_hostname = False; ctx.verify_mode = ssl.CERT_NONE
buf = np.frombuffer(urllib.request.urlopen(
    "https://images.cocodataset.org/val2017/000000039769.jpg", context=ctx).read(), dtype=np.uint8)
_ = cv2.imdecode(buf, cv2.IMREAD_COLOR)   # just confirming the URL loads

T0, T1 = (40, 60, 380, 470), (210, 60, 550, 470)   # ground truth around each cat
P0, P1 = (40, 60, 380, 470), (108, 60, 448, 470)   # two predictions

preds = sv.Detections(xyxy=np.array([P0, P1], np.float32),
                      confidence=np.array([0.95, 0.90], np.float32),
                      class_id=np.array([0, 0]))
gts   = sv.Detections(xyxy=np.array([T0, T1], np.float32),
                      class_id=np.array([0, 0]))

tp, fp, fn = _split_detections_by_outcome(preds, gts, conf_threshold=0.0, iou_threshold=0.50)
print("splitter        :", len(tp), len(fp), len(fn))
print("Recall  @ 0.5   :", float(Recall().update(preds, gts).compute().recall_at_50))
print("F1Score @ 0.5   :", float(F1Score().update(preds, gts).compute().f1_50))

Output I'm seeing:

splitter        : 2 0 0
Recall  @ 0.5   : 0.5
F1Score @ 0.5   : 0.5

The IoU matrix for these boxes is [[1.000, 0.667], [0.333, 0.538]]. At threshold 0.5 there's exactly one valid pairing (T0 with P0, T1 with P1) which I think should give 2 TPs, but the metric classes seem to be returning 1.

Reading the code on develop, here's what I'm finding:

One style of matcher, which uses np.lexsort plus a greedy loop that tracks which targets/predictions are already taken:

• _split_detections_by_outcome (metrics/detection.py:196-340, np.lexsort at line 310)
• ConfusionMatrix.evaluate_detection_batch (metrics/detection.py:832-994, np.lexsort at line 955)

Another style, which sorts the candidate matches by IoU and then deduplicates with two np.unique calls:

• Recall._match_detection_batch (lines 311 and 312)
• F1Score._match_detection_batch (lines 357 and 358)
• MeanAverageRecall._match_detection_batch (lines 540 and 541)
• MeanAveragePrecision._match_detection_batch (lines 1524 and 1525)
• ConfusionMatrix._drop_extra_matches (looks like it isn't called anymore but I wasn't sure)

I get that the np.unique style is what YOLOv5's process_batch and the kaanakan upstream do (with a documented caveat in kaanakan#21 and yolov5#12673 about it dropping valid matches in some overlap patterns), so I assume that's where it came from originally. What I'm trying to figure out is why the two styles now sit side by side in the same module.

A few things I wanted to ask:

1. Is the np.unique style being kept on purpose in Recall / F1Score / MeanAverageRecall for parity with YOLOv5, while ConfusionMatrix.evaluate_detection_batch and _split_detections_by_outcome were intentionally moved to the other style?
2. Or was the move to np.lexsort not meant to stop at those two functions, and the sibling classes just haven't been updated yet?
3. If (2) is closer to the truth, would you be open to a PR that pulls the np.lexsort style out into a single helper and points the other matchers at it?

Happy to send the PR if (2) is what you'd want. Thanks for reading and apologies if I'm missing context here.

[VectorPeak]

This doesn’t look intentional. For this repro, the expected matching at IoU 0.5 is:

T0 -> P0
T1 -> P1

That gives 2 TP, 0 FP, 0 FN, so Recall@0.5 == 1.0 and F1@0.5 == 1.0.

The mismatch comes from the older “sort by IoU, then dedupe predictions, then dedupe targets” style of matching. In your matrix it sees (T0, P0)=1.0, (T0, P1)=0.667, (T1, P1)=0.538; after deduping by prediction it keeps (T0, P1), and then deduping by target drops the valid (T1, P1) match. That is greedy-looking, but it is not the same as a one-to-one greedy assignment loop.

The ConfusionMatrix / _split_detections_by_outcome behavior is the one I’d expect here: iterate candidate pairs in priority order, and accept a pair only if both the prediction and target are still unmatched. Recall, F1Score, and MeanAverageRecall should use the same matching semantics, ideally through one shared helper, so these metrics don’t disagree on TP attribution.

This would be a good regression test case. Something like your repro reduced to just the two-box IoU setup should assert:

assert len(tp) == 2
assert len(fp) == 0
assert len(fn) == 0
assert float(Recall().update(preds, gts).compute().recall_at_50) == 1.0
assert float(F1Score().update(preds, gts).compute().f1_50) == 1.0

If current develop already routes those metric classes through _greedy_match, then this may be fixed there and the discrepancy is from the released package version. If not, I’d treat it as a bug rather than intended metric-specific behavior.

---

If my answer solved your problem, you can click answered the question. I'm really here to help, and along the way I'm also collecting Galaxy Brain badges haha 😆