Skip to content

fix(metrics): count false positives on empty-GT images - #2397

Merged
Borda merged 4 commits into
developfrom
fix/metrics
Jul 3, 2026
Merged

fix(metrics): count false positives on empty-GT images#2397
Borda merged 4 commits into
developfrom
fix/metrics

Conversation

@Borda

@Borda Borda commented Jul 3, 2026

Copy link
Copy Markdown
Member

This pull request updates the mean average precision (mAP) metric calculation for object detection, ensuring that predictions on images with no ground-truth objects (background images) are properly penalized as false positives. This change corrects the prior behavior where such predictions did not reduce the mAP score, making the metric more accurate. The update is accompanied by new tests to verify the improved logic.

Metric calculation improvements:

  • Updated from_tensors in src/supervision/metrics/detection.py to treat all predictions on ground-truth-empty images as false positives, ensuring these predictions lower the mAP score as expected.

Testing enhancements:

  • Added TestMeanAveragePrecisionBackgroundFalsePositives in tests/metrics/test_detection.py with tests verifying that background false positives reduce mAP, and that normal cases without false positives are unaffected.

- legacy `MeanAveragePrecision.from_tensors` appended per-image stats only when ground truth was present, so predictions on background images were never counted as false positives and mAP was inflated
- append an FP-only stats tuple for prediction-present, target-empty images, mirroring the pattern in `f1_score.py`
- add regression test asserting background false positives lower mAP

---
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
@Borda
Borda requested a review from Copilot July 3, 2026 13:03
@Borda
Borda requested a review from SkalskiP as a code owner July 3, 2026 13:03
@codecov

codecov Bot commented Jul 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85%. Comparing base (f173905) to head (def82b2).

Additional details and impacted files
@@           Coverage Diff           @@
##           develop   #2397   +/-   ##
=======================================
  Coverage       85%     85%           
=======================================
  Files           70      70           
  Lines        10035   10036    +1     
=======================================
+ Hits          8489    8490    +1     
  Misses        1546    1546           
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the legacy MeanAveragePrecision.from_tensors computation in src/supervision/metrics/detection.py so that predictions made on images with no ground-truth objects (background images) are counted as false positives and reduce the resulting mAP, aligning the metric with expected detection-evaluation behavior.

Changes:

  • Updated MeanAveragePrecision.from_tensors to record all predictions on GT-empty images as false positives in the collected per-image stats.
  • Added tests ensuring background false positives reduce map50 and that the GT-present path remains numerically stable.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
src/supervision/metrics/detection.py Adjusts from_tensors stats collection to include GT-empty images so predictions are penalized as false positives.
tests/metrics/test_detection.py Adds regression tests covering background false positives and a pinned GT-present scenario.

Comment thread src/supervision/metrics/detection.py
Comment thread tests/metrics/test_detection.py
Borda and others added 3 commits July 3, 2026 15:43
- Guard empty average_precisions in from_tensors: return 0.0 for
  map50/map75/map50_95 when no GT classes exist (all-background dataset)
- Fix dtype of empty GT class array: np.zeros((0,), dtype=np.float32)
  to match explicit-dtype pattern at line 1408 and avoid float64 upcasting

[resolve #1] Copilot + /review — foundry:qa-specialist + foundry:challenger (gh)
[resolve #6] /review finding by foundry:linting-expert (report)

---
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
- Add test_all_background_images_return_zero_not_nan: verifies that a
  dataset with only background images (no GT objects anywhere) returns
  map50=0.0 rather than NaN after the from_tensors fix
- Strengthen test_background_false_positives_lower_map: add upper-bound
  assertion (map50 < 0.5), add map75 and map50_95 penalization checks

[resolve #2] Copilot + /review — foundry:qa-specialist (gh)
[resolve #4] /review finding by foundry:qa-specialist (report)

---
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
…ring

- Remove "Legacy" from TestMeanAveragePrecisionBackgroundFalsePositives
  docstring — from_tensors is the active tensor constructor, not deprecated
- Add N=0 background-image contract to targets arg description in
  from_tensors docstring (empty array = all predictions are false positives)
- Add docstring example demonstrating that predictions on a background
  image produce map50=0.0 (covers new empty-stats code path)
- Fix Returns section: add MeanAveragePrecision type prefix per Google style

[resolve #3] /review finding by foundry:doc-scribe (report)
[resolve #5] /review finding by foundry:doc-scribe (report)
[resolve #7] /review finding by foundry:doc-scribe (report)
[resolve #8] /review finding by foundry:doc-scribe (report)

---
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
@Borda
Borda merged commit 0e1056d into develop Jul 3, 2026
26 checks passed
@Borda
Borda deleted the fix/metrics branch July 3, 2026 14:17
@Borda Borda added the bug Something isn't working label Jul 3, 2026
Borda added a commit that referenced this pull request Jul 3, 2026
Resolve duplicate-change conflicts: several commits from this branch were
cherry-picked into separate PRs (#2391/#2393/#2394/#2396/#2397/#2398/#2399)
and squash-merged to develop, so merging develop back produced same-change
conflicts across 19 files.

Resolution:
- Keep this branch's later refactors: public `check_no_basename_collisions`
  (standardized collision-guard layer, renamed all `_check_*` call sites in
  dataset/core, yolo, labelme, tests), `save_pascal_voc_annotations` helper,
  value-based from_lmm->from_vlm mapping (drop dead lmm_to_vlm dict),
  CropAnnotator source-scene snapshot for overlapping boxes.
- Take develop's hardening/refinements: from_vlm empty-list guard, explicit
  float() mAP casts, np.float32 dtypes, MJPG/avi VideoSink tests, shared
  make_panoptic_png test helper, expanded empty-input adapter tests.
- Combine where both improve: np.int32 crop dtype + snapshot; keep both
  sides' added tests.

Full suite: 2924 passed, 1 xfailed. pre-commit (ruff, mypy, codespell) clean.

---
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
@Borda Borda mentioned this pull request Jul 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants