Proposal
I maintain ultrafast-pycocotools, a BSD-2-Clause Rust COCO evaluator. I would like to propose an explicit optional backend in the Detection reference scripts, selected with --coco-backend ultrafast, while retaining the default pycocotools path and existing dataset dependencies.
The proposed scope covers bbox, segmentation and keypoints, including the training/test-only CLI and engine.evaluate(). No torchvision core dependency or process-wide import replacement is needed.
This requires a real distributed adapter: ultrafast 0.1.11 evaluates and accumulates together, so assigning externally merged evalImgs does not recompute its curves. The implementation instead gathers prepared predictions, preserves the existing sorted image-ID / first-occurrence deduplication rule, and runs native evaluation during synchronization. Per-image records are requested explicitly so they remain inspectable. The per-batch evaluator timer consequently does not include this final work; I am not claiming a speed or memory improvement from unrelated standalone benchmarks.
A concrete implementation and tests are ready for a draft PR. Nine tests pass on macOS (Torch 2.14) and Linux (Torch 2.10 CPU), including every precision/recall/scores array and per-image matching record, repeated accumulation, pickle before synchronization, conflicting duplicate images, two-rank Gloo with an empty rank, CLI parsing and the engine evaluation path with fixed model outputs.
Would an optional evaluator backend in the reference scripts fit the project scope? I will keep the PR in draft while this proposal is discussed.
Proposal
I maintain ultrafast-pycocotools, a BSD-2-Clause Rust COCO evaluator. I would like to propose an explicit optional backend in the Detection reference scripts, selected with
--coco-backend ultrafast, while retaining the default pycocotools path and existing dataset dependencies.The proposed scope covers bbox, segmentation and keypoints, including the training/test-only CLI and
engine.evaluate(). No torchvision core dependency or process-wide import replacement is needed.This requires a real distributed adapter: ultrafast 0.1.11 evaluates and accumulates together, so assigning externally merged
evalImgsdoes not recompute its curves. The implementation instead gathers prepared predictions, preserves the existing sorted image-ID / first-occurrence deduplication rule, and runs native evaluation during synchronization. Per-image records are requested explicitly so they remain inspectable. The per-batch evaluator timer consequently does not include this final work; I am not claiming a speed or memory improvement from unrelated standalone benchmarks.A concrete implementation and tests are ready for a draft PR. Nine tests pass on macOS (Torch 2.14) and Linux (Torch 2.10 CPU), including every precision/recall/scores array and per-image matching record, repeated accumulation, pickle before synchronization, conflicting duplicate images, two-rank Gloo with an empty rank, CLI parsing and the engine evaluation path with fixed model outputs.
Would an optional evaluator backend in the reference scripts fit the project scope? I will keep the PR in draft while this proposal is discussed.