Benchmark results — open tracker (contribute a row!)

[Hollis36]

The honest story

The README says "Benchmarks: pending." Here's why.

Why no numbers yet

TianWen v0.2 has the wiring in place but hasn't been run end-to-end on a controlled comparison. Production benchmarks require:

• 3 seeds per configuration
• Frozen baseline (raw detector) for delta comparison
• Identical compute budget per arm
• Logged exactly enough so reviewers can reproduce

Doing this casually means publishing numbers nobody can defend. We'd rather wait.

What we commit to publish

For each (detector, VLM, fusion, dataset) cell we evaluate:

Field	Notes
mAP@50, mAP@50:95	torchmetrics standard, full COCO val
FPS at deploy	Detector-only on a single A100 (no VLM at inference)
Training GPU-hours	Real wall-clock, not theoretical
Seed	3 runs, mean ± std
Config hash	Reproducible from python tools/train.py experiment=...
Notes	Anything weird

Including the rows where TianWen doesn't help, because that's how you learn where the wiring is or isn't pulling weight.

Current target — v0.3

• YOLOv8-L + Qwen2-VL-7B + feature distillation on COCO val (3 seeds)
• Same on LVIS rare-class subset (long-tail stress test)
• Detector-only baseline for both
• Same with distill_mode=logit for ablation

Want to contribute a benchmark?

If you have GPUs and want to run any combination:

1. Pick a cell from the matrix (or invent one)
2. Run python tools/train.py experiment=<your-config> seed=42 × 3 seeds
3. Reply on this thread with:
   • Config name + any overrides
   • Seeds used
   • mAP@50, mAP@50:95
   • FPS (detector only)
   • Training wall-clock + hardware
   • Anything you noticed
4. We'll add it to the tracking table and credit you in the README

That's how the table fills up. Even one contributed row would unlock the first README update.