RampNet v1.1 — corrected evaluation protocol

Same code, same model weights, same dataset as v1.0-iccv2025. Only the evaluation protocol changed.

After publication we found that the prediction/ground-truth matching rule described in §3.3 of the paper — and implemented in the evaluators at v1.0-iccv2025 — differs from standard detection evaluation in two ways, both of which bias precision and recall upward. This release is the repository at the point the Stage 2 gold-set numbers had been re-measured under standard one-to-one matching, and both evaluators had been unified onto a single matching core.

Nothing here supersedes the paper's model or dataset — those are unchanged and remain the published artifacts. It supersedes the paper's evaluation of them.

What was wrong

1. One detection could count as multiple true positives. A single predicted point within the matching radius of two ground-truth curb ramps — a common configuration at dual-ramp corners and pedestrian islands — was credited with both. 41% of gold-set ground-truth points sit within 2× the matching radius of a neighbour.
2. Redundant detections were ignored rather than counted as false positives. A second detection of an already-matched ramp appeared in neither the TP nor the FP column; VOC/COCO-style protocols count it as a false positive.

Separately, the Stage 1 and Stage 2 evaluators implemented different matching rules, so the paper's direct Stage 1 vs. Stage 2 comparison mixed protocols. At this tag both route through the same core, greedy_match() in rampnet/metrics.py.

Corrected results — Stage 2, 1,000-panorama gold set, flip TTA, conf ≥ 0.55

Metric	As published	Corrected
Precision	0.938	0.949
Recall	0.935	0.873
AP	0.9236¹	0.9205²

¹ Uninterpolated AP under the paper's matching protocol. ² Interpolated AP under one-to-one matching — the conventions differ, so compare with care.

Swapping only the matching rule on identical model outputs moves precision by −1.0 points and recall by −4.4 points; the remaining difference is reproduction drift (environment, JPEG re-encoding).

The comparison with prior work is unaffected — both systems were scored under the same protocol, and the gap is far larger than the correction.

conf ≥ 0.55 above is the paper's operating point, held fixed so the two columns are comparable. It is not a recommendation: later work on main moved the recommended threshold to 0.30 (+7.7 recall / −4.8 precision; issues #54 and #55). The operating-point guidance in the v1.0-iccv2025 release notes (P=0.938 / R=0.935 at 0.55) is the as-published pair, superseded by the table above.

What is not corrected here

Stage 1 dataset-agreement precision (94.0%) has not been re-measured. The code is fixed at this tag; the number is not, because re-running it needs the generated Stage 1 dataset, which is not in git. When it is re-run, two changes will move it, not one:

1. redundant points now count as false positives — lowers precision; and
2. matching is nearest-unclaimed rather than first-in-list-order — can raise it, by assigning points to ramps more sensibly and leaving fewer false positives.

They pull in opposite directions, so the delta from 0.9403 must not be attributed to redundancy alone. Recall (0.9245) is affected by (2) as well, since nearest-unclaimed changes which ground-truth points get claimed. Tracked in #18 and #33.

Reproducing these numbers

They are a property of three things, not one:

Code	this tag — ee42044
Weights	projectsidewalk/rampnet-model at revision v1.0-paper (1078bcd6) — the Hub tag for the paper-state weights, i.e. what this repo's v1.0-iccv2025 tag corresponds to on the Hub. The two tags refer to the same weights — the repository and the Hub were tagged separately, so the names differ. There is one set of paper weights, not two.
Gold set	the 1,000 panoramas listed in manual_labels/ (YOLO-format points; the images themselves live in the HF dataset)

The Hub's main (606a1195, the July 2026 re-publish) reproduces this table too: it is the same weights repackaged, not a retrain. Its model.safetensors differs from v1.0-paper by exactly 2,312 bytes — 380 tensor keys × 6 characters of model. prefix, plus 32 bytes of {"format": "pt"} metadata — with the same 380 tensors at the same dtypes, shapes and byte offsets. The 360,202,244-byte tensor payload is bit-identical on both revisions:

sha256(tensor payload) = ea8c148eee227b32bf69ab2a2da0bc937b28405e46397d6cf38f1397e512132a

The dataset needs no revision pin: projectsidewalk/rampnet-dataset has one branch, main, whose last commit is 2025-08-15, and no tags — so it is paper state by construction.

Committed artifacts:

• stage_two/evaluation_results_new/ — the corrected metrics, PR curves and CSVs behind the table above
• stage_two/evaluation_results/ — the as-published results, unchanged

Full analysis — including the exact published code, executable traces, and visual examples of double-counted detections — is in docs/eval_protocol_verification.html on main (open it in a browser) and in issue #9. It is linked at main rather than at this tag because it is a maintained explainer, not part of the frozen artifact; the tag's copy is the same analysis with an earlier header.

Which tag do I want?

• Reproducing the paper exactly as written → v1.0-iccv2025
• Using RampNet's model with a standard evaluation protocol → this release
• Building on RampNet → main, which has moved well past both

Citation

Unchanged — cite the paper:

@inproceedings{omeara2025rampnet,
  author    = {John S. O'Meara and Jared Hwang and Zeyu Wang and Michael Saugstad and Jon E. Froehlich},
  title     = {{RampNet: A Two-Stage Pipeline for Bootstrapping Curb Ramp Detection in Streetscape Images from Open Government Metadata}},
  booktitle = {{ICCV'25 Workshop on Vision Foundation Models and Generative AI for Accessibility: Challenges and Opportunities (ICCV 2025 Workshop)}},
  year      = {2025},
  doi       = {https://doi.org/10.48550/arXiv.2508.09415},
  url       = {https://cv4a11y.github.io/ICCV2025/index.html},
  note      = {DOI: forthcoming}
}