Low-floor operating-point sweep: recommend lowering the peak threshold 0.55 → 0.30 (#54, #55)

[jonfroehlich]

Closes #54. Closes #55.

Draws the precision/recall curve below RampNet's deployed 0.55 peak threshold — which had never been characterised — across all seven benchmark splits, completes the #55 GT-completeness correction on every city split, and recommends a deployment operating point.

Recommendation: 0.55 → 0.30, uniform

Pooled over the five US/VA splits (n = 609 panos), with #55's correction applied:

	at 0.55 (deployed)	at 0.30
precision	0.965	0.917 corrected (0.897 raw, band high 0.928)
recall	0.744	0.821 corrected
F1	0.840	0.866 corrected
detections/pano	1.85	2.23

+7.7 recall points for −4.8 precision points. Recall is RampNet's weak metric on every split; precision is not.

The correction changed the answer

On raw numbers F1 peaked at 0.32 and the conservative pick was 0.35. With the GT-completeness correction applied, corrected F1 peaks at 0.30, where corrected precision (0.917) exceeds the raw precision at 0.35. That is exactly what #55 existed to determine.

@JonF tagged the four splits that had never been spot-checked, so all six now carry a verdict on every incremental false positive — an unmatched prediction scoring in [0.25, 0.55), i.e. one that only a lowered threshold surfaces:

• A — a real curb ramp the ground truth missed. The model was right and the GT was incomplete, so this should count as a true positive.
• B — a genuine false positive (driveway apron, crosswalk paint, stairs, concrete patch, hood, seam artifact). The model was wrong.
• unsure — not callable from the imagery. Abstains from the corrected figure and widens the reported band instead of forcing a guess.

The A-rate is A over the total, i.e. what share of the apparent new false positives were real ramps all along — the correction factor on low-threshold precision.

split	items	A	B	unsure	A-rate
richmond	29	5	22	2	17.2%
bend	24	7	15	2	29.2%
clovis	23	7	13	3	30.4%
morgantown	30	4	25	1	13.3%
annapolis	27	6	15	6	22.2%
budapest_district5	89	23	59	7	25.8%

Pooled, 26.9% of the incremental false positives in [0.30, 0.55) were real curb ramps the ground truth had missed.

Findings worth reading even if you skip the rest

The 0.92 minimum does not gate AI label submission. AiController.submitAiLabel checks an internal API key plus the per-city boolean ai-label-submission-enabled — there is no accuracy threshold on that path. ai-validation-min-accuracy belongs to the DINOv2 validator subsystem. And label_ai_info persists each label's confidence, so a lower floor is reversible server-side. The constraint is a policy call about review burden, not a gate.

Sub-0.55 precision is a lower bound, by construction. Decomposing every TP by the origin of the GT point it matched: below the review floor, every true positive comes from a reviewer missed-mark and none from a reviewed detection, on every split. The city GT was built from ≥0.55 detections, so a real ramp nobody flagged is scored as a false positive. low_floor_sweep.py gtbias reproduces this.

The A-rate does not track imagery quality. Morgantown — the cleanest split in the benchmark — has the lowest A-rate (13%) and clovis, the softest, the highest (30%). So no cross-split GT-completeness constant is quoted; the correction is applied per split.

Clovis is the cheapest place to lower the threshold, not the binding constraint. It gains +10.3 recall points, the largest of any US split.

Not per-tier. Tier optima cluster at 0.32–0.35 across a 2018 GoPro Fusion and a survey-grade Trimble MX7. Only GSV/bend differs (0.50) and it is confounded three ways — one split, in-domain, and the resample caveat below. A per-tier policy would fit noise.

This is not recall-by-carpet. Density rises 1.85 → 2.23 detections/pano, leaving RampNet 25–40× sparser than the open detectors at their operating points (docs/model_comparison.md's null-recall finding).

The recall gain stacks with multi-view. It is uniform across distance bands (+0.071 near / +0.078 mid / +0.072 far), so it does not overlap with what labeler#27 would recover.

Verification

Parity gate — peaks at ≥0.55 must reproduce each split's committed records.jsonl. Measured in match radii, since bit-exactness is the wrong bar:

split	identical cell	within 0.5 R	max
richmond / clovis / morgantown / annapolis / budapest	100%	100%	0.000 R
bend	79.0%	98.4%	0.439 R
manual_gold †	80.9%	99.9%	0.472 R

Every Mapillary split reproduces bit-exactly, on different hardware. Bend is the sole exception for a structural reason: it is the only GSV split, and the GSV production path built a 4096×2048 intermediate, so production fed the model a different resample than the native-res bundle does. This was predicted before the four remaining splits ran, and held. (Consequence: bend's GT points and predictions come from different resamples, making its numbers mildly pessimistic.)

† manual_gold is deliberately not gated — its committed detections used flip-TTA, so its row is a TTA delta. Free data point for #78: TTA yields 3.5% more detections at ≥0.55, 99.9% co-located.

Re-extraction is lossless. richmond re-extracted on an L40S reproduced the earlier cache with 0 of 537 peaks moved (max confidence delta 6e-05), and all 222 committed #55 tags across six splits still resolve (low_floor_sweep.py tagcheck, which exists because tag ids are keyed to peak coordinates and a re-extraction could silently orphan reviewer work).

What is in here

• scripts/analysis/low_floor_sweep.py — parity | sweep | hist | gtbias | distance | corrected | tagcheck, all CPU-only off the committed caches
• scripts/analysis/run_low_floor_extract.slurm — the one GPU step (one L40S, 41 min for 1,625 panos)
• operating_point_curve.py — extract now reads both bundle kinds, so manual_gold (independent YOLO GT, no verdict review) joins the sweep; per-split skip makes a preempted job resumable
• docs/operating_point.md — the full analysis
• analysis_out/op_cache/*.json — 780 KB of image-free detections, committed so every number re-derives without a GPU
• 44 new unit tests (222 total, green)

Honest limits

• Tuned on the benchmark — no separate validation split, so 0.30 is optimistic. Mitigated by the F1 curve being flat (0.849–0.857 across 0.25–0.55): the choice is robust even where the argmax is not.
• The correction rests on one rater, who also produced the GT.
• Clovis is the split to watch — corrected precision 0.883 at 0.30 against 0.914 deployed, the only US split giving up meaningfully more than the others.
• Budapest is excluded from the recommendation (single-rater low-confidence GT) and needs its own decision.
• A claim that the anchoring bias "runs in both directions" was removed during analysis: the aggregate check did not support it (sub-0.50 precision 0.195 anchored vs 0.208 un-anchored). docs/operating_point.md reports that tension rather than the stronger claim.

Hand-offs

• labeler#20 applies the constant; nothing in this PR touches labeler code.
• labeler#27 stage 4 gets the calibration histograms — see the doc's final section. The measured single-view crossover is ≈0.40–0.45 on both anchored and un-anchored GT; the promotion floor should sit below it, but by how much is not answerable from 30 m-thinned per-pano data.

🤖 Generated with Claude Code (claude-opus-5[1m])

[jonfroehlich]

Both additions are in — and (1) comes out against the current floor

Pushed as c4921e7. New subcommand low_floor_sweep.py floor + plot_storage_floor.py, both CPU-only off the committed caches. 231 tests green.

One implementation note: greedy matching is highest-confidence-first, so dropping low-confidence predictions can never change what a higher one matched. That means a single pass yields, per GT ramp, the confidence of its best candidate — and every floor's answer falls out of that one vector. There's a test asserting recall_at[f] equals the swept recall at f, so the two paths can't silently drift.

(1) GT-matched counts by band — this falsifies DETECTION_STORAGE_FLOOR = 0.1

Ramps whose best candidate lands in each band, i.e. what a 0.1 floor makes permanently unrecoverable:

split	GT	best in [0.05, 0.10)	share	best in [0.10, 0.20)
richmond	310	8	2.58%	5
bend	327	8	2.45%	5
clovis	195	6	3.08%	5
morgantown	267	7	2.62%	12
annapolis	294	9	3.06%	10
POOLED (5 US)	1393	38	2.73%	37
budapest*	300	7	2.33%	18
manual_gold†	3919	19	0.48%	48

The decisive fact isn't the 2.73% — it's that the top-50 cap never binds. At a 0.1 floor the busiest pano in the entire benchmark holds 14 candidates against a cap of 50 (medians 2–5). At a 0.05 floor the maximum is still 14.

So the cap is not the volume bound labeler#28's design assumed. The floor is, and it's costing ~2.7% of findable ramps for a saving of roughly one extra candidate per pano. labeler#28's own argument — storing too little is irreversible, storing too much only costs disk, and the cap bounds it regardless — is correct and points at 0.05, not 0.1.

Recommendation: lower DETECTION_STORAGE_FLOOR to 0.05. Labeler-side change; recorded here because this is where the measurement lives.

(2) Recall-at-floor — the ceiling on stage-4 consensus

	pooled (5 US)
deployed 0.55	0.744
recommended 0.30	0.818
ceiling at the 0.10 storage floor	0.872
ceiling at the 0.05 extraction floor	0.899

labeler#27 stage 4 has +12.8 recall points of headroom over the deployed threshold, +5.4 over the 0.30 recommended here — but it is capped at 0.872 by storage, not by the 0.899 the model actually produces. Whatever k consensus requires, it cannot promote a candidate that was never written.

Figure

Two panels rather than one, because they're different questions and a shared axis would misrepresent both. Two things I changed after rendering and looking at it: the left panel plots a share of each split's GT, not raw counts (manual_gold has 3,919 GT ramps against ~300 for a city split — counts made its bar tower while its rate is in fact the lowest of any split, which is exactly backwards), and the legends moved below the axes where they were colliding with the POOLED bars. Palettes were run through the validator, not eyeballed.

On checking in analysis_out/

Already partly done in c7098be — the seven detection caches (780 KB, image-free) were committed there, which is what makes every number re-derivable on CPU. This commit adds the derived result tables (~80 KB: the swept grid, calibration, corrected P/R, distance, storage floor), so any figure quoted in the prose can be checked against the table it came from.

Deliberately still ignored: the 181 MB of gallery crops. They regenerate from the caches plus the panos, and their irreplaceable part — @JonF's A/B tags — is already committed under benchmark/<city>/incremental_fp_tags.json. Happy to include them if you'd rather, but 181 MB of regenerable PNGs in git seemed like the wrong trade.

.gitignore needed restructuring rather than a negation, incidentally: git can't re-include a file whose parent directory is excluded, so the obvious analysis_out/ + !analysis_out/op_cache/*.json is silently a no-op.

*budapest GT is single-rater, low confidence. †manual_gold is in-distribution GSV with independent GT.

🤖 Generated with Claude Code (claude-opus-5[1m])

[jonfroehlich]

Added the PR-response figure — you're right, that's the headline artifact

Pushed as c6fc89f. I'd built the storage-floor figure for the review question and skipped the one #54 actually asks for ("the full PR curve (precision & recall vs. threshold) … with the deployed 0.55 point marked"). scripts/analysis/plot_operating_point.py, CPU-only off the committed caches.

Left — the PR curve, one line per split, threshold swept from 0.05 to 0.90. Hollow marker = deployed 0.55, filled = recommended 0.30, on every curve. The shape is the argument: the curves run almost flat across the operating region, so recall is bought at a shallow precision cost right up until they turn down hard past ~0.85 recall. The two operating points sit on the flat part on every split.

Right — F1 vs threshold, and it's there to make a negative point. The pooled curve varies by under 0.01 across 0.25–0.55. So "F1-optimal" isn't a sufficient reason to pick any number, and the 0.30 recommendation rests on the recall-first policy, the density budget and reversibility rather than on the argmax. The per-split optima (dots) cluster tightly while the levels differ a lot — the same pattern as the per-tier table.

Both panels colour by split consistently, so a line is the same entity in each. Held-out splits (budapest, manual_gold) are dashed, so line style carries the held-out status and colour is free to carry identity alone.

One design note worth recording

I tried per-line direct labels first — three of the seven palette slots sit below 3:1 on a light surface, so the relief rule demands visible labels or a table view. They were illegible: in the PR panel the splits converge into a cluster around (0.80, 0.88) and seven labels can't be placed there without overlapping. Screenshotting and looking at it is what caught it. Switched to a shared legend plus the committed low_floor_sweep.csv as the table view, which discharges the same requirement without the collision.

Same class of fix as the storage-floor figure, where rendering it revealed that raw counts made manual_gold tower on the strength of having 3,919 GT ramps.

Both figures are now in docs/operating_point.md — the PR curve leads the Results section, the storage-floor one sits with the labeler#28 analysis.

🤖 Generated with Claude Code (claude-opus-5[1m])

[jonfroehlich]

Added a city-onboarding protocol — and everything is checked in

Two things in this push.

1. Everything is committed and pushed

Working tree clean, HEAD == origin, 231 tests green, 10 commits on the branch. Verified rather than assumed:

• 7 detection caches (analysis_out/op_cache/*.json, 780 KB image-free) — every number re-derives on CPU
• 9 derived result tables (analysis_out/op/*.csv|json) — so a figure quoted in prose can be checked against its source table
• 6 incremental_fp_tags.json — the irreplaceable human A/B pass
• 2 figures, 4 scripts, 1 Slurm launcher, 43 new tests, 3 docs

I also found a gap while checking: scripts/analysis/README.md indexes every analysis script (PR #65 was literally adding a row to it) and this branch had added eleven entry points without touching it. Fixed, plus two intro statements the branch had made wrong — it is no longer "benchmark/{richmond,bend}" and analysis_out/ is no longer entirely git-ignored.

Deliberately still ignored: the 181 MB of gallery crops. Regenerable, and the human tags are committed.

2. docs/adding_a_benchmark_city.md

benchmark/README.md had a five-row table for adding a city that stops at "add the split to the HF dataset." That was right when the benchmark was two splits and one scorer. It is now badly incomplete — a new split silently invalidates the pooled numbers, the per-tier curves, the #55 correction, both figures and four documents.

The protocol covers phases 0–6 with a checklist to paste into the PR. Three parts worth calling out:

Phase 0 — decide what the city is for before spending the review. Every existing split answers a question the others cannot; one that duplicates an existing answer costs a day of review and buys a table row. On that basis the two highest-value additions right now are a second GSV city (to disentangle bend's GSV-vs-in-domain confound) and a second rater on budapest — which is a review, not a city.

Phase 4 — the step that gets skipped. Extraction → the parity gate → the five sweep commands → #55 tagging at op-threshold 0.25 (matching every existing split, or the A-rates aren't comparable) → tagcheck → regenerate figures.

Phase 5 — the code touchpoints that fail silently. US_SPLITS membership is what puts a split into the pooled recommendation; a missing tier_of() branch dumps a new rig into unknown; the figure's SERIES needs a palette colour in slot order. None of these raise — they quietly omit or mislabel the city.

It also records the traps with their evidence: never review above model resolution (the error that forced re-reviewing richmond and bend, correcting recall 0.895 → 0.765 and 0.831 → 0.758), don't pool GT you don't trust, and panos/ is git-ignored and irreplaceable if Mapillary decays.

Every code reference was checked against the source rather than written from memory — including build_benchmark_dataset.py, still hardcoded to bend + richmond, flagged in place as the known HF-dataset lag (#21).

Linked from benchmark/README.md and docs/operating_point.md.

🤖 Generated with Claude Code (claude-opus-5[1m])