Extend RampNet output from points to boxes/polygons (point-prompted SAM2; CenterNet-style size head; segmentation)

[jonfroehlich]

Context

RampNet's output is points — by design, because Stage 1 translates government GPS locations into pixel points, and the model regresses fixed-sigma Gaussians (sigma 10 on the 512×1024 heatmap) around them. Extent (bounding box / polygon) is nowhere in the supervision, so the predicted blob size reflects target sigma and confidence, not ramp size. Thresholding the current heatmaps cannot recover object extent, and nothing in the 850k-label training set can teach it.

But extent would be valuable: it upgrades the output from "where" to "where and how big," enables area/geometry-based condition features (RampNet-2.0-adjacent), and makes RampNet directly comparable to box-native baselines (#51) on their own terms.

Proposed paths (increasing effort)

1. Point-prompted SAM2 — boxes and polygons with zero training

RampNet's high-precision points (P 0.949 at the 0.55 operating point) are exactly what SAM-family models accept as prompts:

• RampNet detects a point on the pano
• crop a local region around it at full pano resolution (prompting on a crop sidesteps equirectangular distortion and the small-object problem)
• SAM2 with the point prompt → instance mask → polygon; tight bbox for free

RampNet stays the detector; SAM adds extent. This also doubles as a label-minting machine: run it over the ~850k Stage 1 training points to generate pseudo-boxes/masks at scale for paths 2–3.

Main risk: segmentation quality on low-contrast ramps that blend into surrounding concrete. This is an empirical question, and we can answer it cheaply — see "First experiment" below.

2. CenterNet-style size head on the existing architecture

Keep the ConvNeXt V2 backbone and center heatmap; add two regression channels (w, h at the peak). Minimal delta to native boxes, preserves the one-pass story. Needs box supervision at scale — which path 1 mints (SAM pseudo-boxes, filtered by mask quality). The 1k gold boxes are too few to train on and are the benchmark; they stay eval-only.

3. Full segmentation output

If polygons become first-class: Mapillary Vistas includes a curb-cut class with polygon annotations (coverage/quality to be verified before committing) — a fine-tune source that dovetails with the Mapillary direction (#25, #48) and sidewalk-auto-labeler. SAM pseudo-masks scale that supervision the same way Stage 1 scaled points.

First experiment (cheap, decisive)

manual_labels/ is YOLO-format boxes — 3,919 of them on the 1,000 gold panos — which we have only ever consumed as centers. Run point→SAM2 on the gold panos (prompting from GT centers to isolate segmentation quality from detection quality, then from RampNet detections for the end-to-end number) and report the IoU distribution against the gold boxes. One day of GPU time; the IoU histogram tells us whether path 1 is production-grade or needs work, and is a committable result either way (per the repo's negative-results rule).

Related

• Train a supervised YOLO baseline (YOLO11 / YOLO26) on the RampNet dataset — isolate architecture vs. data #51 / Rerun the collapsed YOLOv11 pano baselines with a stabilized (larger physical batch) config #70 — supervised YOLO baselines (box-native comparison this would enable on equal terms)
• Experiment: hybrid RampNet + open VLM (Qwen3-VL) — ensemble / cascade / distillation #35 — hybrid RampNet + foundation-model cascades (same "RampNet as prompter" shape)
• Multi-capture / multi-view fusion for per-ramp recall: correspondence, agreement-as-arbiter, 3D (RampNet 2.0) #48 — multi-view fusion / RampNet 2.0 (extent + geometry feeds correspondence)
• Experiment: curb-ramp detection performance vs. input resolution (Mapillary high-res) #25 — Mapillary high-res experiments (path 3 supervision source)
• Set up a harness for experimenting with newer detection models/backbones #20 — newer-model harness (where a size-head variant would be trained/evaluated)

🤖 Generated with Claude Code (claude-fable-5)

[jonfroehlich]

Cross-link: paths 2 and 3 depend on #84 (path 1 doesn't)

Filed #84 today after finding that Stage 2 trains for exactly 1 epoch — 9,379 optimizer steps at a constant lr of 1e-5 with no LR scheduler at all (train.py:35, :269; global batch 16 from run_train.slurm's 4 nodes × 4 GPUs). For scale, COCO detection's shortest standard schedule is ~88k steps, and the YOLO baseline in #51 is getting ~2.25M. #84 measures the epoch curve to find out whether that budget is adequate.

That lands squarely on this issue, but only on part of it:

Path 1 (point-prompted SAM2) is unaffected — no Stage 2 retraining, so the first experiment here is independent and can proceed regardless. It stays the cheap decisive step.

Path 2 (CenterNet-style size head) is directly downstream, in two ways:

1. The budget question is sharper for a size head than for the current point head. The w/h regression channels are new, randomly initialised, and trained from scratch — unlike the existing head, which rides a pretrained ConvNeXtV2 backbone at a fine-tuning LR. New heads generally need more steps than a fine-tune does, so a 9.4k-step budget that might be adequate for the point head is even less likely to be adequate for a size head. Stage 2 trains for 1 epoch at constant LR (~9.4k steps): measure the epoch curve, and with it the auto-label ceiling #84's curve sizes it.
2. The comparison baseline moves. "Minimal delta to native boxes" has to be measured against a properly trained point-only model. If Stage 2 trains for 1 epoch at constant LR (~9.4k steps): measure the epoch curve, and with it the auto-label ceiling #84 shows the current recipe is undertrained, the released checkpoint is the wrong baseline for judging whether a size head costs point accuracy, and any such measurement taken before Stage 2 trains for 1 epoch at constant LR (~9.4k steps): measure the epoch curve, and with it the auto-label ceiling #84 would need redoing.

Path 3 (full segmentation fine-tune) inherits the same dependency, being another Stage 2 retrain.

The practical point: if a Stage 2 retrain is happening for path 2 or 3 anyway, running it at the epoch count and schedule #84 selects avoids paying for the training twice. #84's Run A is ~8 epochs and produces per-epoch checkpoints (the code already saves them), so it yields the point-head curve that any size-head variant has to beat.

One more thing worth knowing before path 2's supervision design: best_model.pth is currently selected on validation heatmap loss against the auto-generated labels — a proxy loss, not a detection metric, and machine labels rather than human ones. #84 also checks whether that rule picks the right checkpoint. If path 2 adds a size loss, the selection criterion becomes a live design question rather than an inherited default, since the combined loss would then mix two objectives of different scales.

Related: #82 (training augmentation) is also downstream of #84 for the same retraining reason.

🤖 Generated with Claude Code (claude-opus-5[1m])