OSM change detection: incorporate OSM changes and closures in conflation

[njhenry]

Overview

The current codebase only conflates between Overture Maps and currently open OpenStreetMap points of interest; it does not consider OSM POIs that have recently closed. However, the latter is useful information that can improve the quality of our combined dataset.

This PR uses the OSM history file to develop a list of POI closures and substantial changes. These changes are then compared to the Overture Maps POIs. Overture POIs with that match a previously-changed OSM POI on location, type, and name have their confidence downweighted.

Changes

Adds an OSM-history-derived change-detection layer to the conflation pipeline. For each unmatched-Overture POI in the conflated dataset, we shadow-match against a "ghost" — a previous state of an OSM element we believe no longer reflects ground truth (deletion / lifecycle-prefix / primary-tag-removal / substantial rename). Matches get their conf_mean multiplied by the per-shared_label δ from the fitted turnover model, with audit columns appended so the demoted rows can be inspected.

make conflate is now the canonical entry point for national runs and includes the CD penalty automatically. The pre-CD conflated parquet is preserved as conflated_baseline.parquet for spot-checks and ablation.

What's in the diff

~2,730 line-changes across 14 files, plus a Makefile target and docs.

File	Role
src/openpois/conflation/ghost_osm.py (new)	Flat-scan _scan_all_changes over osm_changes.parquet; emits at most one ghost per (element, version) of type hard_delete, lifecycle_prefix_added, primary_tag_deleted, or substantial_rename. Nodes only.
src/openpois/conflation/change_detection.py (new)	find_shadow_matches (BallTree + composite scoring), apply_current_survivor_filter (R1 — drops penalties when a live OSM POI with the same name lives within 50 m), apply_shadow_match (orchestrator + audit columns + δ penalty).
src/openpois/io/osm_history_pbf.py	Two-pass filter (osmium tags-filter → ID list → osmium getid --with-history) so the parsed parquet retains deletion versions. Recovers ~600 k hard_delete events nationwide that a single tags-filter pass silently drops.
scripts/conflation/build_ghosts.py, apply_change_detection.py, diff_change_detection.py (all new)	Driver scripts.
scripts/conflation/conflate.py	Adds --output-suffix so the no-CD baseline and the CD-applied canonical output can coexist.
Makefile	New conflate / build_ghosts / conflate_baseline / apply_cd targets. Each tees a per-run log under ~/data/openpois/logs/.
vetting_viz/ (new)	Single-page Leaflet app for hand-vetting demoted POIs (load CSV → click points → tag → export CSV). Static HTML/JS served via python -m http.server. Includes a Seattle evaluation script.
config.yaml	New directories.ghost_osm, versions.ghost_osm, and conflation.change_detection block (min_shadow_match_score, default_delta, min_prior_name_match_score, suppress_if_current_survivor).
docs/change-detection.md (new)	Architecture, stage-by-stage walkthrough, tunables, known limits. Referenced from .claude/CLAUDE.md.
.claude/skills/conflate-snapshots/SKILL.md	Step 4 updated to point at make conflate and list the new sub-targets and outputs.
.claude/TODO.md	Two new entries: per-region δ calibration; DuckDB ST_Distance_Sphere correctness bug.

How a national run looks now

conda activate openpois

# Refresh OSM/Overture snapshots as before (full-data-pull skill).
# Then a single make target runs the full conflation pipeline:

make conflate            # full CONUS, ~22M POIs
make conflate TEST=1     # Seattle bbox dry run

# Downstream steps (summarize, format_for_upload, prepare_pmtiles,
# upload_to_source_coop) are unchanged — they read conflated.parquet
# which now has the CD penalty applied.

Sub-targets (make build_ghosts, make conflate_baseline, make apply_cd) are exposed for partial re-runs when iterating on a single stage.

Validation

Hand-vetted 290-row Seattle ground-truth set:

	Baseline (no CD)	With CD
Demoted Overture rows	0	293
Vetted true-drops captured	—	221
Vetted false-drops still penalized	—	58
Precision (vs vetted truth)	—	79.2 %

The remaining ~20 % FPR is intrinsic to the design — OSM history captures closures but is silent on openings, so a real closure plus a different current business with a similar name + location + type reads as evidence Overture is stale.

Performance

Stage	Wall time (CONUS)	Peak RSS
build_ghosts	~30 s	< 1 GB
conflate_baseline	unchanged from today	~10 GB
apply_cd	~90-130 s (projected)	~3-5 GB

The R1 current-OSM-survivor filter uses a sklearn BallTree haversine query over rated-snapshot centroids extracted via DuckDB SQL (no shapely-Python materialization). This replaced a DuckDB cross-product spatial join that was ~25 s wall and ~8.5 GB peak on Seattle alone; the new path is 5-6 s wall and 1 GB on Seattle, and scales linearly to nationwide.

Notable findings flagged during development

1. DuckDB v1.4.1 ST_Distance_Sphere returns incorrect distances — the bundled spherical-distance formula is off by ~25 % at continental scale (NYC → LA returns 4,911 km vs the correct ~3,940 km) and similarly at small scales (a 65 m Seattle pair reads as 43 m). The R1 filter avoids this by going through BallTree haversine. Any new code touching this area should avoid ST_Distance_Sphere until the pin is bumped. Tracked in .claude/TODO.md.
2. R2 and R3 suppression rules were tried and stripped. Earlier branch history experimented with "reverted-rename" and "same-changeset way-creation" suppression rules. On the Seattle vetting set R2 fired only twice and R3 had near-random precision (sometimes net-negative). Both were removed; only R1 (current-OSM-survivor) survived. Easy to revisit if a future dataset suggests them.
3. min_prior_name_match_score ships at 0 (loose matcher). Setting it to 70 produces high-precision but narrow-recall results that miss the bulk of real closures where Overture has updated to a different current business name at a churned address (Sleep Train → Roosevelt Square type cases). The knob remains in config + CLI for future data-quality-only modes. The docstring on find_shadow_matches records the reasoning.