feat(overture): add Overture Maps POI integration (Places, off by default)#73
Merged
Conversation
…ma DDL Foundation for the optional Overture Maps integration (off by default): - packages/core conflate() bipartite matcher (union-find + tiered distance windows + Sorensen-Dice name gate), reusing geo-server helpers; provisional default thresholds pending precision calibration. - overtureCategoryMap: first-pass Overture-leaf to OpenMapX CategoryId map for the commercial category subset. - data-manager overture_places schema DDL + assertValidOvertureSchema guard (CDLA<->ODbL theme isolation + strict identifier validation); h3_r8 stored as TEXT; DuckDB static CLI added to the Dockerfile; h3-js dependency.
Spike data foundation for the optional Overture integration (operator-run, off the request path): - data-manager jobs: pullOverture (DuckDB-over-S3 bbox extract), ingestOverture (GeoParquet to staging schema, atomic swap, batched h3-js + category backfill), extractOsmPois (osmium tags-filter to osm_pois with OSM-tag-derived CategoryId, way/relation centroids). - eval harness: pure precision/recall/F1 metrics + 54-cell threshold sweep, candidate generation (150 m, dice-band stratified), and a run.ts that dumps candidates.tsv for human labeling then scores the real conflate() over the grid. - osm_pois DDL added to the schema module.
Adds the `poi-overture` integration (off by default via `enabled: false`) and wires the poi-search orchestrator to fan-out across Overpass + Overture for commercial categories, fusing results with `fusePoiResults` via spatial + name conflation. - Adds `gersId?: string` to `PoiSearchResult` and brand/operator/brand:wikidata to `FILTERABLE_TAG_KEYS` so matched pairs carry attribution end-to-end. - Adds `fusePoiResults(osm, overture, thresholds, link?)` to the conflation engine: OSM wins on name/coords/hours/phone; Overture fills gaps + always supplies brand tags; Overture-only results append as gap-fill. - Extracts the `for (attempt...)` shrink-retry into a module-level `runWithShrink(fn, bbox)` helper; `fn` now receives `currentBbox` on each retry (restoring the shrink behavior). Single-provider short-circuit keeps default-deployment behavior byte-identical (golden test: deep-equal). - Adds `OVERTURE_COMMERCIAL_CATEGORIES` and `openmapxCategoryToOvertureLeaves` to the category map; OSM-only categories (viewpoints, drinking_water, …) are excluded so they always route to the single Overpass provider. - Adds `integrations/poi-overture` with PostGIS-backed provider, manifest (requires postgis, healthCheck custom), and legal strings for overture + foursquare data sources (CDLA-Permissive-2.0 / Apache-2.0). - Adds golden + fusion tests proving optionality guarantee and fan-out.
The provider's queryOverturePlaces() referenced columns that don't exist in the overture_places.places table (id, primary_name, longitude, latitude as scalar columns, brand_name/brand_wikidata as flat columns, operator). Fix to match the actual DDL (schema.ts): - SELECT gers_id, name, ST_X(geom)/ST_Y(geom), openmapx_category, brand->>'name', brand->>'wikidata', phones[1] - WHERE geom && ST_MakeEnvelope(west, south, east, north, 4326) to use the GIST index on geom Fixtures in provider.test.ts updated to match.
Registers a new optional knowledge source that matches a clicked OSM place to its Overture Maps twin and fills empty brand, multilingual names, and structured opening hours — with zero change when the provider is absent (enabled: false by default, requires postgis service dep). Contract changes (all fields strictly optional): - KnowledgeResult += brand?, names?, structuredOpeningHours? - KnowledgeContext += ids? (threads place.ids for link-first GERS lookup) - Place += brand?, names?, structuredOpeningHours? (surfaced by ...knowledge spread) Orchestrator (knowledge/index.ts): passes place.ids into context; merges new scalar fields with first-non-null semantics matching existing fields. Provider (integrations/knowledge-overture): - resolveGers(): link-first via poi_conflation_link (empty until plan 03, falls through to spatial path as expected), then ST_DWithin 150 m + diceSimilarity >= 0.8 name match; tolerates missing context.ids - withDeadline(1500, …) bounds per-lookup latency - overtureRowToKnowledgeResult(): maps brand/names/opening_hours columns; routes brand.wikidata into externalIds.wikidata for the existing chain - Both ST_DWithin sides cast ::geography so 150 is in metres not degrees
Implements Steps 3 and 4 of Plan 03 (offline conflation backbone):
- embeddings.ts: Ollama /api/embeddings client (mxbai-embed-large default),
SHA-256 content-hash cache backed by overture_places.embedding_cache,
injectable EmbeddingCache interface for testing, bounded concurrency (8),
cosineSimilarity helper, ensureEmbeddingModel with {model,stream:false} body.
- conflate.ts: computeLinks (H3-r8 blocking + core conflate cascade →
spatial-name links; in-window residual with embedFn → embedding links;
dedup by best confidence per OSM poi and per gers_id) and conflateOverture
job wrapper (loads places+osm_pois from sql pool, batch-upserts into
poi_conflation_link with param-capped batches: rowsPerBatch*6<=65500).
- schema.ts: adds buildEmbeddingCacheDDL + applyEmbeddingCacheTable.
- service.json: adds OLLAMA_URL env var (default http://local-ai:11434).
- 16 new tests (embeddings: cache hits/misses/partial, cosine correctness,
ensureModel body shape; conflate: spatial-name, embedding residual,
out-of-window guard, embedFn-omitted, dedup).
…s pulled - computeLinks: collect the unique OSM + Overture indices that actually appear in candidatePairs before calling embedFn. Only those texts are embedded (down from all-unmatched); vectors are mapped back via Map<originalIndex, vector> so cosine comparisons are unchanged. - conflateOverture: call ensureEmbeddingModel(DEFAULT_MODEL, ollamaUrl) before constructing embedFn so a fresh local-ai container pulls the model rather than crashing on the first 404. - embeddings: export DEFAULT_MODEL so conflate.ts can import it. - embeddings.test: add zero-vector guard case for cosineSimilarity.
…ngelog cron
Implements the operator plumbing for the Overture conflation backbone (Step 1 + Step 7
of the plan):
- POST /overture/{pull,ingest,conflate} routes in services/data-manager — NDJSON streaming,
hijacked reply, mirroring /download/osm
- DataManagerClient.{pullOverture,ingestOverture,conflateOverture} in packages/core with
a string-message progress callback (different from the bytes-progress of downloadOsm)
- data overture-{pull,ingest,conflate} [region] CLI commands in packages/cli
- changelog.ts: buildInsertSql/buildUpdateSql/buildDeleteSql (pure, testable SQL
builders keyed on gers_id); applyOvertureChangelog tries the S3 delta path and
falls back to full re-ingest automatically (the S3 partition path is unconfirmed)
- cron.ts: monthly Overture cron (OVERTURE_SYNC_CRON, default "0 5 1 * *"), gated
behind OVERTURE_ENABLED=true — default deployments register no cron; feeds into
feed_state with region="overture-places-<region>" and the staleness-alert pipeline
Tests: changelog SQL builders (4, no live S3); cron gating (4, sentinel + enabled guard)
…cron test stub - changelog.ts: count rows per change_type from parquet before applying the DuckDB delta; return real added/updated/removed counts on the success path and -1 sentinel on full-reingest fallback. Remove dead `?? OVERTURE_RELEASE` (release is non-optional). Drop unused OVERTURE_RELEASE import. - conflate.ts: add optional `onProgress` param to `conflateOverture`; emit progress at load (N places / M osm pois), link computation, and upsert. - api.ts: wire `onProgress` on the /overture/conflate route so the client receives progress events (mirrors /overture/pull and /overture/ingest). - overture-cron.test.ts: fix stub DB chain — drop spurious `.orderBy()` level so it matches the `.select().from().where().limit()` chain in writeFeedState. Add test that awaits runOvertureNow with overtureEnabled=true (asserts it resolves without throwing). Add test exercising OVERTURE_ENABLED env-var branch with env restored in finally.
Wire the precomputed poi_conflation_link table into the POI-search request
path so fused results use the offline link first (instant, higher quality)
before the query-time union-find. Absent link / no PostGIS / Overture
disabled → behavior is deep-equal to the prior union-find-only path.
- fusePoiResults: implement the reserved link? param as a first-pass over
OSM results keyed node/id → gers_id; link-fused pairs are consumed before
the union-find runs over the remaining results. Extracted fuseOsmOverturePair
helper so both paths apply identical fusion rules.
- orchestrator: buildConflationLinkMap batch-queries poi_conflation_link for
the full OSM result set in one query (no N+1), builds the Map, and passes
it to fusePoiResults. Guarded on ctx.db presence.
- manifest: add requires [{service:"postgis", optional:true}] so the host
injects ctx.db on deployments with PostGIS, leaves it undefined otherwise.
Optional dep → no healthCheck required; poi-search loads on DB-free installs.
…spatch Register the `overture:` place-id resolver in `poi-overture` `setup` (gated on `ctx.db`) so `overture:<gers>` deep-links resolve on Overture-enabled deployments. Adds `fetchOverturePlaceByGers` (single-row lookup by GERS id) and `overtureRowToPlace` (returns a full `Place` with `primaryScheme:"overture"`). Add `isEnabledIntegrationScheme` to `integration-host.ts`, keyed on `config.enabled` (not the runtime `enabled` flag), so the `/places/:id` 404 trap distinguishes config-disabled integrations from enabled-but-boot-failed ones. Swap the guard in `places.ts` from `isIntegrationScheme` to `isEnabledIntegrationScheme`: a config-disabled scheme with `?lat&lng&name` now coord-falls-back gracefully; an enabled-but-broken scheme still 404s. Tests: resolver suite (known GERS → Place, unknown GERS → null, no-db skips registration); places suite updated with mock + two new dispatch cases.
…spatch Register the `overture:` place-id resolver in `poi-overture` `setup` (gated on `ctx.db`) so `overture:<gers>` deep-links resolve on Overture-enabled deployments. Adds `fetchOverturePlaceByGers` (single-row lookup by GERS id) and `overtureRowToPlace` (returns a full `Place` with `primaryScheme:"overture"`). Add `isEnabledIntegrationScheme` to `integration-host.ts`, keyed on `config.enabled` (not the runtime `enabled` flag), so the `/places/:id` 404 trap distinguishes config-disabled integrations from enabled-but-boot-failed ones. Swap the guard in `places.ts` from `isIntegrationScheme` to `isEnabledIntegrationScheme`: a config-disabled scheme with `?lat&lng&name` now coord-falls-back gracefully; an enabled-but-broken scheme still 404s. Tests: resolver suite (known GERS → Place, unknown GERS → null, no-db skips registration); places suite updated with mock + two new dispatch cases.
…nd enrichment - conflate.ts: filter operating_status <> 'permanently_closed' AND (confidence IS NULL OR confidence >= 0.7) on the places load, matching the runtime provider predicate - knowledge-overture: add operating_status <> 'permanently_closed' to both branches of the spatial+name fallback query so closed venues don't surface on place-cards - knowledge-overture: clear the withDeadline timer after fn() resolves to avoid a dangling handle - poi-search orchestrator: remove the dead `key` field from OsmParsed (computed but never read)
…validate region Two fixes surfaced during Overture staging validation: - The data-manager Docker image (and CI on merge) failed to build: the Overture job code imports @openmapx/core, absent from the image's pruned workspace. Expose the pure conflation/geo/category/osm-filter utilities via @openmapx/core subpath exports (no integration/react drag), split fusePoiResults into poiFusion, add packages/core to the Dockerfile, and fix the DuckDB arm64 URL. Public API unchanged; full suite green; image build verified. - Security: the region parameter reached DuckDB SQL string literals and file paths unvalidated (ingest/extract). Add assertValidRegion (strict allowlist regex) in regionSlug + every Overture job entry point + the /overture API routes; pull.test.ts proves injection strings are rejected.
Staging validation against real Berlin Overture Places revealed the ingest SQL referenced longitude/latitude columns that do not exist — Overture exposes a single `geometry` (WKB) column. DuckDB has no SRID concept, so the staging `geom` column is now plain GEOMETRY and the ingest stamps SRID 4326 + the POINT typmod Postgres-side via ALTER ... USING ST_SetSRID(geom,4326) before the swap. Also drops the duplicate index creation (buildSchemaDDL already makes them). The DuckDB-over-S3 pull and the amd64 image build are confirmed working on the staging VPS; this unblocks the ingest into PostGIS.
…n list Continuing the real-data ingest validation on staging: a raw DuckDB->PostGIS geometry transfer is rejected by Postgres COPY (invalid geometry), so emit ST_AsHEXWKB(geometry) which COPY parses into the geom column. Also supply an explicit 20-column INSERT list (the table's 21st column, imported_at, defaults). Verified on real Berlin data: 171,938 places ingested, geom SRID 4326 with correct coordinates and names.
Staging validation found extractOsmPois unrunnable: pbfPath was a required option the routes/CLI never pass (osmium got 'undefined'), and it asked osmium tags-filter to emit geojson directly (rejected — tags-filter only writes OSM formats). Now derive the PBF path from the region, filter to a temp PBF, then osmium export -f geojson; drop the redundant nwr/name term (it OR-matched every named object, not just POI categories). Verified: produces a parseable FeatureCollection on the real Berlin PBF.
osmium export omits the OSM id by default, so every osm_pois upsert hit 'invalid input syntax for bigint: NaN'. Pass --add-unique-id=type_id (emits n<id>/w<id>/r<id>) and parse that single-char prefix; skip any feature without a numeric id. Verified on the real Berlin extract: ids like n172548.
generateCandidatePairs compared every OSM POI against every Overture place (O(n*m)) — fine on unit fixtures, ~9 billion ops on real Berlin (52k x 172k). Bucket Overture places by H3 r8 and compare each OSM POI only against its cell + ring-1 neighbors (r8 edge ~461m > the 150m radius).
…kfill stall The category map covered only ~120 Overture leaves; real Berlin ingest categorized just 0.9% of places. Two causes, both fixed: - Coverage: regenerate overtureCategoryMap.json by composing the cbeddow/overture2osm dictionary (Overture's ~2k categories → OSM tags) with this repo's CATEGORY_FILTERS (OSM tag → CategoryId), merging the curated synonym leaves on top. Now 635 leaves across 47 categories (568/2073 Overture categories bridged; offices/manufacturing/professional-services correctly stay unmapped). Adds doctors/dentists/hospitals/museums to the provider's commercial set (Overture covers them densely). - Backfill stall: the ingest's openmapx_category loop used a windowed 'WHERE openmapx_category IS NULL LIMIT n' scan that breaks when one window returns only unmatchable rows, abandoning matchable rows further in. Replace with a single full pass + chunked set-based UPDATE.
… duckdb errors - resolveOsmUrl: Geofabrik nests sub-country extracts under their parent, so 'europe/berlin' 404s. Add a region→Geofabrik-path alias (europe/berlin → europe/germany/berlin); only the upstream URL is rewritten, the local PBF filename stays keyed on the original region so extractOsmPois still finds it. - The Overture ingest/changelog DuckDB scripts ATTACH the database by its connection string, so a failing 'duckdb -c' echoed postgres://user:<password> into execa's error message and from there into logs / NDJSON error events. Route those calls through a runDuckDb wrapper that rethrows a redacted error (postgres://user:***@host); pull.ts is untouched (S3 only, no secret).
…dcoded maps Replaces two hand-maintained region tables (the download-osm Geofabrik-path alias and pull.ts REGION_BBOXES) that could never be complete. Region identifiers are now Geofabrik download paths (e.g. europe/germany/berlin), so resolveOsmUrl works for all ~555 Geofabrik regions with no list. The Overture pull bbox is derived from the region's published Geofabrik .poly boundary (computeBboxFromPoly + fetchRegionBbox) instead of hardcoded coordinates — any region, country or sub-country, resolves from its own boundary. Cron's default Overture region updated to the valid path europe/germany/berlin.
Correctness:
- Conflation pipeline was broken in the wired flow: ingest's schema swap
destroyed osm_pois, so conflate's SELECT FROM osm_pois threw. buildSchemaDDL
now creates osm_pois + embedding_cache; extractOsmPois is wired into the api
(/overture/extract), CLI (overture-extract), DataManagerClient, and the cron
changelog paths (best-effort, warns if the OSM PBF is absent); conflate calls
applyOsmPoisTable defensively.
- poi-overture search kept NULL-confidence rows out (NULL >= 0.7 is false) while
ingest/conflate keep them — aligned to (confidence IS NULL OR confidence >= N).
- Multi-provider fusion swallowed OverpassTimeoutError (lost the 422 area-too-
large signal) and dropped any non-overpass/overture provider's results; now
the overpass base propagates the timeout and all augment providers are fused.
- getPlaceKnowledge skipped enrichment for brandless Overture places (osmTags
undefined); now proceeds on coordinates and passes {} osmTags.
- conflate() collapsed multi-member clusters to one pair (duplicate pins); now
greedily pairs all mutually-matching members.
- changelog delta rows never backfilled h3_r8/openmapx_category; factored
backfillDerivedColumns out of ingest and call it from the delta path.
- representativePoint averaged longitude wrong across the antimeridian; circular
mean now. cosineSimilarity guards length mismatch.
Consistency/perf:
- conflateOverture uses DEFAULT_CONFLATION_THRESHOLDS (was a re-literal); CLI
overture region default -> europe/germany/berlin; embedding-link confidence
default 0.9 (matches spatial); conflate result-mapping via Map not find();
embedding residual candidate generation H3-blocked.
…ascade
The cheap cascade compared names with character-bigram Dice on raw strings —
case- and accent-sensitive, no tokenization — so 'PENNY' vs 'Penny' scored 0
and was dropped (a large share of the low link recall). Add normalizeName
(NFKD accent-strip + lowercase + punctuation/whitespace collapse) and
nameSimilarity = max(charDice, tokenDice) over normalized names, and use it in
shouldMatch and the knowledge-overture spatial match. Token-Dice also rescues
shared-distinctive-token cases ('U Lindauer Allee' vs 'U-Bahnhof Lindauer
Allee', reordered words) that raw char-Dice misses, while unrelated names and
bare-brand-vs-phrase stay below the floor. Two blank names no longer match.
The conflate candidate-load filter (confidence IS NULL OR confidence >= 0.7) excluded ~99.8% of Berlin Overture places (only ~278 of 161,975 loaded), capping links regardless of name matching. The precision sweep found 0.5 beats 0.7 in every top cell; lower the floor to 0.5 so the matcher sees the real candidate set.
Overture leaves operating_status unset on the vast majority of places (161,692 of 161,975 Berlin places are NULL). 'operating_status <> ''permanently_closed''' evaluates to NULL for those rows, so the conflate candidate load, the poi-overture search, and the knowledge-overture spatial match all silently dropped ~99.8% of places (conflate saw 278 of 161,975). Use '(operating_status IS NULL OR operating_status <> ''permanently_closed'')' everywhere. Also lower the poi-overture search minConfidence 0.7 -> 0.5 to match the calibrated conflation floor.
The cascade auto-merged any two points within alwaysMergeM (25 m) regardless of
name. On real city-scale data that produced ~75% false links — adjacent but
unrelated businesses ("Apotheke" <-> "Tchibo", "Mr. Kebab" <-> "Seeger")
sit within metres of each other in dense areas. Require nameSimilarity >= a
relaxed close-band floor (0.5) inside alwaysMergeM instead of an unconditional
merge; the soft-window behaviour (full floor + category gate) is unchanged.
Distance alone never forces a merge now.
conflateOverture is a full rebuild but only INSERT ... ON CONFLICT'd, so the link table accumulated across runs (ON CONFLICT overwrites only the same osm<->gers triple; links from prior runs/releases persisted). The table grew to 40k rows mixing fresh and stale links. DELETE the schema's links before the batch insert so the table always equals the current conflation output.
A local harness scoring the matcher against name-INDEPENDENT ground truth (address keys + wikidata) showed the name-only cascade had ~19% of address-checkable links contradicting (false positives) at ~70% building recall. Add, in shouldMatch (consumed offline + at runtime): - wikidata override: equal wikidata/brand-wikidata within the window → match - address corroboration: equal postcode+street+housenumber → same place (name/ category only disambiguate multi-tenant buildings); CONTRADICTING address → reject even on a strong name match - normalizeName now folds ß→ss (NFKD otherwise drops it, breaking ß-streets/names) - geo-server address-key helpers (normalizeStreet, osmAddressKey, overtureAddressKey); ConflationPoint gains optional addressKey/wikidata conflate.ts loads addresses/postcode/brand-wikidata + OSM addr tags/wikidata and populates the keys. Harness result: ~22k links (was ~20k), address-contradiction ~19%→0.7%, building recall ~70%→98%.
normalizeStreet only folded straße/str as a standalone word, but most German streets are compound (Schloßstraße, Hauptstraße — one token), so the suffix form went unfolded: 'schlossstrasse' vs 'schlossstr' became a false address contradiction and the matcher REJECTED genuine same-address links (EDEKA↔Edeka, Sparkasse↔Sparkasse). Fold straße/strasse→str and platz→pl globally. Harness: +1,165 links (23,075), address-contradiction 0.7%→0.3%, building recall 98.8%.
…inks Address corroboration alone merged different businesses sharing one address (mall/multi-tenant: 'Cafe Sack'↔'Schachverband', 'Sparkasse ATM'↔'Mall of Bike'). Use address cardinality: a single-tenant address (one POI per side) is trusted outright, but a multi-tenant address (addressKey shared by >1 POI on a side) additionally requires same-known-category or a name signal. computeLinks precomputes the per-side cardinality and sets ConflationPoint.addressShared. This keeps legitimate single-tenant null-category matches (Nahkauf↔generic Overture name) while rejecting multi-tenant different-business pairs.
…dress
A shared address is necessary but NOT sufficient: OSM and Overture each map
DIFFERENT businesses at the same building, so cardinality-1 ('one POI per side')
still linked different businesses (Essi↔Temial, South Rock↔Area-85). Drop the
single-tenant trust and the address-cardinality machinery; a matching address
now always requires same-known-category or nameSimilarity≥0.5 to confirm the
business (a contradicting address still rejects outright). Precision over the
extra recall the address-only matches gave.
…r categories A shared address + equal category was treated as a match even without a name signal, which wrongly merged co-located same-category businesses (two restaurants in a food court, two gyms). But for categories that are singular per street address (supermarket, pharmacy, bank, hospital, fuel, school, …) a shared address + equal category IS reliable. Gate the category-equality shortcut on a SINGULAR_PER_ADDRESS allowlist; plural categories (restaurants, cafes, bars, gyms, …) still require a name match. Harness: keeps the reliable singular matches (Nahkauf↔Markt) and drops the plural-category mis-links (Tandoori↔Greece).
Art galleries (mapped as museums) cluster in shared arts buildings, so a shared
address + 'both museums' merged different galleries ('Reiter'<->'Esther
Schipper'). Remove museums from SINGULAR_PER_ADDRESS so co-located galleries
require a name match like other plural categories.
Phone is the most specific business-identity signal and the conflation matcher now uses it decisively both ways: a shared, prefix-folded number confirms a match even when name/address differ (rebrands, moved listings); a DIFFERENT number rejects the pair even when name, address, or brand coincide (distinct co-located businesses, or two branches of one chain within the window). Website host is a weaker confirm-only signal. On the Berlin eval set this drives the phone-contradiction rate among phone-bearing links from ~20% to 0% while raising recall (phone matches the name/address tiers miss) and netting more links overall — closing the residual error that name+address alone could not see. normalizePhone / websiteDomain land in geo-server; conflate loads phones[1] / websites[1] (Overture) and phone/contact:phone / website/contact:website /url (OSM) and populates them on every ConflationPoint.
Address the xhigh review of 137bde4. The new phone/website branch had ordering and robustness defects: - Website-confirm ran before the address-contradiction gate and matched any shared host, so two distinct businesses both linking a platform host (facebook.com, google.com, a booking site) force-merged. Now checked AFTER the address gate, category-gated, and skipped for a GENERIC_WEBSITE_HOSTS denylist of non-identifying shared hosts. - Phone was a hard rejecter placed before the wikidata short-circuit; an explicit wikidata identity link is now honoured over a phone discrepancy. - Equal-phone confirmed a match before the category check, so a shared switchboard (a hotel's reception number on its restaurant) merged distinct units. The phone confirm is now category-gated. - Phone is now a SET per side (geo-server parsePhones): multi-value OSM tags (";"-separated) and the full Overture phones[] array are parsed, and a pair matches when the sets intersect / is rejected only when both are non-empty and disjoint — so a stale or secondary number never forces a false split. - normalizePhone strips the "49" country code only when the input carried an explicit international prefix (+/00), not from any long bare number. - websiteDomain uses the URL parser's hostname, dropping userinfo and port. - The embedding-link fallback now applies the same phone-disjoint rejection, so it can't re-link what conflate() deliberately dropped. - Query-time fusePoiResults and the eval sweep harness now build ConflationPoint with the same corroboration fields as the batch path, so live search and threshold calibration match what ships.
- Drop the unused `_lang` parameter from the poi-search orchestrator's runWithShrink (the search closures already capture options.lang). - Name the base-provider id behind a BASE_PROVIDER_ID constant with a comment, so the base-vs-augment fan-out split documents its assumption in one place. - Declare the Foursquare Apache-2.0 NOTICE attribution on the foursquare data source in the poi-overture and knowledge-overture manifests. - Replace two non-null assertions in the extract-osm-pois test with a narrowing guard.
The conflation data sources are rendered as a chip (name · license, linked to url) with the SPDX license linked on the licenses page — the structured name/url + license/licenseUrl fields are the attribution; the freeform `attribution` string is a special-case override only. Both the overture (CDLA-Permissive-2.0) and foursquare (Apache-2.0) sources were missing licenseUrl while carrying a freeform attribution. Add licenseUrl to all four entries (poi-overture + knowledge-overture) and drop the unnecessary attribution strings — "Overture Maps"/"Foursquare" + the linked license is the standard credit. Supersedes the foursquare attribution added in 8f9afdf.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Medformatik
changed the title
- js/request-forgery (critical): pin the Geofabrik boundary fetch host in fetchRegionBbox. The region is already constrained to lowercase path segments by assertValidRegion, but parse the resolved URL and reject anything whose host isn't download.geofabrik.de, so a region value can only select a path under the fixed host (SSRF defense-in-depth). - js/polynomial-redos (high): replace the backtracking address-split regex in overtureAddressKey with a linear last-separator scan; the freeform comes from library data and the old `^(.*?)[\s,]+(\d+\S*)\s*$` was a ReDoS vector. - Add a changeset (@openmapx/core minor) for the conflation toolkit.
…bject The previous guard checked a separate parsed URL but still passed the raw template string to curl, so the host allow-list didn't actually guard the fetched value (CodeQL js/request-forgery stayed open). Build the URL object once, check its host, and fetch its serialized form.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds optional Overture Maps Places as a first-party data source, fused with OpenStreetMap for commercial POI search and place-card enrichment.
Ships OFF by default. The integrations self-disable when unconfigured, the search orchestrator short-circuits to the byte-identical Overpass-only path, and the data-manager jobs run only when enabled — a deployer who doesn't opt in keeps today's behavior with zero added disk/RAM/build. Merging this branch changes nothing about current runtime behavior; that's what makes the feature safe to land before it's switched on anywhere.
What's included
packages/core) — query-time tiered union-find matcher (distance windows + Sørensen-Dice name similarity + category gate), generalizing the EV-charging dedup pattern, plus name-independent corroboration: address keys, wikidata, and phone/website signals.services/data-manager/src/jobs/overture) — DuckDB-over-S3 extract → PostGIS ingest (overture_places.*, GIST + H3) → osmium OSM-POI extract → H3-blocked conflation →poi_conflation_link. Monthly changelog deltas + cron;data overture-*CLI; precision-sweep eval harness.poi-overture(commercial POI search fusion) andknowledge-overture(enrichment: brand, multilingual names, structured opening hours, wikidata cross-reference).poi-searchorchestrator — fans out to Overpass (authoritative base) + Overture (augment), merging via the precomputed link table plus a query-time union-find pass over residuals.Conflation quality
Validated end-to-end on a full Berlin ingest (52.7k OSM × 140.6k Overture). Beyond name + address, phone is used as a business-identity signal: a shared number confirms the same business even across rebrands/moves; a different number rejects distinct co-located businesses and chain branches that name + address alone cannot separate. Phones are handled as sets (absorbing multi-value/stale numbers); website host corroborates, with a generic-platform denylist so a shared
facebook.com/google.comnever force-merges. Measured 0% phone-set contradiction among phone-bearing links. The signal logic was hardened after an xhigh code review (ordering, multi-value parsing, generic-host handling, query-time/eval consistency).Before enabling in production — follow-ups, not merge blockers
Scope is Places only (CDLA-Permissive); the Addresses / ODbL themes are deferred.