Geocode addresses locally against the regional address points - #167
Merged
Conversation
The address point data is published as normalized components, but every facility registry publishes a single free-text line. Matching one against the other needs a parser on the query side that produces exactly the vocabulary the reference side already uses, which is why this lives beside normalize_street_type and normalize_directional rather than in a source repository. Two defects in the published data are reduced here rather than worked around at each call site. Counties that publish the house number as a numeric column leave a float tail, and Knox zero pads to five digits, so a literal join against Knox fails for all 26,990 of its records. Numbered routes are spelled three ways across the region -- "STATE ROUTE 33", "ST RT 33", "SR 33" -- and are folded to one prefix so that a name joins across a county line. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
morpc.geocode() calls Nominatim, an external rate limited service, which the group quarters source strategy identifies as a prerequisite to be replaced: eight of the proposed source repositories emit geometrybasis = geocoded, and none of them should depend on a public API to do it. geocode_addresspoints() matches against morpc-addresspoints-standardize instead. The published database cannot be queried directly -- no index on the join fields, geometry as a WKB blob, and components normalized inconsistently between counties -- and it cannot be indexed in place, because its descriptor records a hash of the file. So build_geocode_index() derives a smaller database from it once, normalizing both sides through the same functions, and rebuilds when the release or the normalization version moves on. Matching runs in tiers and reports which one fired, how many points it found and how far apart they were. Points that are metres apart are units of one building and are returned as their centre; points that are kilometres apart are different places and the address is left unmatched with a note saying so. A null with a reason is worth more here than a point that is quietly wrong. Measured against the 305 facilities that publish both an address and a point: DODD 91% matched, median 4 m from the published coordinate; CMS 95%, median 117 m, where the spread is largely CMS's own geocoding error, which it flags in geocodingfootnote. A crude match without address parsing reached 79% and 88% on the same facilities and put one point 126 km away. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #161.
morpc.geocode()wraps Nominatim, an external rate-limited service. The group quarters source strategy calls a local geocoder a prerequisite rather than a refinement: eight of the proposed facility-source workflows emitgeometrybasis = geocoded, and none of them should depend on a public API to do it. This addsgeocode_addresspoints(), which matches againstmorpc-addresspoints-standardizeinstead — offline, reproducible, and better on in-region addresses.geocode()is unchanged, so the Nominatim path inmorpc-mls-standardizekeeps working.Why an index has to be built
The published address point database cannot be queried directly. It carries no index on the fields a match joins on, so every lookup scans 1.25 million rows; its geometry is a WKB blob rather than coordinates; and its components are normalized inconsistently between counties. It also cannot be indexed in place — its Frictionless descriptor records a hash of the file, so adding an index makes
resolve_data_path()refuse to load it.So
build_geocode_index()derives a smaller database once: match fields normalized through the same functions the query side uses, geometry decoded to coordinates, indexed over the join. 29 seconds, 114 MB, source file untouched. It rebuilds itself when either the source release orCONST_GEOCODE_INDEX_VERSIONmoves on — the latter because the index stores names already normalized, so a change to the normalization silently invalidates it in a way the source hash cannot detect.Matching, and what it refuses to do
Three tiers, from every component down to house number and street name alone. The tier that fired is reported, along with how many points it found and how far apart they were.
Points metres apart are the units of one building or the buildings of one campus, and are returned as their centre. Points kilometres apart are different places, and the address comes back unmatched with a note saying why. The default tolerance of 500 m sits above the widest campus in the validation set (353 m across 133 points) and well below the closest genuine collision (two
3000 BETHEL RDaddresses 75 km apart).That refusal is the point. These outputs feed
geometrybasis = geocodedin eight downstream repositories, and a null with a stated reason is worth more there than a point that is quietly wrong.Measured
Against the 305 facilities in
morpc-doddfacilities-standardizeandmorpc-cmsnursinghomes-standardizethat publish both a street address and a point — so the published coordinate scores the match:The crude baseline is a component join with no address parsing, which is roughly what the earlier Ohio Department of Health attempt did. It put one point 126 km away; the worst case here is 4,070 m.
CMS carries a
geocodingfootnotecolumn and geocodes its own addresses, so its 117 m median is largely CMS's error rather than ours. DODD is the honest measure.Of the 22 remaining misses, most are addresses the counties never published — the street is in the data, that house number is not. Two are source typos (
BELLFONTAINEfor Bellefontaine,JOHN SHIELDfor John Shields) that only a fuzzy tier would catch.Deliberately not included
rapidfuzz, a new dependency, to recover roughly 2 of 305. The miss analysis does not justify it yet.Two things found along the way
cityandfulladdress, zero-padded house numbers — and mixed-casestreetnamecosts matches in Franklin too. This PR normalizes past all of it on both sides, which is most of the 79% → 91%. Fixing it upstream removes the workaround.resolve_data_path()cannot fetch a private repository's release asset. The URL 404s anonymously, so the address point database has to be downloaded withghfirst.morpc-addresspoints-standardize's README documentsload_data()as the way to get it, and that path does not work for anyone today. Not fixed here; worth its own issue.Tests
32 in
tests/test_geocode.py, up from 10. The matcher tests build a small index in a fixture and pass it viaindexPath, so they exercise tiers, centroid collapse and ambiguity refusal without the 183 MB download.The 4 failures in
tests/test_utils.py(datetime parsing) predate this branch and are untouched by it.🤖 Generated with Claude Code