Rigour 2.4.0

A feature release: address comparison and keying move into the Rust core and become public API, and LangStr equality is made consistent with its hash.

Highlights

Address comparison (rigour.addresses.compare_address, compare_address_many)

rigour previously normalised addresses but left the comparison to downstream callers, which fell back on string similarity over keyword-stripped text. This release adds a native scorer that analyses both address strings into classed tokens — numbers, address keywords (str./ул./blvd), territory names, free text — and aligns them greedily, order-independently:

• Numbers match exactly or not at all; Unicode digits (١٧, １７, 三) fold to ASCII first, and digit runs glued to letters (д39, 30th) are split so they still align.
• Keywords match across alias forms and languages (boulevard/blvd/бульвар); territory names match across languages via their code (Syria/Сирия/Syrian Arab Republic).
• Free text matches by edit distance over transliterated forms.
• A pair of unmatched numbers where each side asserts a value the other lacks — a differing house or unit number — is penalised far beyond its token weight, pushing the score toward 0.0.

The score is the length-weighted share of aligned tokens. Equivalent renderings score 1.0; transliterated or partly translated matches typically land between 0.5 and 0.9; conflicting house numbers approach 0.0. The accuracy-optimal decision threshold on the benchmark corpus is around 0.3, noticeably lower than typical name-similarity calibrations.

compare_address_many(queries, results) returns the best pairwise score between two lists, the natural shape for two entities each carrying several address renderings. Analysis results are memoised in a process-wide LRU, so one query against many candidates re-analyses the repeated side once. Both functions release the GIL.

Address fingerprint (rigour.addresses.address_fingerprint)

A deterministic key for deduplication and graph-node identity, built on the same analysis: numbers reduce to plain digit strings, keywords to their canonical short form, unambiguous territory names to their code, free text is transliterated where a narrow romanisation exists. Token order is preserved deliberately, so д. 17 стр. 1 and д. 1 стр. 17 key differently.

address_fingerprint("Main Boulevard 5, Syrian Arab Republic")  # "main blvd 5 sy"
address_fingerprint("Main Blvd. 5, Syria")                     # "main blvd 5 sy"

Output is lowercase and space-separated; free text in scripts without a systematic romanisation (Chinese, Arabic) passes through in its native script, so callers needing pure-ASCII identifiers should slugify the result. Fingerprints are stable within one rigour version but may shift between versions as the resources grow.

Address keyword corpus (resources/addresses/forms.yml)

The canonical keyword table grew from 94 to roughly 120 forms, mined from address tokens in the OpenSanctions statement corpus: settlement and district terms (city, town, village, township, ward, commune, prefecture), site descriptors (complex, phase, yard, compound, mansions, shopping, harbour, metro), and upper/lower. Every form carries aliases across the supported languages. shorten_address_keywords and remove_address_keywords pick these up as well.

Normalize.ADDRESS

A new flag for rigour.text.normalize that tokenises with the address-specific category table: & and № are kept as token content, spacing marks separate tokens, and decimal-digit runs are emitted as their own tokens. Mutually exclusive with NAME, which wins when both are set.

LangStr identity hinges on the language tag

An untagged LangStr is now indistinguishable from its content string: it compares equal to the plain str, hashes the same, and deduplicates against it in sets and dict keys. A tagged LangStr is identified by (content, lang) and is not equal to the bare string or to a LangStr with a different tag. Previously equality with str was value-based while the hash included the tag, so str-keyed dict and set lookups silently missed. Closes #255.

Benchmark

The scorer was developed against a new labelled corpus, contrib/address_bench, of about 10,000 address pairs drawn from OpenSanctions source data and hand-adjudicated for whether they denote the same physical location. At this release the scorer reaches AUC 0.936 overall and 0.976 on the STRONG-evidence slice, with 91.9% accuracy at a fixed threshold. Warm throughput on a release build is about 5 µs per compare_address call and 0.4 µs per address_fingerprint call.

Deprecations

• rigour.addresses.remove_address_keywords — use compare_address instead of comparing keyword-stripped strings. Emits DeprecationWarning; removal in a future version.
• rigour.addresses.shorten_address_keywords — use address_fingerprint, which reduces keywords to the same canonical short forms as part of a full keying serialisation. Emits DeprecationWarning; removal in a future version.

Upgrade notes

• Code that relies on a tagged LangStr comparing equal to its plain str content will see those comparisons become False. Untagged LangStr values are unaffected and now round-trip cleanly through str-keyed containers.
• normalize_address is unchanged. The expanded keyword table affects shorten_address_keywords and remove_address_keywords output for the newly added forms.
• Wheels now embed the address forms table; no new runtime dependency.

Full changelog: v2.3.1...v2.4.0