Rigour 2.2.3
This release substantially improves language-code resolution in rigour.langs and hardens the name-matching internals against pathological, very long inputs.
Highlights
Language resolution (rigour.langs)
- Resolve BCP 47-style tags via their primary subtag, so codes carrying a script or region variant (
zh-Hans,pt-BR,sr-el,be-tarask) now resolve instead of returningNone. The full tag keeps precedence, so synonyms likechi_simandaze_cyrlare unaffected, and collective or non-language subtags (roa-tara,mul-x-foo) still returnNone. - Fixed several language-code conflations and invalid-code leaks:
- Old Norse (
non) is no longer aliased to Norwegian (nor). - Albanian now uses its canonical code
sqi(previously conflated with Lower Silesiansli, which had injected the invalid codealb). - The Burmese synonym group no longer conflates distinct Burmish languages (Intha, Tavoyan, Taungyo, Rakhine, Marma) with Burmese.
list_to_alpha3output is now filtered to canonical ISO 639-3 codes; ISO 639-2/B and Tesseract-style codes (ger,chi, …) are still accepted as inputs but no longer emitted.- Added
zxx("no linguistic content") to the non-language set alongsidemis/mul/und.
- Old Norse (
- Unified Nepali on the FtM-whitelisted macrolanguage
nep; the individual-language codenpinow resolves to it. - Greatly expanded label coverage for every language followthemoney whitelists:
- Native endonyms (plus common exonym variants) for 15 previously code-only languages:
amh,est,gle,kan,lav,lit,ltz,mlt,mon,pus,slv,tgk,tgl,tuk,uzb. - Russian, French, Spanish and Ukrainian labels for all whitelisted languages.
- Arabic and Chinese labels, plus common English exonym variants seen in sanctions/corporate data (Mandarin, Cambodian, Castilian, Flemish, Byelorussian, Bokmål, Pushto, …).
- Added "Central Khmer" (UK FCDO usage) →
khm. - Fixed Pashto: "Pashto"/"Pashtu" now resolve (the SIL ref-name is "Pushto").
- Native endonyms (plus common exonym variants) for 15 previously code-only languages:
Name matching
- Capped the per-character inputs to the O(n·m) dynamic-programming paths (
compare_parts,pick,ordering) at a fixedMAX_NAME_LENGTHof 384. Previously these allocated unbounded cost/backpointer matrices over the joined characters of both names — two 600-part names measured at 379 MB RSS and 0.62 s for a single call, growing quadratically. Overflow parts fall out as solo clusters, preserving the "every part appears exactly once" contract. MAX_NAME_LENGTHis now exported from the Rust core and re-exported byrigour.envas the single source of truth shared by the Python distance wrappers and the Rust DP paths.
Internal
Spanlength is now computed on demand rather than eagerly, so building a span that is never measured no longer pays for the character count.- Removed the private
rigour._core.string_numberexport, which had no consumers outside the test suite (the parser remains used Rust-internally).
Upgrade notes
list_to_alpha3no longer emits non-canonical codes such asgerorchi. They remain valid inputs but will not appear in output; consumers expecting ISO 639-2/B output should map accordingly.- The
RR_MAX_NAME_LENGTHenvironment variable is now a no-op —MAX_NAME_LENGTHis fixed at 384.
Full changelog: v2.2.2...v2.2.3