Skip to content

Rigour 2.2.3

Choose a tag to compare

@pudo pudo released this 19 Jul 11:23
· 90 commits to main since this release

This release substantially improves language-code resolution in rigour.langs and hardens the name-matching internals against pathological, very long inputs.

Highlights

Language resolution (rigour.langs)

  • Resolve BCP 47-style tags via their primary subtag, so codes carrying a script or region variant (zh-Hans, pt-BR, sr-el, be-tarask) now resolve instead of returning None. The full tag keeps precedence, so synonyms like chi_sim and aze_cyrl are unaffected, and collective or non-language subtags (roa-tara, mul-x-foo) still return None.
  • Fixed several language-code conflations and invalid-code leaks:
    • Old Norse (non) is no longer aliased to Norwegian (nor).
    • Albanian now uses its canonical code sqi (previously conflated with Lower Silesian sli, which had injected the invalid code alb).
    • The Burmese synonym group no longer conflates distinct Burmish languages (Intha, Tavoyan, Taungyo, Rakhine, Marma) with Burmese.
    • list_to_alpha3 output is now filtered to canonical ISO 639-3 codes; ISO 639-2/B and Tesseract-style codes (ger, chi, …) are still accepted as inputs but no longer emitted.
    • Added zxx ("no linguistic content") to the non-language set alongside mis/mul/und.
  • Unified Nepali on the FtM-whitelisted macrolanguage nep; the individual-language code npi now resolves to it.
  • Greatly expanded label coverage for every language followthemoney whitelists:
    • Native endonyms (plus common exonym variants) for 15 previously code-only languages: amh, est, gle, kan, lav, lit, ltz, mlt, mon, pus, slv, tgk, tgl, tuk, uzb.
    • Russian, French, Spanish and Ukrainian labels for all whitelisted languages.
    • Arabic and Chinese labels, plus common English exonym variants seen in sanctions/corporate data (Mandarin, Cambodian, Castilian, Flemish, Byelorussian, Bokmål, Pushto, …).
    • Added "Central Khmer" (UK FCDO usage) → khm.
    • Fixed Pashto: "Pashto"/"Pashtu" now resolve (the SIL ref-name is "Pushto").

Name matching

  • Capped the per-character inputs to the O(n·m) dynamic-programming paths (compare_parts, pick, ordering) at a fixed MAX_NAME_LENGTH of 384. Previously these allocated unbounded cost/backpointer matrices over the joined characters of both names — two 600-part names measured at 379 MB RSS and 0.62 s for a single call, growing quadratically. Overflow parts fall out as solo clusters, preserving the "every part appears exactly once" contract.
  • MAX_NAME_LENGTH is now exported from the Rust core and re-exported by rigour.env as the single source of truth shared by the Python distance wrappers and the Rust DP paths.

Internal

  • Span length is now computed on demand rather than eagerly, so building a span that is never measured no longer pays for the character count.
  • Removed the private rigour._core.string_number export, which had no consumers outside the test suite (the parser remains used Rust-internally).

Upgrade notes

  • list_to_alpha3 no longer emits non-canonical codes such as ger or chi. They remain valid inputs but will not appear in output; consumers expecting ISO 639-2/B output should map accordingly.
  • The RR_MAX_NAME_LENGTH environment variable is now a no-op — MAX_NAME_LENGTH is fixed at 384.

Full changelog: v2.2.2...v2.2.3