Skip to content

Rigour 2.6.0

Choose a tag to compare

View all tags
@pudo pudo released this 21 Oct 18:08
· 8 commits to main since this release
v2.6.0
d2c0f55

A feature release: maybe_ascii now produces the ASCII spellings names are actually written in for Cyrillic, Greek, Georgian and Armenian. A new rigour.text.ascii_text makes those spellings the basis of every ASCII key rigour builds. Org-type and symbol tables gain broad Chinese, Japanese and Korean coverage. ASCII forms, address and identifier keys, and territory lookup keys change as a result.

Highlights

Common spellings from maybe_ascii

maybe_ascii relied on CLDR's scholarly romanizations: Шевченко → Sevcenko, Βασίλης → Basiles, კედელაშვილი → k'edelashvili. Some letters also came out as punctuation or non-ASCII: Ґудзь → Gudz', Қазақстан → Kˌazakˌstan, Armenian Քոչարյան → K'oc'aryan, and Georgian capitals (Mtavruli) not transliterated at all.

A per-script pre-pass now substitutes the common ASCII spelling before the ICU4X transliterator runs, while the source letter is still known. Native Latin diacritics keep folding to the bare letter (Škoda → Skoda, Ștefan → Stefan).

from rigour.text.translit import maybe_ascii

maybe_ascii("Шевченко")      # "Shevchenko"   (was "Sevcenko")
maybe_ascii("ШЕВЧЕНКО")      # "SHEVCHENKO"
maybe_ascii("Милошевић")     # "Milosevic"
maybe_ascii("Βασίλης")       # "Vasilis"      (was "Basiles")
maybe_ascii("კედელაშვილი")   # "kedelashvili" (was "k'edelashvili")
maybe_ascii("Քոչարյան")      # "Kocharyan"    (was "K'oc'aryan")
maybe_ascii("Škoda")         # "Skoda"
  • Cyrillic: ш ч щ ж ц → sh ch shch zh ts, я ю ё → ya yu yo, й → i, word-final ий/ый → y. ь/ъ are dropped.
  • Serbian and Ukrainian spellings: a word containing a Serbian or Macedonian letter spells ш ч ж ц as s c z c (Милошевић → Milosevic). A word containing a Ukrainian or Belarusian letter spells г и as h y (Гіркін → Hirkin).
  • Other Cyrillic languages: Serbian/Macedonian, Kazakh, Tajik, Tatar, Bashkir, Chechen, Abkhaz, Chuvash and pre-1918 letters that had no ASCII rendering now have one.
  • Greek: modern spellings: β → v, η → i, φ → f; μπ ντ γκ → b d g at word start; αυ ευ → av ev.
  • Georgian: ejectives lose their apostrophe, and Mtavruli capitals fold to lowercase.
  • Armenian: sibilants and affricates become digraphs (շ → sh, ց → ts), and aspirates lose their apostrophe.
  • Latin: ɔ → o, ɣ → g; ʕ is dropped.
  • Case: replacements follow the case of the neighbouring letter.

Each mapping was checked against the human ASCII spellings of the same names in person_names, and rules that matched fewer real spellings were left out (Cyrillic х → kh, Ukrainian є → ye, …). Exact agreement went from 29.5% to 42.1% for Cyrillic, 22.9% to 41.4% for Greek, 27.9% to 71.3% for Georgian and 46.1% to 62.8% for Armenian (#292, #293).

maybe_ascii also NFC-composes its input, so decomposed text gets the same result as composed text.

rigour.text.ascii_text

A new transliteration function for text in any script. It runs maybe_ascii and hands whatever is still non-ASCII to normality.ascii_text: other scripts, and symbols that maybe_ascii passes through unchanged. Use it instead of normality.ascii_text wherever ASCII keys should agree with NamePart.ascii (#303).

from rigour.text import ascii_text

ascii_text("Шевченко")  # "Shevchenko"  (normality.ascii_text: "Sevcenko")
ascii_text("北京")       # "bei jing"

Identifier (StrictFormat), address (normalize_address(..., latinize=True)) and territory name normalisation now go through it. normalize_territory_name no longer decomposes names and re-folds latinize_text's diacritics, and Cyrillic queries now find more territories (lookup_territory("Чернигов") resolves to Chernihiv).

Chinese, Japanese and Korean org types

The org-type and symbol tables gain 110 aliases: 79 Chinese, 18 Korean and 13 Japanese, covering legal forms (有限责任公司, 유한공사, 公開株式会社, …) and generic components such as group, branch, trading, industry and development, in both simplified and traditional forms. Over about 79k Han-script organisation names, the text left after stripping these terms is 17% shorter (#290).

Other changes

  • pick_lang_name lets names without a language tag compete in the preferred-language tier. A single Arabic-, Persian- or Urdu-tagged name no longer beats any number of untagged Latin names for the caption (#300).
  • "mul" (multiple languages) is accepted as a language code: iso_639_alpha3("mul") == "mul", and rigour.langs.MULTI_LANG is exported (#301).
  • Numeric symbol identifiers are no longer kept in the process-wide interner, so analysing names with many distinct numbers doesn't grow memory for the life of the process. Fixes #229 (#302).
  • person_names data refreshed.

Dependencies

  • pyo3 0.29 → 0.30 (Rust).
  • zstd 0.14.0 → 0.14.3 (Rust).
  • lru 0.18.4 → 0.18.5 (Rust).
  • serde_json 1.0.151 → 1.0.154 (Rust).
  • bitflags 2.13.1 → 2.13.2 (Rust).
  • Development: the ruff pin is widened to < 0.18.

Upgrade notes

  • maybe_ascii output changes for Cyrillic, Greek, Georgian and Armenian text, and so do NamePart.ascii, comparable, metaphone codes, name picking and address tokens. Stored folded forms, such as yente indexes, should be rebuilt.
  • ASCII keys built from ascii_text change: normalize_address(..., latinize=True), StrictFormat.normalize and normalize_territory_name. Recompute any stored keys.
  • Territory lookup keys change. Two weak aliases that collided under the new keys were dropped: Greece's Gəlɛ́s (it now keys like the Spanish Gales for Wales) and Tonga's Tongo (it now keys like Togo's names).
  • rigour.text.ascii_text and rigour.langs.MULTI_LANG are new public API.

Full changelog: v2.5.0...v2.6.0

Assets 2
Loading