A CJK honorific reaches the vocabulary only when the name is typed with the exact codepoints the lexicon expects. Three compatibility spellings miss it, all silently — the honorific lands in a name field and nothing is reported:
parse("김민준 씨.") # family 김, given 민준, suffix 씨. <- ASCII period, works
parse("김민준 씨.") # given 김, middle 민준, family 씨. <- U+FF0E fullwidth
parse("김민준 씨。") # given 김, middle 민준, family 씨。 <- U+3002 ideographic
parse("김민준 씨。") # given 김, middle 민준, family 씨。 <- U+FF61 halfwidth ideographic
The fullwidth and ideographic stops are the ones a CJK writer is more likely to type than the ASCII one, since they are what a Japanese or Chinese IME produces by default.
Three different mechanisms, one candidate fix
1. Full stops. _lexicon._normalize does .strip("."), ASCII only, so _normalize("씨.") is "씨." and the vocabulary lookup misses before any predicate is consulted.
2. Halfwidth katakana. is_initial("ラ.") is True while is_initial("ラ.") is False. _SCRIPT_RANGES deliberately excludes U+FF65–FF9F ("legacy bank/CSV data uses it, but it is a separate normalization problem"), and _policy._NO_INITIALS silently inherits that exclusion — so the constant does not enforce its own stated rule on the halfwidth spelling of the same character. Nothing misroutes today, because every kana honorific is multi-character, but the inconsistency is now written into a documented rule.
3. NFD hangul. Decomposed Korean misses where decomposed Japanese does not:
parse(NFD("김민준, 씨.")) # title 씨., family 김민준 <- wrong
parse(NFD("田中さん, 様.")) # family 田中, suffix さん, 様. <- right
Hangul decomposes into jamo (3+ characters, so no single-character shape matches); Han does not decompose at all. #272 deliberately kept segmentation matching raw, so this is a known consequence — but the ko/ja asymmetry is not recorded anywhere.
The shared decision
An NFKC fold for vocabulary lookup only would address all three — U+FF0E→., ラ→ラ, and NFD→NFC by composition. That is a single change with one risk profile to evaluate, rather than three patches.
The anti-#100 invariant makes this delicate: token spans must index the original string exactly, so any normalization has to happen at lookup and must never rewrite token text. _normalize already works this way, which is why it is the natural home.
Found while reviewing #320 (PR #321). Not a regression — master agrees on every row above. #320's release note is scoped to the ASCII period so it does not imply otherwise.
A CJK honorific reaches the vocabulary only when the name is typed with the exact codepoints the lexicon expects. Three compatibility spellings miss it, all silently — the honorific lands in a name field and nothing is reported:
The fullwidth and ideographic stops are the ones a CJK writer is more likely to type than the ASCII one, since they are what a Japanese or Chinese IME produces by default.
Three different mechanisms, one candidate fix
1. Full stops.
_lexicon._normalizedoes.strip("."), ASCII only, so_normalize("씨.")is"씨."and the vocabulary lookup misses before any predicate is consulted.2. Halfwidth katakana.
is_initial("ラ.")isTruewhileis_initial("ラ.")isFalse._SCRIPT_RANGESdeliberately excludes U+FF65–FF9F ("legacy bank/CSV data uses it, but it is a separate normalization problem"), and_policy._NO_INITIALSsilently inherits that exclusion — so the constant does not enforce its own stated rule on the halfwidth spelling of the same character. Nothing misroutes today, because every kana honorific is multi-character, but the inconsistency is now written into a documented rule.3. NFD hangul. Decomposed Korean misses where decomposed Japanese does not:
Hangul decomposes into jamo (3+ characters, so no single-character shape matches); Han does not decompose at all. #272 deliberately kept segmentation matching raw, so this is a known consequence — but the ko/ja asymmetry is not recorded anywhere.
The shared decision
An NFKC fold for vocabulary lookup only would address all three — U+FF0E→
.,ラ→ラ, and NFD→NFC by composition. That is a single change with one risk profile to evaluate, rather than three patches.The anti-#100 invariant makes this delicate: token spans must index the original string exactly, so any normalization has to happen at lookup and must never rewrite token text.
_normalizealready works this way, which is why it is the natural home.Found while reviewing #320 (PR #321). Not a regression —
masteragrees on every row above. #320's release note is scoped to the ASCII period so it does not imply otherwise.