Japanese names: the kana license by default and a pluggable segmenter (#272) - #297
Open
derek73 wants to merge 13 commits into
Open
Japanese names: the kana license by default and a pluggable segmenter (#272)#297derek73 wants to merge 13 commits into
derek73 wants to merge 13 commits into
Conversation
Task 1 of the #272 Japanese-support plan: classification only, no order/segmentation defaults change. Script gains HIRAGANA (U+3040- U+309F) and KATAKANA (U+30A0-U+30FF, which includes the prolonged mark and the U+30FB middle dot). _vocab.effective_script extends single_script with the kana license from the 2026-07-29 amendment: a mixed Han/hiragana/katakana token is Japanese and resolves to the HIRAGANA carrier entry; pure-katakana keeps single_script's own answer since nothing defaults on it. Updated two pre-existing pins that this table change broke: the single_script test asserting kana was unclassified (now KATAKANA), and test_regex_sync's #271 differential-rule sync check, which now scopes its expected spans to the HAN/HANGUL scripts #271 actually governs rather than every _SCRIPT_RANGES entry, since #272 grows the same table for an unrelated reason. Quality-review fix-first pass, folded into this same commit: both classifiers now run on an NFC-normalized copy of the text, never the raw input. NFD decomposes precomposed katakana onto a base character plus a combining voiced/semi-voiced sound mark (U+3099/U+309A), which sit in the HIRAGANA block rather than katakana's -- classifying raw NFD text could therefore see a pure-katakana token as Han-free but kana-mixed and wrongly hand it the kana license (HIRAGANA) instead of declining it (KATAKANA). Separately, NFD decomposes Hangul syllables onto bare jamo (U+1100-U+11FF), entirely outside the HANGUL range, so raw NFD Korean input missed the shipped family-first order rule altogether and silently fell back to the positional default -- a live gap in #294's shipped behavior, not merely a #272 nicety. Both directions are demonstrated in tests/v2/pipeline/test_vocab.py by simulating the pre-fix classifier against NFD input built with unicodedata.normalize() (never pasted as decomposed literals): the pre-fix classifier returns None for both an NFD katakana and an NFD hangul string, and would have returned HIRAGANA (wrongly) for the raw NFD katakana case under a naive license check. Matching (is_initial, suffix lookups, etc.) deliberately stays on raw text elsewhere in the module -- NFD only ever costs a match there, never a wrong one, so that asymmetry is safe and unchanged. The normalized copy is used for classification only; token text and rendered spans are exactly what the caller wrote, still NFD if that's what came in (see the new test_nfd_korean_input_still_reads_family_first in test_parser.py, which compares NFC-normalized field values since the rendered family/ given text itself stays NFD). Also folded in from the same review: reworded the "no supplementary- plane kana to chase" claim (false -- 311 assigned codepoints exist; none are worth chasing, since they're archaic/phonetic-extension forms no modern name uses); added the halfwidth-kana exclusion rationale and the block-vs-UAX#24 clause to the range-table comment; corrected "kana-only" to "katakana-only" in effective_script's docstring (さくらエミ is kana-only and licensed); added a forward-pointer from single_script to effective_script; moved _JA_SCRIPTS/_JA_PATTERN up beside _SCRIPT_PATTERNS and changed _JA_SCRIPTS from a sorted frozenset to a plain table-ordered tuple (nothing uses membership); made HIRAGANA's Policy docstring self-contained; folded the temporal-named Script-enum test into test_script_values_are_the_public_names; and trimmed the Han/Hangul-duplicate assertions out of the kana single_script test (renamed test_kana_singles_classify). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The pluggable-segmenter seam from the locales spec §4, as amended 2026-07-29: public Segmentation/Segmenter types, a keyword-only Parser(segmenter=...) field, and the value plumbed into ParseState. Nothing consults it yet -- the script_segment stage picks it up in the next commit, pinned here by a strict xfail on the totality exception. Segmentation is an index protocol (strictly ascending interior offsets plus an optional confidence), so a segmenter physically cannot invent, drop, or rewrite characters; Segmentation(()) means "confidently one token" where None declines. parser_for carries base.segmenter through -- packs are pure data and cannot supply one, so base=Parser(segmenter=...) is how a pack and a segmenter combine. The two promises the spec deferred with the hook land with it: Parser pickles iff its segmenter does, and a segmenter's own exceptions propagate rather than being swallowed as content errors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The script_segment stage now reaches the Parser(segmenter=...) hook it has been carrying since Task 3. Vocabulary first, segmenter on decline, so parser_for(ZH, JA, segmenter=...) composes: a listed surname is a dictionary certainty and wins, and the segmenter takes what is left. Its Segmentation may cut any number of times, so the split path is written for n cuts and the vocabulary hit is its one-cut case -- one implementation, so the two sources cannot drift apart on index arithmetic. The token gate swaps single_script for effective_script: kana-licensed composites (高橋みなみ) resolve to the HIRAGANA carrier and gate in under the JA pack's set, while pure katakana resolves to KATAKANA, which no activation set contains. A declined answer (None), a confident whole (empty splits), and an out-of-bounds one (a buggy segmenter -- Segmentation never sees the text, so the upper bound is the stage's to check) all leave the token alone. The segmenter's own exceptions propagate, the declared totality exception, which flips Task 3's strict xfail to a live test. An answer scoring under _SEGMENTER_CONFIDENCE_FLOOR attaches a SEGMENTATION report naming the score; the floor is a placeholder pending Task 5's measurement against namedivider's real distribution. Also narrows Segmentation's non-iterable guard to probe iter() rather than wrapping tuple(): a TypeError raised inside a caller's generator is that generator's bug and was being relabeled as "splits must be an iterable". Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
locales.JA activates segmentation for the Japanese repertoire and locales.ja_segmenter() adapts namedivider-python (the new nameparser[ja] extra) to the Segmenter protocol; parser_for gains a segmenter= keyword so the two combine without a base Parser. Two findings from measuring namedivider 0.4.1 for real: * Its scores are bimodal, not clustered near 1 as the amendment assumed -- 1.0 for a rule-based division, a length-driven softmax in 0.23-0.68 for a statistical one, with the errors straddling the correct median. No floor inside that band separates error from success, so _SEGMENTER_CONFIDENCE_FLOOR stays 0.9, now sitting in the measured empty gap between the two modes rather than on a guess. * A segmenter must only be shown a whole name. Handed one token of an already-divided "山田 太郎" it answers anyway (scoring the two-character 山田 1.0 by rule), so the stage now consults it only when the gated token is the name part's only script-written one. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The property fuzzer's script_orders sample and several source docstrings still described the pre-#272 world: four script members where the comment said two, and three forward references to "#272's pluggable segmenter" pointing at code that has now landed in this branch. _SCRIPT_ORDER_TABLES gains a table keyed on Script.HIRAGANA, the kana license's carrier -- the one key reached indirectly, since a mixed kanji+kana name resolves to it rather than to any script it is literally written in, and pointing it at GIVEN_FIRST makes a consulted table distinguishable from the default it would otherwise agree with. AGENTS.md's scoped-exception paragraph gains the kana license itself (hiragana identifies Japanese; pure katakana is excluded because transcribed foreign names keep their source order), the corrected count, and the pack-layering line's new _types entry. test_regex_sync.py's kana work -- the differential-toml pin extension, the _SCRIPT_RANGES completeness assertion and the docstring's stale scoping premise -- lands with the toml change it pins, so that each commit is green on its own. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
usage.rst gains the Japanese story in the East Asian section, told background first: how a Japanese name is written across kanji and the two kana syllabaries, why hiragana identifies a native name and katakana does not, then the two behaviors that follow without configuration (kana-licensed family-first order, the nakaguro as a token separator), then the segmenter opt-in, then the boundaries -- pure katakana, romanized text, the two-character rule, the SEGMENTATION report's own wording, silent declines, and the CLI's inability to attach a segmenter. locales.rst gains a Segmenters section carrying the decline contract (segment_scripts unions, so a segmenter is offered every activated script's tokens and must decline what it cannot read) and the empty-lexicon pack recipe ja is the shipped example of; customize.rst states that the kana behaviors ride the two existing switches and that the middle dot rides neither; concepts.rst qualifies "parsing never raises" with the one exception a user-supplied callable creates. migrate.rst's CJK section absorbs the new shapes rather than growing a parallel one, every claim checked shape by shape against the pinned 1.4 worker: the spaced flips, the lone-token moves, both nakaguro outcomes, the nickname render, and the katakana non-change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The CJK rule's character class gains the two kana blocks, and its label names both issues: #272's kana license and nakaguro separator move name pieces between the same first/middle/last fields on the same native-script input, so this is one diff shape rather than two, and splitting the rule by issue would buy no tightness. The sync pin widens with it, and stops scoping itself by issue. That scoping rested on a premise #272 falsified -- the kana members were classification-only, keyed by no default -- and comparing against the whole _SCRIPT_RANGES table is the stronger promise anyway: a new script now fails this test until someone decides in writing whether the rule should cover it, which the added set assertion says out loud. Both corpora still contain no CJK name at all, so the rule classifies nothing in a real run; the harness passes 654 names with 18 classified diffs and none unexplained. That blind spot is #295's subject, cited in the rule's comment, and it is why the behavioral guarantee lives in the case table rather than here. corpus_issues.jsonl is deliberately not regenerated: a dry run adds three Latin strings from #296 and, as #295 predicts, no CJK -- unrelated tracker content that belongs in its own reviewable act. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
々 (U+3005) repeats the preceding kanji and carries Script=Common under UAX #24, so it fell outside every range in _SCRIPT_RANGES. That made 佐々木 a mixed-script token: "佐々木 太郎" REVERSED, giving given 佐々木 and family 太郎, and the token never gated into segmentation. 佐々木 (Sasaki) is a top-20 Japanese surname, and 野々村 and 奈々 are ordinary names, so this was not an exotic corner. The mark joins the HAN ranges on the same block-style reasoning U+30FB already gets -- it can only appear inside a Han-written name, being a repeat of the character before it -- along with all three hand-copies the sync pins police. Verified with the extra rather than assumed: namedivider reads 佐々木健 as 佐々木 + 健, so the integration assertion pins the real answer. 佐 is absent from the zh surname list, so the zh pack's reading of these names does not change. Also from the branch review: the differential rule gains U+FF65, whose halfwidth transcriptions do split ('ﾏｲｹﾙ･ｼﾞｬｸｿﾝ') even though halfwidth kana is deliberately unclassified -- the sync pin names that one sanctioned extra rather than relaxing to a subset check, so a second source of divergence still fails. The rest of the halfwidth block stays out: a dotless 'ﾏｲｹﾙ ｼﾞｬｸｿﾝ' is byte-identical on both sides, measured, and covering it would pre-excuse a future regression. Plus four documentation corrections: the nakaguro's exemption from both policy opt-outs stated in AGENTS.md where the doctrine lives, usage.rst's forward pointer widened to kana, _tokenize.py's dangling promise of a follow-up issue replaced with the standing reason, and the available() tuple moved off the #271 bullet that predates ja. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #297 +/- ##
==========================================
+ Coverage 98.30% 98.42% +0.12%
==========================================
Files 40 41 +1
Lines 2594 2735 +141
==========================================
+ Hits 2550 2692 +142
+ Misses 44 43 -1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The adapter's guard stack ran only in the extra-gated integration tests, which the coverage-bearing CI job never executes -- so codecov/patch read locales/ja.py at 55% while ja-extra proved it green. A stubbed namedivider module exercises every branch in the plain suite, including the two defensive paths the real 0.4.x library cannot reach, plus the gbdt selection. Also pins the two remaining uncovered new lines: parser_for's base type check and Segmentation's confidence TYPE error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two behavior changes, drawing one line the segmenter path did not previously state: a protocol violation by the segmenter's AUTHOR raises, while an adapter's defense against its third-party library declines. A segmenter answer cutting at or past the end of the token it was handed now raises ValueError naming the offset and the length, matching the wrong-answer-type TypeError beside it. Both are stage-detectable bugs in user code, inside the already-declared totality exception; the old silent decline made an off-by-one segmenter undebuggable, since every answer it gave simply vanished and the name merely looked undivided. The ja adapter's confidence clamp becomes epsilon-tolerant-else- decline. Float noise at the edge of namedivider's softmax still clamps into [0, 1] -- sized for float32, whose eps is ~1.2e-7, since rule-based answers score exactly 1.0 and a tighter band would reject a correct division that came back one ulp high. A score outside that band does not clamp: a divider reporting 87.0 is broken and its division is worth no more than its score, so the adapter declines, which keeps the parse safe and observable by the report's absence where the old clamp mapped garbage to "certain" and silenced it. NaN declines through the same comparison. Also: parser_for's segmenter takes the UNSET sentinel, so an explicit None clears the base's rather than reading as "not given"; the SEGMENTATION docstring covers both of its sources; a bare-string guard on Segmentation.splits; the U+3005 rationale corrected in four files (it is Script=Han, and the BLOCK table is what needed the singleton); the composition claim qualified wherever it appears; and tests for the lot, including the range-disjointness pin, the capture-the-argument and unactivated-neighbour stage cases, and the lone-hiragana, astral-kanji and ZH+JA-collision integrations. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #272.
Summary
Japanese name support for the still-unreleased 2.1.0, completing the East Asian work #294 began:
parse("高橋 みなみ")→ family 高橋). Pure-katakana names are deliberately never licensed: they are predominantly transcribed foreign names in source order.Segmentation/Segmenterprotocol, keyword-onlyParser(segmenter=...), consulted inside the comma-gated stage only where the surname vocabulary declines, with an only-script-token precondition so already-divided names are never re-split. Segmenter exceptions propagate (the declared totality exception); out-of-bounds answers decline.locales.JA+ja_segmenter()(pip install nameparser[ja]→ namedivider-python, MIT + custom-terms data accurately documented):parser_for(locales.JA, segmenter=locales.ja_segmenter())divides unspaced kanji names — 山田太郎 → 山田/太郎, and 高橋一郎 correctly (the exact name the zh pack's docs use as its mis-split warning). Basic divider by default;gbdt=Trueopt-in (first use downloads its model files).AmbiguityKind.SEGMENTATIONreport with the score; rule-based ones (measured: exactly 1.0) stay silent. The 0.9 floor sits in the measured empty gap of a bimodal distribution.Review focal points
Parserpickles iff its segmenter does; 2.0.xParserpickles raise at load (layout guard — second consecutive release with a pickle break, called out in the release log).ja-extrajob pinned to 3.14 (namedivider's dependency chain has no 3.15 wheels); the main matrix still proves the zero-dependency install.Known residual risks (from the whole-branch final review)
>=0.4— CI won't see upstream API/score changes until the lock moves; worth a periodic upgrade run.--locale jaon the CLI cannot attach a segmenter and is inert by itself (documented; auto-attach is an open post-merge decision).🤖 Generated with Claude Code