diff --git a/AGENTS.md b/AGENTS.md index 75afba6..d8ad28c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -133,7 +133,7 @@ The 2.0 rewrite lands as underscore-private modules alongside the v1 code. These - **Invariants guard harm, not no-ops**: add a constructor check when violating it produces a *wrong parse*, not when it produces *nothing*. A false positive costs a working configuration; a true positive on an inert condition costs the user nothing, so that trade is never worth taking. `suffix_acronyms_ambiguous ∩ suffix_words` is guarded because the overlap loses a family name; `given_name_titles` is not, because an unreachable entry is simply never consulted (see Gotchas). Before adding one, construct the config it forbids and check what actually breaks. - **The shim TRANSLATES; it never raises on a config v1 accepted, and never silently changes the parse**: `Constants._snapshot()` is a translation boundary between v1's model and v2's invariants, and every transformation there carries its v1-reachability argument in a comment. Six exist today — `first_name_titles` re-folded per word (v1 joins-then-`lc`, v2 normalizes-then-joins), `suffix_acronyms_ambiguous ∩ acronyms` (a provable no-op), `suffix_words − ambiguous` (v1 already accepts the word via the acronym branch, so the addition is inert there), `particles_ambiguous ∪ (bound ∩ particles)` (a pinned deviation, `test_bound_never_given_prefix_deviates_on_two_pieces`), `honorific_tails = GLUED_HONORIFICS ∩ suffix_words` (#308 behavior with no v1 manager of its own, so the one v1 knob that reaches it is deleting the suffix word — which turns the peel off, `test_snapshot_removing_a_honorific_word_turns_the_peel_off`), and `maiden_delimiters − nickname_delimiters` on the POLICY half of the same method (v1 precedence: a pair in both v1 buckets parses as a nickname, while `Policy` resolves the overlap the other way, so the subtraction is what keeps the facade at v1 behavior, `test_snapshot_overlap_keeps_v1_nickname_precedence`). Note that last one is on the `Policy`, not the `Lexicon` — the roster is per-`_snapshot()`, not per-vocabulary-field, so a sweep that only reads the `Lexicon(...)` call misses it. When a v1 config cannot satisfy a v2 invariant, work out what v1 actually *does* with it — usually nothing — and reproduce that; weakening the invariant or letting the raise through are both wrong. **Test the case the translation decides**, not one where both branches agree: a test using an input v1 parses identically with and without the config pins nothing. - **Reprs are bounded**: render which fields deviate from a named baseline and by how much, never contents (`Lexicon(default + titles: +2)`). `PolicyPatch`'s repr shows only set (non-UNSET) fields; `_order_repr` must never raise even on an unvalidated patch's garbage `name_order` (PolicyPatch defers validation to apply time); the sweep test in `tests/v2/test_reprs.py` pins that no config repr leaks the UNSET sentinel. -- **A claim about WHICH STAGE does something is checkable — check it before writing it.** The pipeline is eight stages with a written ownership map (`ParseState`'s docstring, pinned by `tests/v2/pipeline/test_state.py`), and `parse(s).tokens` prints every token's role and tags, so "extract assigns this", "classify never sees that", "group consumes it" each have a one-command answer. #329's prose claimed delimited maiden content is *"claimed whole before classify has tagged anything inside it"*; measured, `classify` tags the marker fine and only the CONSUMING is missing, because `_group`'s rule walks `pieces` and a token that already carries a role is not in `pieces`. Two different mechanisms, one plausible sentence covering both. **And when a mechanism claim turns out wrong, sweep for where else you wrote it**: that one shipped in three places — the release-log entry, `config/maiden_markers.py`'s docstring, and a case-row note — because the correction reached the issue it was found on and nowhere else. Prose density here means one idea routinely lives in a docstring, a case note, a release-log entry and this file; a claim worth writing is worth grepping for when it changes. +- **A claim about WHICH STAGE does something is checkable — check it before writing it.** The pipeline is eight stages with a written ownership map (`ParseState`'s docstring, pinned by `tests/v2/pipeline/test_state.py`), and `parse(s).tokens` prints every token's role and tags, so "extract assigns this", "classify never sees that", "group consumes it" each have a one-command answer. #329's prose claimed delimited maiden content is *"claimed whole before classify has tagged anything inside it"*; measured, `classify` tags the marker fine and only the CONSUMING is missing, because `_group`'s rule walks `pieces` and a token that already carries a role is not in `pieces`. Two different mechanisms, one plausible sentence covering both. **And when a mechanism claim turns out wrong, sweep for where else you wrote it**: that one shipped in three places — the release-log entry, `config/maiden_markers.py`'s docstring, and a case-row note — because the correction reached the issue it was found on and nowhere else. Prose density here means one idea routinely lives in a docstring, a case note, a release-log entry and this file; a claim worth writing is worth grepping for when it changes. It then shipped a fourth and fifth time — the fix's own comment in `_group.py` and its test docstring — written by the commit that closed the bug, after this bullet existed: a wrong mechanism gets reused most readily by the person implementing against it, so the sweep belongs at the END of the change too, over the words the change itself just added. The correction pass then made a FRESH stage error in the very sentence fixing the old one — "extract assigned the whole clause `Role.MAIDEN`", when `extract_delimited` produces no tokens at all (it records `extracted`/`masked` spans; `tokenize` sets the role) — which is this bullet turned on itself: rewriting a mechanism claim is writing one, and earns the same one-command check. - **Typing/docs**: `from __future__ import annotations`; `frozen=True, slots=True` on every public dataclass; strict-profile mypy flags via per-module overrides in pyproject (`strict = true` itself is not valid per-module). Docstrings state contracts in prose with **no doctest blocks** — `--doctest-modules` makes every example a test; behavior examples go to unit tests per the lean-docs rule. **Document the positive direction of a partial property**: "a non-empty `ambiguities` is a signal to act on" is checkable, while "an empty one means no fork occurred" is a universal negative needing exhaustive verification -- that claim was written twice and falsified twice, at sites the author had not audited. - **The segmenter contract**: the optional `Parser(segmenter=...)` hook is parse-totality's ONE exception (locales spec section 4). Everything inside that exception is a bug in USER CODE, never a fact about the name, so it is surfaced rather than absorbed: the segmenter's own exceptions propagate, and the two protocol violations the stage can detect for itself — an answer of the wrong type, and one cutting at or past the end of the token it was handed — raise `TypeError`/`ValueError` from `_script_segment` for the same reason. The line to hold when adding a check there: a protocol violation by the segmenter's AUTHOR raises, while an adapter's defense against its own third-party library (`locales/ja.py`'s repertoire, length, reconstruction and score guards) declines with `None`, because what those catch is a fact about the content. - **Pickling**: v2 types must round-trip (`Parser` is picklable by construction, and it holds a `Lexicon`; the one qualifier is that a `Parser` pickles iff its segmenter does — see the segmenter bullet above). Every frozen type assigns `_guarded_getstate`/`_guarded_setstate` (`_types.py`) in its class body (`@dataclass(slots=True)` would override inherited pickle methods) — unpickling fails at the LOAD site on field-layout skew, and values are deliberately NOT re-validated (pickle is not a security boundary; canonical state comes from a validated instance). `Lexicon` keeps its own copy of the guard (layering) plus the `mappingproxy` slot rebuild; a new unpicklable slot type needs the same treatment plus a round-trip test. @@ -187,7 +187,7 @@ Add a dedicated `copy.deepcopy()` round-trip test for it too (see `test_regexes_ **`_normalize` must reach a fixed point** — storage and match-time share the one fold, and `Lexicon.__setstate__` re-validates, so a value that changes on re-normalization changes under its owner. `strip().strip(".")` alone is not idempotent (`'. a .'` → `' a '` → `'a'`). The loop is the fix; keep any new stripping inside it. **Anything built on `_normalize` must converge too** — `_title_key` joins per-word `_normalize` and DROPS words that fold away; keeping the empty slot stored `'lt .'` as `'lt '`, a key match-time can never rebuild (so the entry is silently inert) and `__setstate__` rejects on the next round-trip as "not written by this version". -**Perf regressions are caught by the scaling test, not the absolute-time ones** — `tests/v2/test_benchmark.py::test_parse_cost_grows_no_worse_than_linearly` times a repeated unit at n vs 4n over ten shapes (one per pipeline inner loop) and bounds the ratio; the `_thousand_names` tests use constant-size, delimiter-free input and are structurally blind to a complexity regression. Two rules when touching it: calibrate `_MAX_RATIO` against the WEAKEST quadratic's signal (a mixed quadratic surfaces far below the textbook 16×, so the operating point `_BASE` matters more than the bound), and confirm a planted regression fails it across REPEATED runs — one failure is a coin-flip on a timing test. The ten shapes cover different dimensions (segment count only via `commas`, intra-piece accumulation only via `particles`/`conjunctions`, non-ASCII input only via `honorifics` — the other nine are pure ASCII, so `script_segment` returns at its bail and the CJK stages go unmeasured); measure before pruning one. +**Perf regressions are caught by the scaling test, not the absolute-time ones** — `tests/v2/test_benchmark.py::test_parse_cost_grows_no_worse_than_linearly` times a repeated unit at n vs 4n over ten shapes (one per pipeline inner loop) and bounds the ratio; the `_thousand_names` tests use constant-size, delimiter-free input and are structurally blind to a complexity regression. Two rules when touching it: calibrate `_MAX_RATIO` against the WEAKEST quadratic's signal (a mixed quadratic surfaces far below the textbook 16×, so the operating point `_BASE` matters more than the bound), and confirm a planted regression fails it across REPEATED runs — one failure is a coin-flip on a timing test. The ten shapes cover different dimensions (segment count only via `commas`, intra-piece accumulation only via `particles`/`conjunctions`, non-ASCII input only via `honorifics` — the other nine are pure ASCII, so `script_segment` returns at its bail and the CJK stages go unmeasured); measure before pruning one. A stage gated on an opt-in `Policy` field needs a `_POLICY_SHAPES` entry instead, since bare `parse()` never enters it — and that table's rows carry a **reachability probe** run before the measurement, because a precedence change can quietly stop the shape reaching the stage and leave a green test measuring a no-op (`_POLICY_SHAPES` is also asserted non-empty: an empty `parametrize` is a skip, not a failure, so deleting its last row would retire the guard silently). **Expected-failure tests use `@pytest.mark.xfail`** — the conftest parametrized fixture breaks `@unittest.expectedFailure`; always use `@pytest.mark.xfail` instead. diff --git a/docs/customize.rst b/docs/customize.rst index 3e9725b..6a9a7b4 100644 --- a/docs/customize.rst +++ b/docs/customize.rst @@ -248,8 +248,15 @@ listed below. * - ``maiden_delimiters`` - ``frozenset[tuple[str, str]]`` - Routes content enclosed by these delimiter pairs to ``maiden`` - instead, and drops them from the effective nickname set. - Defaults to empty — see the routing example below. + instead, and drops them from the effective nickname set. A + marker word opening the enclosed content is dropped from the + value, so ``"Jane Smith (née Jones)"`` gives maiden ``Jones`` + — but only where that content holds more than one *token*, + since a lone ``"(Nee)"`` is a maiden name rather than a + marker. Tokens, not words: a marker written against the name + it marks is one token with them, so ``"山田花子（旧姓佐藤）"`` + keeps its ``旧姓``. Defaults to empty — see the routing + example below. * - ``extra_suffix_delimiters`` - ``frozenset[str]`` - Adds separators that split suffix groups, e.g. ``" - "`` for diff --git a/docs/release_log.rst b/docs/release_log.rst index cc7298a..6404a12 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -33,7 +33,8 @@ Release Log - Fix an ASCII period after a CJK honorific stopping it being recognized: ``씨.``, ``様.``, ``氏.``, ``님.``, ``군.``, ``양.`` and ``殿.`` now route to ``suffix`` like their periodless spellings, where the trailing period had left them inside the name — the family name in ``"김민준 씨."``, the given name in ``"김민준, 씨."``. The cause was v1's initial regex, ``^(\w\.|[A-Z])?$`` (``REGEXES["initial"]``, still public v1 API), whose ``\w`` is Unicode-aware and so matched a hangul syllable or a Han ideograph as readily as a letter; the strict suffix test applies that as a veto (``V.`` in ``"John V. Smith"`` is a middle initial, not roman five), and a veto written for Latin was being asked of scripts it was never about. The cost ran past the honorific itself: because the vetoed token read as name text, the glued-honorific peel's scan back for its site stopped at it instead of stepping over it, took it as the site, found no honorific at the end of it and gave up — so ``"田中さん 様."`` kept ``さん`` inside the name, reading given ``田中さん`` and family ``様.``, while ``"田中さん 様"`` peeled it. The comma form ``"田中さん, 様."`` reached the same scan when this was written; since the #319 entry above it no longer does — its post-comma run is now declined as suffix-shaped before the scan begins, so ``様.`` gets to ``suffix`` through classification rather than by being stepped over. Measured against 1.4.0, ``"김민준, 씨."`` and ``"田中さん, 様."`` were returning exactly what 1.x returns, so the honorific work earlier in this release had a hole in it wherever the honorific was written with a period. An initial is a single LETTER standing in for a name, and Han ideographs, hangul syllables and kana are morphemes and syllables rather than letters, so the veto now asks its question only of the scripts where it means something. Alphabets keep their initials untouched — ``"А. С. Пушкин"``, ``"م. الفارسي"`` and ``"Ա. Խաչատրյան"`` are unaffected, and so is the Ukrainian conjunction entry below, where a punctuated ``Й.`` still outranks the conjunction ``й``. The public ``initial`` tag follows the same line: ``씨.`` no longer carries it. Read ``period`` strictly here: the fix is scoped to the ASCII full stop U+002E, because that is the only period ``_normalize`` strips. The fullwidth U+FF0E and the ideographic U+3002 (with its halfwidth twin U+FF61) — the stops a CJK writer is likelier to type — leave the honorific unmatchable by the vocabulary lookup, which runs before the veto is ever consulted, so ``"김민준 씨．"`` still reads the honorific as the family name. That is a separate, still-open gap in ``_normalize`` rather than in the veto: those spellings parse identically before and after this change, and widening the strip is follow-up work. **Default-on**, and it reaches ``HumanName`` too (#320) - Fix NFD-decomposed input missing the East Asian defaults entirely: script classification now normalizes to NFC before deciding, so a Korean or Japanese name typed on macOS — where decomposed text is routine — gets the same order rule as its composed twin, which it silently did not before. Segmentation MATCHING deliberately stays raw, so an unspaced NFD hangul name is ordered correctly but not split, rather than being split in the wrong place. One gotcha worth stating plainly: parse output preserves the encoding it was given, so for NFD input ``name.family == "김"`` is ``False`` even though it is the same name — compare NFC-normalized text when comparing across encodings (#272) - Fix the Ukrainian conjunction ``й`` not joining the pieces around it: it is the euphonic alternate of ``і``, the two chosen by the surrounding vowel and consonant rather than by meaning (``"Олесь і Олена"`` but ``"Марія й Петро"``), so real Ukrainian data carries both spellings and shipping only ``і`` recognized just one of them. ``"Олесь й Олена Коваленки"`` now gives given ``"Олесь й Олена"`` where the ``й`` previously landed in ``middle``. Same treatment as the ``и``/``і`` entries added in 2.0.0, single-letter carve-out included: the conjunction joins only once the name has enough pieces, and a punctuated initial still wins, so ``"Й. Сліпий"`` is unaffected. Raised in a comment on #267 - - Add the Japanese maiden-name marker ``旧姓`` to the default vocabulary (#309): ``"山田花子 旧姓 佐藤"`` now gives family ``山田花子`` and maiden ``佐藤``, where 1.4.0 left the marker in the name (first ``山田花子``, middle ``旧姓``, last ``佐藤``). It sits beside the Cyrillic ``урожд.`` and German ``geb.`` entries rather than in ``locales.JA``, on the rule that admitted those: a native-script marker cannot collide with a Latin-script name and matching is whole-token, so it is safe as a default and needs no my-data-is-Japanese declaration -- it can only ever match Han text. **Scope worth knowing before you rely on it:** this reaches the SPACED form only. Japanese more often brackets the marker, and ``"山田（旧姓：佐藤）"`` under ``Policy(maiden_delimiters=...)`` still returns maiden ``"旧姓：佐藤"`` with the marker and its colon attached, because delimited content is claimed whole before the marker inside it is classified. That is not a Japanese limitation -- ``"Jane Smith (née Jones)"`` keeps its marker the same way -- and closing it is #329. **Default-on**, and it reaches ``HumanName`` too (#309) + - Add the Japanese maiden-name marker ``旧姓`` to the default vocabulary (#309): ``"山田花子 旧姓 佐藤"`` now gives family ``山田花子`` and maiden ``佐藤``, where 1.4.0 left the marker in the name (first ``山田花子``, middle ``旧姓``, last ``佐藤``). It sits beside the Cyrillic ``урожд.`` and German ``geb.`` entries rather than in ``locales.JA``, on the rule that admitted those: a native-script marker cannot collide with a Latin-script name and matching is whole-token, so it is safe as a default and needs no my-data-is-Japanese declaration -- it can only ever match Han text. **Scope worth knowing before you rely on it:** matching is whole-token, so the marker has to *be* a token — which for Japanese, written without spaces between words, means something has to divide it from the name it marks. A space does, and so does a configured delimiter, the brackets being masked out of the text before it is tokenized: ``"山田（旧姓 佐藤）"`` needs no space in front of ``旧姓``. So the bare ``"山田花子 旧姓 佐藤"`` qualifies, and so does the bracketed ``"山田 花子（旧姓 佐藤）"`` under ``Policy(maiden_delimiters=...)``, once the entry below closed #329. What divides nothing is the fullwidth colon that the spelling Japanese more often uses puts after the marker: ``"山田（旧姓：佐藤）"`` still returns maiden ``"旧姓：佐藤"`` with the marker and its colon attached. Not because delimited content escapes classification — a marker is tagged wherever it is a token — but because that colon is no separator, so marker and name arrive as a single token with nothing to divide; the wholly unspaced ``"山田花子（旧姓佐藤）"`` reads as one token for the same reason. Peeling a marker off the head of a token is #317's job. **Default-on**, and it reaches ``HumanName`` too (#309) + - Fix a maiden marker inside bracketed content staying in the ``maiden`` value: where a delimiter pair is routed to ``maiden`` by ``Policy(maiden_delimiters=...)``, a marker word at the head of the bracketed clause is now dropped the way it has always been dropped in the bare form, so ``"Jane Smith (née Jones)"`` gives maiden ``Jones`` where it gave ``née Jones`` before — the same answer as the unbracketed ``"Jane Smith née Jones"``. The Japanese spelling moves with it: ``"山田 花子（旧姓 佐藤）"`` gives maiden ``佐藤``. Nothing else about either name changes — the marker leaves the ``maiden`` value without turning up anywhere else, the way it already vanished from the bare form, and every other field reads exactly as before. ``maiden`` is in fact the only field that ever differs, which is measured rather than reasoned. The cause was never that the marker went unrecognized — a maiden marker is classified wherever it stands as a token, inside brackets as anywhere else. What was missing was the *consuming*: bracketed content is claimed as a region before the name is tokenized at all, and the tokens cut inside it are born already holding the maiden role, which keeps them out of the grouped runs the marker rule walks, so that rule could not reach a marker sitting in one. The new pass is scoped to the delimited clause, keyed on the extraction spans rather than on a maiden token's neighbours, and both halves of that are load-bearing. A maiden role is not proof of extraction — the bare rule sets it too — so a neighbour test would fire on the bare path and eat the surname out of ``"Jane Smith nee Nee Jones"``, which still gives maiden ``Nee Jones``. And separate clauses are separate content: ``"Jane Smith (Nee) (Jones)"`` also still gives maiden ``Nee Jones``, because the drop takes a clause's first token only where the clause holds more than one — ``Nee`` is a real surname (Irish Ní/Nee, and a Chinese romanization), so a one-token clause is a name rather than a marker, and ``"Jane Smith (née)"`` keeps its ``née`` for the same reason. Each clause loses its own leading marker, so ``"Jane Smith (née Jones) (geb Braun)"`` gives maiden ``Jones Braun``. One edge deserves naming because what it changes is a name's truthiness rather than a field: where a maiden clause is the entire input and everything in it besides the marker is punctuation, dropping the marker leaves nothing carrying an alphanumeric character, and 2.0's standing rule that such input is not a name empties the parse — ``"(née —)"`` returned maiden ``"née —"`` and is now falsy throughout. That follows from treating the marker as structural, the same reading under which ``"(-)"`` has always come back empty, and it takes a name that is nothing but the clause: ``"Jane Smith (née —)"`` still gives given ``Jane``, family ``Smith``, maiden ``—``. Scope before you count on it: ``Policy.maiden_delimiters`` is **empty by default**, so this reaches only callers who have opted a pair into it — under the default policy brackets route to ``nickname`` and nothing here applies. ``HumanName`` reaches it only through the v1 bucket *move* — ``maiden_delimiters['parenthesis'] = nickname_delimiters.pop('parenthesis')`` — since v1 precedence gives a pair held in both buckets to ``nickname``, and the facade preserves that; with the move, ``HumanName("Jane Smith (née Jones)")`` gives maiden ``Jones`` as well. And the agreement with the bare form is not total: the Japanese form written with a fullwidth colon, ``"山田（旧姓：佐藤）"``, still returns maiden ``"旧姓：佐藤"``, since the colon leaves marker and name a single token with nothing to separate — that one wants the head-peel #317 tracks, and the entry above says the rest of it (closes #329) **Documentation** diff --git a/nameparser/_parser.py b/nameparser/_parser.py index d7d73a7..2e104d1 100644 --- a/nameparser/_parser.py +++ b/nameparser/_parser.py @@ -127,7 +127,11 @@ def revise(self, name: ParsedName, **fields: str) -> ParsedName: role choices and ambiguities are discarded -- every harvested token takes the named field's role -- and its structural behavior applies: delimiter characters do not become tokens, - and a mid-value maiden marker is consumed as in parsing. + and a maiden marker is consumed as in parsing -- mid-value + always, and leading a DELIMITED value under a policy routing + that pair to maiden, where "(née Jones)" revises to "Jones" + while the bare "née Jones" keeps its marker, a leading marker + in an undelimited value being no marker at all (#329). Tokens are synthetic (span=None); original is unchanged; a value with no name content (empty, whitespace, or punctuation only) clears the field; ambiguities referencing replaced diff --git a/nameparser/_pipeline/_assemble.py b/nameparser/_pipeline/_assemble.py index f811166..a29f366 100644 --- a/nameparser/_pipeline/_assemble.py +++ b/nameparser/_pipeline/_assemble.py @@ -30,13 +30,31 @@ def assemble(state: ParseState) -> ParsedName: continue role = t.role if t.role is not None else Role.GIVEN final[i] = Token(t.text, t.span, role, t.tags) - # No alphanumeric character anywhere means no name: a bare '.' or - # '- -' is not a person. v1 kept such input (parse('.') -> first - # '.'); 2.0 empties it so bool() stays an honest "did I get a name?" - # check. isalnum() is Unicode-aware, so every real name in any - # script has content and only pure punctuation/symbols empty out. - # (Embedded junk in a name with content -- 'John . Smith' -- is left - # alone: that parse is truthy, so bool() is not misled.) + # No alphanumeric character among the SURVIVING tokens means no + # name: a bare '.' or '- -' is not a person. v1 kept such input + # (parse('.') -> first '.'); 2.0 empties it so bool() stays an + # honest "did I get a name?" check. isalnum() is Unicode-aware, so + # every real name in any script has content and only pure + # punctuation/symbols empty out. (Embedded junk in a name with + # content -- 'John . Smith' -- is left alone: that parse is truthy, + # so bool() is not misled.) + # + # This read "anywhere" until #329, and the two said the same thing + # for as long as nothing could take a content-bearing token away. + # Dropping a maiden marker inside a delimited clause can, and + # "(née —)" under Policy(maiden_delimiters=...) is the input where + # the readings part: the marker is the clause's only alnum token, + # so once it goes structural the em dash is all that survives and + # the whole parse empties -- where 1.4.0 and pre-#329 both gave + # maiden 'née —'. Deliberate, not fallout: a dropped marker is + # structural like a delimiter character, and brackets plus a marker + # plus a dash name no one, exactly as "(-)" already named no one. + # A guard keyed on what ELSE is in the clause would be a different + # rule, and would leave maiden holding marker-plus-punctuation. + # Pinned by cases.py's maiden_marker_delimited_content_free and, + # for the other side of it -- the same clause inside a name, where + # the em dash survives as the maiden value because Jane Smith + # carries the content -- ..._content_free_in_a_name beside it. # # The TOKENS go, not the diagnostics: this drops the name, and # "was the input malformed?" is the one question still worth diff --git a/nameparser/_pipeline/_extract.py b/nameparser/_pipeline/_extract.py index d343586..bf6b82f 100644 --- a/nameparser/_pipeline/_extract.py +++ b/nameparser/_pipeline/_extract.py @@ -4,6 +4,9 @@ Produces: extracted (role + inner span per delimited region), masked (full regions incl. delimiter chars, skipped by tokenize), UNBALANCED_DELIMITER ambiguities for opens with no close. +A Role.MAIDEN region is the WHOLE inner span, marker word included -- +nothing here strips one. classify tags a marker inside it like any +other token, and group drops it from a multi-token clause (#329). Reads: Policy.nickname_delimiters, Policy.maiden_delimiters. Matching rules (the #273 mechanism): one left-to-right scan over the diff --git a/nameparser/_pipeline/_group.py b/nameparser/_pipeline/_group.py index 6a29740..fd4cadd 100644 --- a/nameparser/_pipeline/_group.py +++ b/nameparser/_pipeline/_group.py @@ -1,6 +1,8 @@ """Stage: group. -Consumes: tokens (classified), segments, structure. +Consumes: tokens (classified), segments, structure, extracted (the +role + inner span per delimited region, for the #329 pass below -- +the only stage after tokenize that reads it). Produces: pieces + piece_tags per segment (runs of token indices -- tokens are NEVER joined into strings: the anti-#100 invariant); maiden tail tokens get role=MAIDEN; marker tokens land in dropped. @@ -12,13 +14,16 @@ segments drop delimiter-core tokens (v1 suffix_delimiter parity). Ports v1's join_on_conjunctions + prefix chains + _join_bound_first_name -plus two additions: the "Ph. D."-split merge (v1 fix_phd, recorded plan -deviation #1) and the maiden-marker consuming rule (#274: marker plus +plus three additions: the "Ph. D."-split merge (v1 fix_phd, recorded +plan deviation #1), the maiden-marker consuming rule (#274: marker plus following pieces until a suffix become maiden; the marker itself is -structural, like a delimiter char, and is dropped from assembly). +structural, like a delimiter char, and is dropped from assembly), and +the same marker dropped inside EXTRACTED maiden content (#329), which +#274 cannot reach because extract's content never enters pieces. """ from __future__ import annotations +import bisect import dataclasses from collections.abc import Sequence, Set from enum import IntEnum @@ -339,6 +344,64 @@ def group(state: ParseState) -> ParseState: ptags[m:j] = [] all_pieces.append(tuple(tuple(p) for p in pieces)) all_ptags.append(tuple(frozenset(t) for t in ptags)) + # A marker inside EXTRACTED maiden content (#329). classify tags + # such a marker like any other token -- what the #274 rule above + # lacks is not the TAG but the token: extract claims a delimited + # clause and tokenize gives its tokens Role.MAIDEN up front, so + # segment (main stream = role is None) leaves them out of every + # segment, they never enter `pieces`, and a rule that walks pieces + # cannot reach them. + # + # Scoped to the CLAUSE, via state.extracted (one role + inner span + # per delimited region), rather than to a maiden token's + # neighbours. Both reasons are load-bearing: + # * Role.MAIDEN is not proof of extraction -- the #274 rule above + # sets it too, on the bare form, earlier in this same function. + # A neighbour test would fire there and eat the 'Nee' out of + # "Jane Smith nee Nee Jones". Keying on extracted spans puts + # the bare path out of reach by construction. + # * Separate clauses are separate content. In "(Nee) (Jones)" the + # two land as one contiguous run of maiden tokens, so only the + # clause bound keeps the lone "(Nee)" intact. + # Drop the clause's FIRST token only when the clause holds more + # than one: `Nee` is a real surname (Irish Ni/Nee, and a Chinese + # romanization), so a one-token "(Nee)" is a maiden name, not a + # marker. FIRST token and no more, whatever the clause holds past + # it: cases.py's maiden_marker_delimited_three_token_clause is the + # row that bounds this in both directions, every other delimited + # row having a two-token clause where the two readings agree. + # Spans index the original string by the anti-#100 + # invariant, and script_segment only ever splits a token into + # sub-slices, so containment stays exact. + # + # Bisect rather than scan the token list per clause: that is + # quadratic in the number of delimited pairs, and "(a) " * 3200 -- + # 4x test_benchmark's base, NOT a doubling -- measured a 14.1x cost + # against the 4.1x the same shape holds under a policy with no + # maiden_delimiters. The control is what says which unit a ratio is + # in: linear is ~4 for 4x the input and ~2 for a doubling, so a 4.1x + # control cannot be per doubling. Re-measured 2026-08-03 at 11.2x + # against 4.2x (3.2x against 2.1x per doubling) -- the separation + # replicates, the exact ratio moves with the runner. Same idiom, + # and the same reason, as _extract._overlaps and _tokenize's origin + # resolution. test_benchmark's maiden_pairs shape is the guard. + if any(role is Role.MAIDEN for role, _ in state.extracted): + starts = [t.span.start for t in tokens] + for role, clause in state.extracted: + if role is not Role.MAIDEN: + continue + # first token starting at or after the clause opens; tokens + # are span-sorted and group never reorders or resizes them + first = bisect.bisect_left(starts, clause.start) + # Testing the SECOND token's end proves BOTH are inside: + # tokens do not overlap, so first.end <= second.start, and + # bisect already put first.start at or after clause.start. + # That is also the "more than one token" test, since the + # tokens inside a clause are contiguous in index order. + if (first + 1 < len(tokens) + and tokens[first + 1].span.end <= clause.end + and "vocab:maiden-marker" in tokens[first].tags): + dropped.append(first) return dataclasses.replace( state, tokens=tuple(tokens), pieces=tuple(all_pieces), piece_tags=tuple(all_ptags), dropped=tuple(dropped), diff --git a/nameparser/_policy.py b/nameparser/_policy.py index ed5e925..833433c 100644 --- a/nameparser/_policy.py +++ b/nameparser/_policy.py @@ -623,6 +623,9 @@ class Policy: #: field instead; a pair listed here is dropped from the effective #: nickname set (maiden wins, see __post_init__), so #: maiden_delimiters={("(", ")")} is the whole recipe (#274). + #: A maiden_markers word opening the enclosed content is dropped + #: from the value, but only where that content holds more than one + #: token: a lone "(Nee)" is a maiden NAME, not a marker (#329). maiden_delimiters: frozenset[tuple[str, str]] = frozenset() #: Additional separators that split suffix groups (e.g. " - " for #: "Jane Smith, RN - CRNA"). Additive only: the comma always diff --git a/nameparser/config/maiden_markers.py b/nameparser/config/maiden_markers.py index 0f7c82a..6014384 100644 --- a/nameparser/config/maiden_markers.py +++ b/nameparser/config/maiden_markers.py @@ -39,14 +39,23 @@ where a pure-Han string cannot say which language wrote it -- and this needs none, since it can only ever match Han text. -It reaches the SPACED form only ("山田花子 旧姓 佐藤" gives maiden -佐藤). Japanese more often writes the marker inside brackets, and -"山田（旧姓：佐藤）" under Policy(maiden_delimiters=...) still yields -maiden "旧姓：佐藤" with the marker and its colon attached: extract -claims delimited content whole, before classify has tagged anything -inside it, so group's marker-consuming rule never sees it. That -asymmetry is not Japanese -- "Jane Smith (née Jones)" keeps its marker -the same way -- and closing it is #329. +Matching being whole-token, the marker has to BE a token -- which for +Japanese means something has to divide it from the name it marks. A +space does, and so does a configured delimiter: extract masks the +whole bracketed region, delimiter characters included, before tokenize +runs, so a bracket bounds a token exactly as a space does and +"山田（旧姓 佐藤）" needs no space in front of 旧姓 at all. The bare +"山田花子 旧姓 佐藤" and -- since #329 -- the bracketed +"山田 花子（旧姓 佐藤）" under Policy(maiden_delimiters=...) alike give +maiden 佐藤. What divides nothing is the fullwidth colon that the form +Japanese more often writes puts after the marker: "山田（旧姓：佐藤）" +still yields maiden "旧姓：佐藤" with the marker and its colon attached. +Not because delimited content escapes classification -- classify tags +a marker wherever it is a token -- but because ： is no separator +tokenize knows, so marker and name arrive as ONE token and there is +nothing to drop. The wholly unspaced "山田花子（旧姓佐藤）" reads as one +token for the same reason. Peeling a marker off the head of a token is +#317's job. Consumed by the 2.0 parser's default lexicon. The 1.x parser does not read this module. diff --git a/tests/test_nicknames.py b/tests/test_nicknames.py index 2034a04..e720765 100644 --- a/tests/test_nicknames.py +++ b/tests/test_nicknames.py @@ -246,6 +246,45 @@ def test_suffix_shaped_content_in_maiden_bucket_stays_in_place(self) -> None: self.m(hn.suffix, "Jr.", hn) self.m(hn.maiden, "", hn) + def test_marker_inside_maiden_parenthesis_is_consumed(self) -> None: + # #329 through the v1 API, which is where the release log + # promises it. The facade runner covers the same ground by + # translating the shared case table, but it decides for itself + # which rows it can express -- when that gate was wrong these + # rows silently skipped, so a test that spells the v1 idiom out + # directly is what keeps the promise pinned independently of it. + # Same value as the bare "Jane Smith née Jones", which is the + # agreement #329 was about. + C = Constants() + C.maiden_delimiters['parenthesis'] = C.nickname_delimiters.pop('parenthesis') + hn = HumanName("Jane Smith (née Jones)", constants=C) + self.m(hn.first, "Jane", hn) + self.m(hn.last, "Smith", hn) + self.m(hn.maiden, "Jones", hn) + + def test_unmarked_maiden_parenthesis_keeps_every_word(self) -> None: + # The other side of the same rule: the drop is conditioned on + # the first word being a marker, so ordinary two-word content + # arrives whole. + C = Constants() + C.maiden_delimiters['parenthesis'] = C.nickname_delimiters.pop('parenthesis') + hn = HumanName("Jane Smith (Mary Jones)", constants=C) + self.m(hn.first, "Jane", hn) + self.m(hn.last, "Smith", hn) + self.m(hn.maiden, "Mary Jones", hn) + + def test_lone_marker_word_in_its_own_parenthesis_is_a_name(self) -> None: + # `Nee` is a real surname (Irish Ní/Nee, and a Chinese + # romanization), so a one-word clause is content whatever it + # spells -- even with a second clause following that could read + # as the name it marks. + C = Constants() + C.maiden_delimiters['parenthesis'] = C.nickname_delimiters.pop('parenthesis') + hn = HumanName("Jane Smith (Nee) (Jones)", constants=C) + self.m(hn.first, "Jane", hn) + self.m(hn.last, "Smith", hn) + self.m(hn.maiden, "Nee Jones", hn) + def test_maiden_appears_in_as_dict_via_routing(self) -> None: C = Constants() C.maiden_delimiters['parenthesis'] = C.nickname_delimiters.pop('parenthesis') diff --git a/tests/v2/cases.py b/tests/v2/cases.py index e659421..c1b33f7 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -193,9 +193,12 @@ def __post_init__(self) -> None: policy=Policy(maiden_delimiters=frozenset({("(", ")")})), notes="listing a pair in maiden_delimiters drops it from the " "effective nickname set (maiden wins, 2026-07-19) -- the " - "one-liner replaces the bucket-move idiom; the v1 facade " - "keeps v1's nickname-wins precedence via the shim's " - "pre-subtraction (pinned in test_config_shim)"), + "one-liner replaces the bucket-move idiom. The facade " + "runner reaches this row by making that same move, so " + "it agrees. What v1 does NOT agree on is a pair left in " + "BOTH buckets, which it gives to nickname; that spelling " + "is a different config, not this row, and is pinned in " + "test_config_shim"), # #269 follow-up: Arabic-script bound given names, mirroring the # Latin transliterations' behavior (probed live 2026-07-19: bound # join fires only with 3+ tokens, eats the NEXT token into given; @@ -434,13 +437,19 @@ def __post_init__(self) -> None: notes="旧姓 is default vocabulary, not pack data: a " "native-script marker cannot collide with a Latin-script " "name and matching is whole-token, the same rule that " - "admitted урожд. Reaches the SPACED form only -- Japanese " - "more often brackets the marker, and '山田（旧姓：佐藤）' " - "under maiden_delimiters still gives maiden '旧姓：佐藤', " - "marker and colon attached, because extract claims " - "delimited content before classify tags anything inside " - "it. Not a Japanese problem: 'Jane Smith (née Jones)' " - "keeps its marker the same way (#329). 1.4.0 read this " + "admitted урожд. Reaches a marker that is its own TOKEN. " + "Japanese more often brackets the marker and writes a " + "fullwidth colon after it, and '山田（旧姓：佐藤）' under " + "maiden_delimiters still gives maiden '旧姓：佐藤', marker " + "and colon attached -- not because the marker escapes " + "tagging (classify tags it fine wherever it is a token; " + "what #329 fixed was the CONSUMING, since group's #274 " + "rule walks pieces and a role-bearing token is not in " + "pieces) but because ：glues marker to name into a " + "single token, leaving nothing to drop. The spaced " + "bracketed form is pinned by " + "maiden_marker_kyusei_delimited below; the glued one " + "wants the head-peel #317 tracks. 1.4.0 read this " "first 山田花子 / middle 旧姓 / last 佐藤 -- the marker " "sat in the name"), Case("maiden_marker_kyusei_segmented", "山田 花子 旧姓 佐藤", @@ -453,6 +462,233 @@ def __post_init__(self) -> None: "pieces before it are what could have gone wrong. " "1.4.0 read this first 山田 / middle '花子 旧姓' / " "last 佐藤"), + Case("maiden_marker_delimited", "Jane Smith (née Jones)", + {"given": "Jane", "family": "Smith", "maiden": "Jones"}, + policy=Policy(maiden_delimiters=frozenset({("(", ")")})), + classification="fix(#329)", + notes="#329: the bracketed form now agrees with the bare " + "maiden_marker row above -- both give maiden 'Jones'. " + "Before, the marker rode along inside the value: " + "extract records the clause as a span (it makes no " + "tokens at all), tokenize gives the tokens it cuts " + "there Role.MAIDEN, and group's #274 consuming rule " + "walks pieces, which hold no role-bearing token; the " + "fix drops the marker inside the CLAUSE instead. The " + "facade runner reaches this row by performing the " + "bucket move itself, so it is exercised twice. 1.4.0 " + "expresses the policy the same way: " + "measured 2026-08-02 through the bucket-move idiom " + "maiden_delimiters['parenthesis'] = " + "nickname_delimiters.pop('parenthesis'), it gave first " + "Jane / last Smith / maiden 'née Jones' -- same name " + "fields, marker still inside the value, which is the " + "single field this change moves"), + Case("maiden_marker_delimited_unaccented", "Jane Smith (nee Jones)", + {"given": "Jane", "family": "Smith", "maiden": "Jones"}, + policy=Policy(maiden_delimiters=frozenset({("(", ")")})), + classification="fix(#329)", + notes="the row above with the marker spelled unaccented, and " + "it is the ONLY row in the suite whose value depends " + "on 'nee' being in the default MAIDEN_MARKERS: " + "everything else reaches the marker branch through " + "'née', 'geb' or '旧姓', or through a stage lexicon of " + "its own. Removing the entry now fails exactly this " + "row's two tests, one per runner (measured 2026-08-03); " + "before this row existed it left the whole suite green, " + "so the shipped spelling English writes most often was " + "one vocabulary edit from silence. 1.4.0 under the " + "bucket-move idiom gave first Jane / last Smith / " + "maiden 'nee Jones' (2026-08-03) -- the same diff the " + "accented row records, which is the point: the two " + "spellings behave alike on both sides"), + Case("maiden_marker_delimited_unmarked_content", + "Jane Smith (Mary Jones)", + {"given": "Jane", "family": "Smith", "maiden": "Mary Jones"}, + policy=Policy(maiden_delimiters=frozenset({("(", ")")})), + classification="parity", + notes="the row the rest of the #329 battery leaves out: a " + "multi-token clause whose first token is NOT a marker, " + "which keeps every one of its tokens. The clause-size " + "test and the marker-tag test are separate conditions, " + "and this shape is one of the two that separates them " + "-- with the tag test removed the pass eats the opening " + "word of every delimited maiden name ('Jones' here), " + "and five tests go red across this row and " + "maiden_marker_delimited_trailing_marker (measured " + "2026-08-03). What is this row's alone is that NO token " + "in its clause is a marker; the trailing-marker row has " + "one, just not first. Measured against " + "1.4.0 2026-08-03 through the bucket-move idiom " + "maiden_delimiters['parenthesis'] = " + "nickname_delimiters.pop('parenthesis'): first Jane / " + "last Smith / maiden 'Mary Jones', so #329 leaves this " + "input exactly where v1 had it"), + Case("maiden_marker_delimited_three_token_clause", + "Jane Smith (née Mary Jones)", + {"given": "Jane", "family": "Smith", "maiden": "Mary Jones"}, + policy=Policy(maiden_delimiters=frozenset({("(", ")")})), + classification="fix(#329)", + notes="the only clause in the battery holding THREE tokens, " + "which is what bounds the drop in both directions: it " + "takes the marker and stops. Every other delimited row " + "has a two-token clause, where 'the first token' and " + "'all but the last token' agree, so two opposite " + "mistakes both survive them -- restricting the drop to " + "a clause of exactly two tokens gives maiden 'née Mary " + "Jones' here (marker never dropped), and letting it eat " + "the token after the marker gives maiden 'Jones' " + "(a name eaten). Both measured 2026-08-03. 1.4.0 under " + "the bucket-move idiom " + "maiden_delimiters['parenthesis'] = " + "nickname_delimiters.pop('parenthesis') gave first Jane " + "/ last Smith / maiden 'née Mary Jones' (2026-08-03) -- " + "marker inside the value, the single field #329 moves"), + Case("maiden_marker_delimited_trailing_marker", + "Jane Smith (Jones née)", + {"given": "Jane", "family": "Smith", "maiden": "Jones née"}, + policy=Policy(maiden_delimiters=frozenset({("(", ")")})), + classification="parity", + notes="the drop takes the clause's FIRST token or nothing: a " + "marker anywhere else in the clause is content. Pinned " + "because the cheap generalization -- drop every marker " + "in the clause -- passes the whole battery above and " + "gives maiden 'Jones' here, and because no marker the " + "shipped vocabulary carries is written after the name " + "it marks. Measured against 1.4.0 2026-08-03 " + "through the bucket-move idiom: first Jane / last " + "Smith / maiden 'Jones née'"), + Case("maiden_marker_delimited_beside_a_nickname_clause", + 'Jane "née Janie" Smith {née Jones}', + {"given": "Jane", "family": "Smith", "nickname": "née Janie", + "maiden": "Jones"}, + policy=Policy(maiden_delimiters=frozenset({("{", "}")})), + classification="fix(#329)", + notes="the pass is scoped to MAIDEN clauses, and this is the " + "row that says so: two extracted clauses, both opening " + "with a marker word, and only the maiden one loses it. " + "Braces route to maiden here precisely so the default " + "nickname set survives untouched -- the parenthesis " + "rows above cannot show this, since Policy's " + "maiden-wins canonicalization would take ( ) away from " + "nickname. Without the role filter the nickname reads " + "'Janie'. 1.4.0 cannot express a brace delimiter at " + "all (its buckets hold the NAMES of compiled regexes " + "and there is no brace one; measured 2026-08-03, " + "maiden_delimiters['brace'] = ('{', '}') is accepted " + "and then raises ValueError('references unknown " + "regexes key') at parse time), so the " + "classification compares against its single reading, " + "first Jane / middle 'Smith {née' / last 'Jones}' / " + "nickname 'née Janie' -- braces as name text, the same " + "convention maiden_marker_kyusei_delimited uses for a " + "knob with no v1 spelling. The nickname agreed even " + "there"), + Case("maiden_marker_delimited_two_clauses", + "Jane Smith (Nee) (Jones)", + {"given": "Jane", "family": "Smith", "maiden": "Nee Jones"}, + policy=Policy(maiden_delimiters=frozenset({("(", ")")})), + classification="parity", + notes="the scoping pin for #329, and the row a simplification " + "would break: the drop is CLAUSE-scoped, so a one-token " + "clause keeps its token even when the next clause could " + "read as the name it marks. A neighbour-scoped rule -- " + "drop a marker whose successor is also maiden -- gives " + "'Jones' here, eating a real surname (Irish Ní/Nee, and " + "a Chinese romanization). Unaccented 'nee' is in the " + "default MAIDEN_MARKERS, so the 'Nee' token really is " + "tagged and the CLAUSE bound is the only thing keeping " + "it -- which is what makes this row kill a rule that " + "drops the bound. The VALUE does not depend on that " + "vocabulary entry, though: the clause test is checked " + "before the tag test, so 'nee' leaving MAIDEN_MARKERS " + "would leave this expectation green. " + "maiden_marker_delimited_unaccented above is the row " + "that fails when it goes. " + "Parity is measured, not inferred from the row being " + "untouched: 1.4.0 under the same bucket move gave first " + "Jane / last Smith / maiden 'Nee Jones' (2026-08-02), " + "so the two clauses joined with a space on that side " + "too -- the classification the facade runner checks " + "against, since it expresses this policy through the " + "same bucket move"), + Case("maiden_marker_delimited_content_free", "(née —)", + {}, + policy=Policy(maiden_delimiters=frozenset({("(", ")")})), + classification="fix(#329)", + notes="the drop can empty the WHOLE parse, and that is a " + "decision rather than fallout. assemble's content test " + "runs over the SURVIVING tokens, so once the marker " + "goes structural the em dash is the only one left, no " + "alnum character remains and every field clears -- " + "bool() False. Reachable only where a maiden clause is " + "the entire input and its non-marker tokens are pure " + "punctuation -- the same clause inside a name is " + "maiden_marker_delimited_content_free_in_a_name below. " + "Coherent with the model 2.0 " + "already had: a dropped marker is structural like a " + "delimiter character, and '(-)' empties on both sides " + "of this change. Structurally unreachable on the bare " + "#274 path, whose scan starts at piece 1, so a token " + "always survives ahead of the marker ('née —' gives " + "given 'née', family '—'). Do NOT restore the old value " + "with a guard on what else the clause holds: that is a " + "different rule, and it would leave maiden holding " + "marker-plus-punctuation. fix rather than parity " + "because 1.4.0 CAN express this policy and disagrees: " + "measured 2026-08-03 through the bucket-move idiom " + "maiden_delimiters['parenthesis'] = " + "nickname_delimiters.pop('parenthesis'), it gave maiden " + "'née —' and a truthy name, as pre-#329 did. The 2.0 " + "content rule already deviated from 1.4.0 here ('(-)' " + "is maiden '-' in 1.4.0); this change moves one more " + "input into its reach"), + Case("maiden_marker_delimited_content_free_in_a_name", + "Jane Smith (née —)", + {"given": "Jane", "family": "Smith", "maiden": "—"}, + policy=Policy(maiden_delimiters=frozenset({("(", ")")})), + classification="fix(#329)", + notes="the row above with a name in front of the clause, " + "which is what bounds the emptying: assemble's content " + "test is about the WHOLE parse, so a clause of " + "marker-plus-punctuation empties only a name that is " + "nothing else. Here Jane Smith carries the alnum " + "content and maiden keeps the em dash. Pinned because " + "the drop could plausibly have been widened to take the " + "clause's punctuation with the marker -- that mutation " + "gives maiden '' here (measured 2026-08-03) and leaves " + "the row above green, since both readings empty a parse " + "that is only the clause. 1.4.0 under the bucket-move " + "idiom gave first Jane / last Smith / maiden 'née —' " + "(2026-08-03)"), + Case("maiden_marker_kyusei_delimited", "山田 花子（旧姓 佐藤）", + {"given": "花子", "family": "山田", "maiden": "佐藤"}, + policy=Policy( + maiden_delimiters=frozenset({("(", ")"), ("（", "）")})), + classification="fix(#329)", + notes="the Japanese bracketed form that #329 reaches: the " + "marker is spaced off inside fullwidth brackets, so it " + "is a token of its own and the clause-scoped drop " + "applies. The form Japanese more often writes puts a " + "fullwidth colon after the marker instead, and " + "'山田（旧姓：佐藤）' is ONE token -- nothing reaches it, " + "and it wants the head-peel #317 tracks (see " + "maiden_marker_kyusei above). Unlike its two Latin " + "siblings, 1.4.0 cannot express this policy at all: v1's " + "delimiter buckets hold the NAMES of compiled regexes, " + "and no fullwidth pair is among them (#273 added it), so " + "maiden_delimiters['fullwidth_parenthesis'] = ('（', '）') " + "raises ValueError('references unknown regexes key') at " + "parse time. The classification therefore compares " + "against 1.4.0's single reading, first 山田 / middle " + "'花子（旧姓' / last '佐藤）' -- the brackets were name " + "text -- the same convention " + "ko_honorific_period_under_strict_comma_suffixes uses " + "for a knob with no v1 spelling. That reading is also " + "what the differential harness sees, since it runs the " + "corpus under the DEFAULT policy where （） is a #273 " + "NICKNAME delimiter and nothing in #329 is reachable; " + "the diff is classified there under " + "fix(cjk-fullwidth-paren-nickname)"), Case("east_slavic", "Сидоров Иван Петрович", {"given": "Иван", "middle": "Петрович", "family": "Сидоров"}, policy=_ES), diff --git a/tests/v2/pipeline/test_assemble.py b/tests/v2/pipeline/test_assemble.py index 6e48b26..9d67fa4 100644 --- a/tests/v2/pipeline/test_assemble.py +++ b/tests/v2/pipeline/test_assemble.py @@ -53,10 +53,13 @@ def test_empty_parse_is_falsy() -> None: def test_content_free_input_parses_to_empty() -> None: - # An input with no alphanumeric character anywhere is not a name. - # v1 kept it (parse('.') -> first='.'); 2.0 empties it so bool() - # stays an honest "did I get a name?" check. isalnum is - # Unicode-aware, so this only fires on pure punctuation/symbols. + # An input with no alphanumeric character is not a name. v1 kept + # it (parse('.') -> first='.'); 2.0 empties it so bool() stays an + # honest "did I get a name?" check. isalnum is Unicode-aware, so + # this only fires on pure punctuation/symbols. The test is over the + # SURVIVING tokens, which since #329 is not the same as over the + # input -- see cases.py's maiden_marker_delimited_content_free, + # where dropping a marker is what leaves nothing behind. for junk in [".", ".,", "- -", ". .", "'", "∫≜⩕", "()", "-"]: pn = _parse(junk) assert not pn, f"{junk!r} should be falsy" diff --git a/tests/v2/pipeline/test_group.py b/tests/v2/pipeline/test_group.py index 654fa80..7f3208f 100644 --- a/tests/v2/pipeline/test_group.py +++ b/tests/v2/pipeline/test_group.py @@ -2,10 +2,11 @@ from nameparser._pipeline._classify import classify from nameparser._pipeline._extract import extract_delimited from nameparser._pipeline._group import group +from nameparser._pipeline._script_segment import script_segment from nameparser._pipeline._segment import segment from nameparser._pipeline._state import ParseState from nameparser._pipeline._tokenize import tokenize -from nameparser._policy import Policy +from nameparser._policy import Policy, Script from nameparser._types import Role _LEX = Lexicon( @@ -123,6 +124,96 @@ def test_leading_marker_is_not_consumed() -> None: assert _piece_texts(out) == [["née", "Jones"]] +_MAIDEN_PARENS = Policy(maiden_delimiters=frozenset({("(", ")")})) +#: `nee` a marker AND `Nee` a surname in one vocabulary -- the collision +#: the clause-size guard exists for. _LEX alone leaves `Nee` untagged, +#: which would let the guard tests pass under every mutant. +_NEE_LEX = _LEX.add(maiden_markers=frozenset({"nee"})) + + +def test_delimited_marker_is_dropped() -> None: + """#329: the marker IS tagged -- classify reaches it like any other + token. What it never enters is `pieces`: extract claims the clause + and its tokens carry Role.MAIDEN from tokenize, so segment keeps + them out of the main stream. The #274 rule above walks pieces, so + the tag alone does it no good.""" + out = _grouped("Jane Smith (née Jones)", policy=_MAIDEN_PARENS) + maiden = [t.text for i, t in enumerate(out.tokens) + if t.role is Role.MAIDEN and i not in out.dropped] + assert maiden == ["Jones"] + née_idx = next(i for i, t in enumerate(out.tokens) if t.text == "née") + assert née_idx in out.dropped + + +def test_lone_delimited_marker_is_kept_when_a_clause_follows() -> None: + """The clause-size guard, and it is load-bearing rather than + defensive: `Nee` is a real surname (Irish Ni/Nee, and a Chinese + romanization), so a one-token clause is a maiden NAME, not a marker. + + The trailing "(Jones)" is what makes this pin the guard. With + "(Nee)" alone the marker is also the last token in the string, so a + rule that merely checked for a following token would keep it too + and the mutant would live. Here a token does follow -- only the + CLAUSE bound distinguishes them.""" + out = _grouped("Jane Smith (Nee) (Jones)", policy=_MAIDEN_PARENS, + lexicon=_NEE_LEX) + kept = [t.text for i, t in enumerate(out.tokens) + if t.role is Role.MAIDEN and i not in out.dropped] + assert kept == ["Nee", "Jones"] + + +def test_marker_in_the_bare_form_is_left_to_the_piece_rule() -> None: + """#274 sets Role.MAIDEN on consumed tokens itself, so a maiden role + is NOT proof that extract produced it. Keying this pass on + state.extracted spans is what keeps it off the bare path -- a + neighbour test would eat the `Nee` here, the very surname the guard + above exists to protect. + + Reads inert and is not: deleting the #329 pass outright leaves this + green, because the value comes from #274's piece rule and the pass + never touches the bare path. What it kills is a SPELLING of the + pass -- drop a maiden marker whose next token is also maiden, the + form this fix originally took -- and nothing else here covers the + bare path against it. Checked both ways 2026-08-03.""" + out = _grouped("Jane Smith nee Nee Jones", lexicon=_NEE_LEX) + kept = [t.text for i, t in enumerate(out.tokens) + if t.role is Role.MAIDEN and i not in out.dropped] + assert kept == ["Nee", "Jones"] + + +def test_every_delimited_marker_is_dropped_not_only_the_first() -> None: + """Two maiden clauses land as ONE contiguous run of maiden-role + tokens, so a rule keyed on the run would strip the first marker and + keep the second. Each clause is scoped separately, so each loses its + own leading marker.""" + out = _grouped("Jane Smith (née Jones) (geb Braun)", + policy=_MAIDEN_PARENS) + kept = [t.text for i, t in enumerate(out.tokens) + if t.role is Role.MAIDEN and i not in out.dropped] + assert kept == ["Jones", "Braun"] + + +def test_clause_containment_survives_script_segmentation() -> None: + """The #329 pass finds the clause's first token by SPAN, and the + comment on it rests that on script_segment only ever cutting a + token into sub-slices. _grouped omits that stage, so this is the + one place the two meet: 王小明 becomes 王 + 小明 before group runs, + which shifts every token index after it while the spans stay + exact, and the marker is still the token the clause drops.""" + lex = Lexicon(surnames=frozenset({"王"}), + maiden_markers=frozenset({"旧姓"})) + policy = Policy(segment_scripts=frozenset({Script.HAN}), + maiden_delimiters=frozenset({("（", "）")})) + state = ParseState(original="王小明（旧姓 李四）", lexicon=lex, + policy=policy) + out = group(classify(script_segment(segment( + tokenize(extract_delimited(state)))))) + assert [t.text for t in out.tokens] == ["王", "小明", "旧姓", "李四"] + kept = [t.text for i, t in enumerate(out.tokens) + if t.role is Role.MAIDEN and i not in out.dropped] + assert kept == ["李四"] + + def test_initials_do_not_count_as_rootnames_for_conjunction_carveout() -> None: # v1 parity: 'J.' is an initial, so total rootnames stay under 4 and # the single-letter conjunction 'y' is treated as an initial, not joined diff --git a/tests/v2/test_benchmark.py b/tests/v2/test_benchmark.py index 50f3d96..122a04d 100644 --- a/tests/v2/test_benchmark.py +++ b/tests/v2/test_benchmark.py @@ -8,15 +8,24 @@ subsumes the other, and the absolute tests are structurally blind to a complexity regression: their workload is a short name with no delimiters, so a stage that goes quadratic in the number of delimiter -pairs stays far inside the one-second bound. Two such quadratics -(_extract's mask-overlap scan, ParsedName's token-subset check) shipped -and were caught in review rather than here -- hence the scaling test. +pairs stays far inside the one-second bound. Three such quadratics +(_extract's mask-overlap scan, ParsedName's token-subset check, and +_group's #329 clause scan) were caught in review rather than here -- +hence the scaling test. + +The third is why _POLICY_SHAPES exists below: it was quadratic in a +shape the scaling test ALREADY had ("(a) "), and still went unseen, +because the stage is gated on an opt-in Policy field that bare parse() +leaves empty. A shape guards nothing if the default policy cannot +reach the code under it. """ import time +from collections.abc import Callable import pytest -from nameparser import parse +from nameparser import Parser, parse +from nameparser._policy import Policy def test_parse_thousand_names_under_a_second() -> None: @@ -110,25 +119,86 @@ def test_facade_thousand_names_under_a_second() -> None: _MAX_RATIO = 6.0 -def _best(text: str, repeats: int = 7) -> float: +def _best(text: str, parse_: Callable[[str], object], + repeats: int = 7) -> float: """Minimum of several runs: on a shared runner the mean carries the noise of whatever else is running, while the minimum approaches the true cost.""" - parse(text) # warm the default parser cache + parse_(text) # warm the parser cache best = float("inf") for _ in range(repeats): start = time.perf_counter() - parse(text) + parse_(text) best = min(best, time.perf_counter() - start) return best -@pytest.mark.parametrize("unit", _SHAPES.values(), ids=list(_SHAPES)) -def test_parse_cost_grows_no_worse_than_linearly(unit: str) -> None: - small = _best(unit * _BASE) - large = _best(unit * (_BASE * _FACTOR)) +def _assert_grows_linearly(unit: str, + parse_: Callable[[str], object]) -> None: + small = _best(unit * _BASE, parse_) + large = _best(unit * (_BASE * _FACTOR), parse_) ratio = large / small assert ratio < _MAX_RATIO, ( f"{unit!r} x{_BASE} took {small * 1e3:.2f}ms, " f"x{_BASE * _FACTOR} took {large * 1e3:.2f}ms -- {ratio:.1f}x for " f"{_FACTOR}x the input, which is superlinear (linear is ~{_FACTOR})") + + +@pytest.mark.parametrize("unit", _SHAPES.values(), ids=list(_SHAPES)) +def test_parse_cost_grows_no_worse_than_linearly(unit: str) -> None: + _assert_grows_linearly(unit, parse) + + +# Shapes that need a NON-DEFAULT POLICY to reach the code they guard. +# Every _SHAPES entry runs bare parse(), and a stage gated on an opt-in +# Policy field is dead there -- #329's clause loop exits on its first +# line when maiden_delimiters is empty, so the quadratic it shipped +# with was invisible to all ten shapes above even though "(a) " was +# already one of them. The unit is the SAME string as delimiter_pairs; +# only the policy differs, which is the whole point. +# +# Calibrated like the others, per the note above: the #329 clause scan +# measured 14.1x at base 800 (against a 4.1x bare-policy control), and +# 4.3x once bisected -- inside the clean column, so _MAX_RATIO did not +# move. Every ratio in this file is for _FACTOR x the input, never per +# doubling; re-measuring the scan on another runner gave 11.2x against +# the same 4.2x control, which is the spread to expect here. +# +# Third element: a REACHABILITY probe, run before the measurement. +# "the shape must reach the code" is the whole premise of this table, +# and it is a premise about precedence, which moves -- route ( ) back +# to nickname and the parse below stops producing a maiden clause, +# leaving this test measuring a no-op at a comfortable 4.2x forever. +# That is the module docstring's failure mode one level up: a guard +# whose subject has quietly left the building. The probe is per shape +# because what "reached" means is per shape. +_POLICY_SHAPES: dict[str, tuple[str, Parser, Callable[[Parser], bool]]] = { + "maiden_pairs": ( + "(a) ", + Parser(policy=Policy(maiden_delimiters=frozenset({("(", ")")}))), + # a maiden clause whose marker the #329 pass consumes: the + # parenthesis pair must route to maiden AND the clause loop + # must run for this to read "b" rather than "" or "née b" + lambda p: p.parse("a (née b)").maiden == "b", + ), +} + + +def test_shape_tables_are_not_empty() -> None: + # pytest turns an EMPTY parametrize into a SKIP, not a failure, so + # deleting the last entry of either table would retire its guard + # into the skip count with nothing going red. _POLICY_SHAPES is + # the nearer risk, holding only shapes whose stage a default parse + # cannot reach at all -- so it gains an entry only when an opt-in + # Policy field turns out to have a scaling cliff behind it. + assert _SHAPES + assert _POLICY_SHAPES + + +@pytest.mark.parametrize("unit,parser,reaches", _POLICY_SHAPES.values(), + ids=list(_POLICY_SHAPES)) +def test_policy_gated_cost_grows_no_worse_than_linearly( + unit: str, parser: Parser, + reaches: Callable[[Parser], bool]) -> None: + assert reaches(parser), "shape no longer reaches the gated stage" + _assert_grows_linearly(unit, parser.parse) diff --git a/tests/v2/test_facade_cases.py b/tests/v2/test_facade_cases.py index b57d19d..da14698 100644 --- a/tests/v2/test_facade_cases.py +++ b/tests/v2/test_facade_cases.py @@ -1,5 +1,7 @@ """Facade runner (migration spec §5): the shared case table asserted through HumanName. Deleted wholesale in 3.0 with the facade.""" +import dataclasses + import pytest from nameparser import Policy @@ -10,13 +12,71 @@ _V1_KEY = {"given": "first", "family": "last"} # identity for the rest -#: the one non-default maiden_delimiters shape the table exercises -#: ("nickname_bucket_wins_when_shared"): expressible in v1 Constants via -#: the delimiter-manager's parenthesis sentinel added to the maiden -#: bucket, leaving the nickname bucket at its (also-parenthesis-holding) -#: default -- see _config_shim._DelimiterManager. +#: The parenthesis maiden shape. A row routing parens to maiden does +#: NOT carry the default nickname set: Policy's maiden-wins +#: canonicalization subtracts the pair from nickname_delimiters at +#: CONSTRUCTION, so the shape to match is the default MINUS the pair. +#: Comparing against the unsubtracted default is what kept this gate +#: false for every row until #329. +#: +#: The v1 side has two spellings and only one of them is equivalent. +#: ADDING parenthesis to the maiden bucket leaves the nickname bucket +#: holding it too, and v1 gives a shared pair to nickname -- measured on +#: "Baker (Johnson), Jenny", that spelling yields nickname Johnson and +#: an empty maiden, the reverse of what maiden_delimiters_win_when_shared +#: asserts. MOVING it (pop from nickname, assign to maiden) yields +#: maiden Johnson and an empty nickname, matching. That is the idiom +#: tests/test_nicknames.py already uses, and the one below. +#: +#: Which rows this leaves skipping is not described here but asserted, +#: by id, in test_core_only_rows_are_the_declared_ones below. _MAIDEN_PARENS = frozenset({("(", ")")}) +#: Policy fields _constants_for does not translate. A row moving one +#: off its default SKIPS: the facade would otherwise run it under the +#: field's inherited default and pass or fail on a knob the row never +#: asked for. Some have no v1 spelling at all -- script_orders and +#: segment_scripts, the v1 surface being frozen -- and the rest have +#: none this runner has ever needed to write. +_UNTRANSLATED = frozenset({ + "name_order", + "script_orders", + "segment_scripts", + "lenient_comma_suffixes", + "strip_emoji", + "strip_bidi", +}) + +#: Expressible in exactly one shape, the parenthesis bucket move. +#: BOTH delimiter fields are listed because the canonicalization that +#: routes the pair to maiden is the same step that takes it out of +#: nickname, so a row expressible this way necessarily differs from +#: the default on both. +_BUCKET_MOVE_ONLY = frozenset({ + "nickname_delimiters", + "maiden_delimiters", +}) + +#: Fields _constants_for writes into the Constants it returns. +_TRANSLATED = frozenset({ + "patronymic_rules", + "middle_as_family", + "extra_suffix_delimiters", +}) + +#: The non-locale rows the facade never sees. Held here rather than +#: described in prose, because pytest reports a skipped row exactly +#: like a row nobody wrote: reverting the _MAIDEN_PARENS shape above +#: to the unsubtracted default pushes every parenthesis-maiden row +#: into this set and turns nothing red. +_CORE_ONLY_IDS = frozenset({ + "maiden_marker_delimited_beside_a_nickname_clause", + "maiden_marker_kyusei_delimited", + "ko_honorific_period_under_strict_comma_suffixes", + "ja_honorific_glued_family_comma_strict_knob", + "ja_honorific_glued_family_comma_credential_pair_strict_knob", +}) + def _constants_for(case: Case) -> Constants | None: """Translate the row's Policy to a Constants, or None if the policy @@ -24,24 +84,19 @@ def _constants_for(case: Case) -> Constants | None: policy = case.policy or Policy() default = Policy() c = Constants() - maiden_via_sentinel = ( + maiden_via_bucket_move = ( policy.maiden_delimiters == _MAIDEN_PARENS - and policy.nickname_delimiters == default.nickname_delimiters + and policy.nickname_delimiters + == default.nickname_delimiters - _MAIDEN_PARENS ) - unexpressible = ( - policy.name_order != default.name_order - # script_orders has no v1 Constants spelling at all (the v1 - # surface is frozen), so a row opting out must SKIP here rather - # than fail against the facade's inherited default. - or policy.script_orders != default.script_orders - or policy.lenient_comma_suffixes != default.lenient_comma_suffixes - or policy.strip_emoji != default.strip_emoji - or policy.strip_bidi != default.strip_bidi - or policy.nickname_delimiters != default.nickname_delimiters - or (policy.maiden_delimiters != default.maiden_delimiters - and not maiden_via_sentinel) - ) - if unexpressible: + # field by field, off the dataclass rather than by hand: a field + # named in no table above would be admitted here and then run + # under the facade's default for it (see + # test_every_policy_field_is_translated_or_skipped). + moved = {f.name for f in dataclasses.fields(Policy) + if getattr(policy, f.name) != getattr(default, f.name)} + if moved & _UNTRANSLATED or (moved & _BUCKET_MOVE_ONLY + and not maiden_via_bucket_move): return None if policy.patronymic_rules: c.patronymic_name_order = True @@ -49,15 +104,34 @@ def _constants_for(case: Case) -> Constants | None: c.middle_name_as_last = True if policy.extra_suffix_delimiters: c.suffix_delimiter = next(iter(policy.extra_suffix_delimiters)) - if maiden_via_sentinel: - # bucket-move idiom (see _DelimiterManager docstring): adds - # parenthesis to maiden while nickname keeps its own default - # (which already includes parenthesis) -- the nickname reading - # still wins on a shared delimiter, matching the row's Policy. - c.maiden_delimiters["parenthesis"] = "parenthesis" + if maiden_via_bucket_move: + # pop, not assign: leaving parenthesis in the nickname bucket + # would give v1's shared-pair reading, which goes to nickname. + c.maiden_delimiters["parenthesis"] = c.nickname_delimiters.pop( + "parenthesis") return c +def test_every_policy_field_is_translated_or_skipped() -> None: + # a Policy field in none of the three tables is neither rejected + # nor written into the Constants, so a row setting it is ADMITTED + # and then asserted under the facade's default value for that + # field. segment_scripts sat in exactly that state until + # 2026-08-03, invisibly, because no row set it. Same job + # test_policy_patch_mirrors_policy_field_names does one file over. + assert (_UNTRANSLATED | _BUCKET_MOVE_ONLY | _TRANSLATED + == {f.name for f in dataclasses.fields(Policy)}) + + +def test_core_only_rows_are_the_declared_ones() -> None: + # the gate that decides this is one comparison away from claiming + # no policy is expressible (see _MAIDEN_PARENS), and every row it + # wrongly rejects leaves the suite green -- a skip reads as a pass. + assert {case.id for case in CASES + if case.locale is None + and _constants_for(case) is None} == _CORE_ONLY_IDS + + @pytest.mark.parametrize("case", CASES, ids=lambda c: c.id) def test_facade_case(case: Case) -> None: if case.locale is not None: diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index 08c64fa..67b2fac 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -431,6 +431,22 @@ def test_revise_sub_parse_structural_behavior() -> None: assert p.revise(n, family="Smith (Jones").ambiguities == () +def test_revise_sub_parses_under_this_parsers_policy() -> None: + # the sub-parse runs on SELF, not on a default Parser: revise's + # docstring promises a marker LEADING a delimited value is + # consumed, and only a policy routing that pair to maiden makes + # the value delimited at all. Sub-parsing with Parser() instead + # leaves the rest of the suite green -- every other revise + # assertion uses a default parser, so nothing else can tell the + # two apart. + p = Parser(policy=Policy(maiden_delimiters=frozenset({("(", ")")}))) + n = p.parse("John Smith") + assert p.revise(n, family="(née Jones)").family == "Jones" + # and the third leg: a leading marker in an UNDELIMITED value is + # no marker at all, so the same words keep it (#329) + assert p.revise(n, family="née Jones").family == "née Jones" + + def test_revise_forces_the_named_role_on_every_harvested_token() -> None: # the sub-parse reads "Dr." as a title and "Jr." as a suffix; the # named field's role must win for every token or the family view diff --git a/tools/differential/corpus_cjk.jsonl b/tools/differential/corpus_cjk.jsonl index d26347d..34f198e 100644 --- a/tools/differential/corpus_cjk.jsonl +++ b/tools/differential/corpus_cjk.jsonl @@ -23,6 +23,7 @@ "山田 エミ" "山田 太郎 (マイケル・ジャクソン)" "山田 花子 旧姓 佐藤" +"山田 花子（旧姓 佐藤）" "山田「タロ」太郎" "山田太郎様" "山田花子 旧姓 佐藤" diff --git a/tools/differential/expected_changes.toml b/tools/differential/expected_changes.toml index 8cfa0ab..9b118c2 100644 --- a/tools/differential/expected_changes.toml +++ b/tools/differential/expected_changes.toml @@ -215,6 +215,40 @@ issue = "fix(cjk-delimited-nickname) delimiter recognition compounds with the CJ name_regex = "(?s)(?=.*[「」『』・･])(?=.*[\\u3005-\\u3006\\u3040-\\u309F\\u30A0-\\u30FF\\u3400-\\u4DBF\\u4E00-\\u9FFF\\uF900-\\uFAFF\\uAC00-\\uD7A3\\uFF65-\\uFF65])" fields = ["first", "last", "nickname"] +[[change]] +issue = "fix(cjk-fullwidth-paren-nickname) fullwidth-parenthesis recognition compounds with the CJK order flip" +# '山田 花子（旧姓 佐藤）', which reached the corpus from the #329 case +# table via build_cjk_corpus.py. It is NOT a #329 diff: that change is +# gated on a non-empty Policy.maiden_delimiters and the harness runs +# the default policy, where （） is a #273 NICKNAME delimiter. So the +# same two intended changes as the rule above, with a different +# delimiter pair -- 1.4 knew no fullwidth pair and read the brackets as +# name text (first 山田 / middle '花子（旧姓' / last '佐藤）'), while 2.0 +# extracts the clause to `nickname` and the wholly-Han remainder left +# behind takes the family-first flip. +# +# Its own rule rather than a widening of fix(cjk-delimited-nickname), +# on both halves. That rule's `fields` deliberately exclude `middle`, +# and this diff has one: the fullwidth clause is written flush against +# the token before it, so 1.4's tokenizer left '花子（旧姓' as a single +# middle-name word -- a spaced corner-bracket name never does that. +# Adding `middle` there would pre-excuse a bare middle regression on +# '山田「タロ」太郎'. Its delimiter set is separately pinned in +# tests/v2/test_regex_sync.py as "its own decision surface", so +# widening it is a deliberate act with a test to update, and there is +# nothing to gain by making it here. +# +# Both lookaheads are required for the reason given on the rule above: +# （） lives in Halfwidth and Fullwidth Forms, outside every classified +# span, so a Latin name can carry it ('John （Jack） Kennedy') and the +# delimiter alone would let this rule absorb a first/last regression on +# one. The span class is the same hand copy of _SCRIPT_RANGES the rules +# above carry, pinned by the same auto-discovering test. The slug +# avoids the literal #271/#272 substrings the canonical-rule pin +# selects by. +name_regex = "(?s)(?=.*[（）])(?=.*[\\u3005-\\u3006\\u3040-\\u309F\\u30A0-\\u30FF\\u3400-\\u4DBF\\u4E00-\\u9FFF\\uF900-\\uFAFF\\uAC00-\\uD7A3\\uFF65-\\uFF65])" +fields = ["first", "middle", "last", "nickname"] + [[change]] issue = "fix(cjk-comma-compound) comma routing compounds with the CJK order flip" # '威廉·莎士比亚, PhD': one name, two intended changes at once -- the