Consume a maiden marker inside delimited content (#329) - #331
Merged
Conversation
The bare form drops the marker and the delimited form kept it, so one person's name gave two different `maiden` values depending on spelling. extract records the delimited clause as a span and makes no tokens at all; tokenize gives the tokens cut inside that span Role.MAIDEN, and classify then tags the marker like any other token -- an earlier account of this said the marker was never tagged, which was wrong. What the marker never is, is in `pieces`: segment builds the main stream from role-less tokens, so #274's consuming rule, which walks pieces, cannot reach a token that already carries a role. The pass is scoped to the CLAUSE, keyed on state.extracted spans, rather than to a maiden token's neighbours. Role.MAIDEN is not proof of extraction -- #274 sets it too -- so a neighbour test also fires on the bare form and eats the `Nee` out of "Jane Smith nee Nee Jones". Clause scoping puts that path out of reach by construction instead of by assumption about where roles come from. Drop a clause's first token only when the clause holds more than one. `Nee` is a real surname, so "(Nee)" is a maiden name -- and it stays one in "(Nee) (Jones)", where the two clauses land as a single contiguous run of maiden tokens that only the clause bound separates. Locate the clause's tokens by bisect, not by scanning the token list per clause: that is quadratic in the number of delimited pairs. "(a) " * 3200 -- four times the 800 the shape starts at -- measured 14.1x against the 4.1x the same shape holds with no maiden_delimiters set; bisected, it is 4.3x. Both figures are costs for 4x the input rather than per doubling, which the control number is what makes legible: a linear shape runs ~4x for 4x the input and ~2x for a doubling, so a 4.1x control can only be the former. test_benchmark gains _POLICY_SHAPES for this. Its ten existing shapes all run bare parse(), so a stage gated on an opt-in Policy field is dead code there -- this quadratic lived in a shape the file ALREADY had ("(a) ") and still went unseen. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three rows for the behavior 5ec39a5 changed, each measured against a
pinned 1.4.0 before it was written. The bracketed
'Jane Smith (née Jones)' now agrees with the bare maiden_marker row
above it; '山田 花子（旧姓 佐藤）' is the Japanese spaced-inside-brackets
form the same drop reaches.
'Jane Smith (Nee) (Jones)' is the scoping pin: the drop is
clause-scoped, so a one-token clause keeps its token. A
neighbour-scoped rule -- drop a marker whose successor is also maiden
-- gives 'Jones' here and eats a real surname.
Two of the three are expressible in v1 through the bucket-move idiom
maiden_delimiters['parenthesis'] = nickname_delimiters.pop('parenthesis').
"Jane Smith (née Jones)" gave maiden 'née Jones' there -- the marker
inside the value, the one field this change moves -- so fix(#329).
"Jane Smith (Nee) (Jones)" gave 'Nee Jones', identical to 2.0, so the
scoping pin is parity, recorded as measured rather than inferred from
the row being untouched. The facade runner declines all three at this
point in the branch; a later commit corrects the gate that makes it.
The Japanese row has no v1 spelling at all. v1's delimiter buckets
hold the NAMES of compiled regexes and no fullwidth pair is among them
(#273 added it), so assigning ('（', '）') raises ValueError at parse
time. Classified against 1.4.0's single reading -- first 山田 / middle
'花子（旧姓' / last '佐藤）' -- the convention
ko_honorific_period_under_strict_comma_suffixes set for a knob v1
cannot express.
The differential row and the rule that explains it land together here,
because separating them leaves the harness red in between. The CJK
corpus name added with the rows runs under the DEFAULT policy, where
（） is a #273 nickname delimiter and nothing in #329 is reachable, so
its diff is fullwidth-delimiter recognition compounding with the
family-first flip -- the shape fix(cjk-delimited-nickname) already
covers for corner brackets. It gets its own rule rather than a
widening of that one: the fullwidth clause is written flush against
the token before it, so 1.4 left '花子（旧姓' a single middle-name
word, and adding `middle` to that rule's fields would pre-excuse a
bare middle regression on 山田「タロ」太郎. The rule is labelled for its
actual cause, not #329. Run after: 107 intentional diffs, 0
unexplained, every pre-existing category's count unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The claim that delimited content is "claimed whole, before classify has tagged anything inside it" was never true: classify tags the marker wherever it stands as a token. What was missing was the CONSUMING -- extract claims the clause, tokenize gives its tokens Role.MAIDEN up front, segment keeps role-bearing tokens out of the main stream, and #274's rule walks pieces. Two of the three sites AGENTS.md names still carried the wrong mechanism, and the fix commit wrote it twice more in its own comment and test docstring. Corrected in config/maiden_markers.py, the release-log 旧姓 entry, _group.py's inline comment and test_group.py's docstring. The surviving Japanese limit is restated with its real cause: 山田（旧姓：佐藤） keeps maiden 旧姓：佐藤 because the fullwidth colon glues marker and name into one token, leaving nothing to drop -- the same reason the wholly unspaced 山田花子（旧姓佐藤） does. The release-log 旧姓 entry also loses its "SPACED form only" framing, which this branch made ambiguous now that （旧姓 佐藤） works, and its false "Jane Smith (née Jones) keeps its marker the same way". Edited rather than contradicted by a later entry, 2.1.0 being unreleased. Adds the #329 entry to 2.1.0's Behavior Changes, scoped honestly: Policy.maiden_delimiters is empty by default, HumanName reaches it only through the v1 bucket move, and the colon form still disagrees with the bare form pending #317's head-peel. Also: _extract.py's stage header now says the marker rides along in the extracted span and group drops it; customize.rst's maiden_delimiters row and Policy.maiden_delimiters' field doc say the head marker leaves the value; AGENTS.md records that the wrong mechanism recurred inside the commit that fixed it. Prose only -- no behavior change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Seven findings from review, each re-measured before rewriting. The rule that replaced the wrong mechanism was itself over-narrow: it said a Japanese marker must be "written with a space on each side", then offered 山田 花子（旧姓 佐藤） as an example, where 旧姓 has a bracket on its left. extract masks the whole bracketed region, delimiters included, before tokenize runs, so a bracket bounds a token exactly as a space does -- 山田（旧姓 佐藤） needs no space at all and still gives maiden 佐藤. Restated in maiden_markers.py and the release-log 旧姓 entry. The colon case survives for the reason both already gave. A fresh stage error had crept into the correction itself: cases.py said "extract assigned the whole clause Role.MAIDEN", but extract_delimited produces no tokens -- it records extracted/masked spans, and tokenize sets the role. Fixed, and AGENTS.md records that the bullet about checkable stage claims caught its own corrector. cases.py also credited the facade skip to the shim's pre-subtraction. Measured: it is Policy.__post_init__'s maiden-wins canonicalization, which subtracts the pair at construction. test_facade_cases.py's note was stale in three ways (four rows, not one; a case id that exists nowhere; a shape called v1-expressible when maiden_via_sentinel is never True) and its sentinel branch is dead code that would also FAIL its row if revived, v1 giving a shared pair to nickname. The two user-facing docs stated the drop unconditionally, omitting the clause-size guard: Policy.maiden_delimiters' field doc and customize.rst now say a lone "(Nee)" keeps its token. Parser.revise's docstring said "mid-value" marker; a leading one in a delimited value is consumed too. Last, the drop can cascade to a wholly empty parse: assemble's content test runs over SURVIVING tokens, so "(née —)" -- where the marker is the clause's only alphanumeric token -- now clears every field and is falsy, where 1.4.0 and pre-#329 gave maiden 'née —'. Taken as a decision, not patched: a dropped marker is structural, and "(-)" already emptied on both sides. assemble's comment said "anywhere", true only while nothing could remove a content-bearing token; it now says "survives" and names the input where the readings part. Pinned by cases.py's maiden_marker_delimited_content_free, classified fix(#329) against a measured 1.4.0. Release-log entry gains the truthiness note and the field census behind its no-other-field claim. Prose plus one case row -- no behavior change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The #329 battery pinned the POSITION of the drop and left its predicates open. Three rows, each measured against 1.4.0 before it was written: * "(Mary Jones)" -- a multi-token clause whose first token is not a marker. Nothing in the suite had that shape, so deleting the marker-tag test left every test passing while the pass ate the opening word of every delimited maiden name. * "(Jones née)" -- a marker that is not first. Dropping every marker in the clause instead of the first also passed the whole suite. * 'Jane "née Janie" Smith {née Jones}' -- a maiden clause and a nickname clause, both opening with a marker word. Deleting the Role.MAIDEN filter passed too, eating the nickname's opening word. Braces route to maiden so the default nickname set survives; the parenthesis rows cannot show this, since maiden-wins canonicalization takes the pair away from nickname. test_facade_cases' enumeration of the rows its runner skips grows with them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
maiden_marker_delimited_two_clauses claimed to exercise the marker branch through the default vocabulary's unaccented 'nee'. It does tag the token, but the clause-size test is checked first, so the row's value never depended on the entry: deleting 'nee' from nameparser/config/maiden_markers.py left the whole suite green (measured 2026-08-03) -- the spelling English writes most often was one vocabulary edit from silence. Adds "Jane Smith (nee Jones)", which is now the only test whose value depends on that entry, and rewrites the two-clause note to say what the tagging actually buys there (it is what makes the row kill a rule that drops the clause bound). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
docs/release_log.rst promises #329 through HumanName and nothing tested it: all of the case table's maiden_delimiters rows skip the facade runner, because Policy's maiden-wins canonicalization takes the pair away from nickname and the row stops matching the v1-expressible shape. The bucket-move idiom the file already uses for its other maiden tests does express it. Three tests, one per branch of the rule -- a marker consumed, unmarked content kept whole, a lone marker word kept as the surname it also is. Deleting the #329 pass fails the first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two ways the maiden_pairs guard could evaporate without a red test. Deleting the last _POLICY_SHAPES entry: pytest turns an empty parametrize into a SKIP, so the quadratic guard would leave via the skip count. test_shape_tables_are_not_empty fails instead (measured: emptying the table gives "1 failed, 12 passed, 1 skipped"). Precedence moving under it: the shape only reaches #329's clause loop while ( ) routes to maiden, and nothing checked that it still does -- route the pair back to nickname and the test would measure a bare parse at a comfortable 4.2x forever. That is the module docstring's own failure mode one level up. Each policy shape now carries a reachability probe, run before the measurement. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The pass locates a clause's first token by span, and its comment rests that on script_segment only ever cutting a token into sub-slices -- but test_group's helper skips that stage and no case row combines segmentation with a maiden clause (Case makes locale and policy mutually exclusive, and the zh reading needs the pack's surname lexicon as well as its policy patch, so the table cannot express it). Measured at stage level: '王小明（旧姓 李四）' segments to 王 / 小明 / 旧姓 / 李四, shifting the marker from index 1 to index 2, and the drop still lands on it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_group's bisect note read "14.1x cost per doubling". The measurement is "(a) " * 3200 against a base of 800 -- a quadrupling -- and its own control says so: linear is ~4 for 4x the input and ~2 for a doubling, so a 4.1x linear control cannot be per doubling. The same phrase 240 lines up, on merge()'s re-flatten, reads right (2.4-2.9x per doubling against a 2.0x control; re-measured, the clean control is 2.04x). Re-measured both figures at 11.2x against 4.2x for 4x the input, and recorded that the ratio moves with the runner while the separation does not. test_marker_in_the_bare_form_is_left_to_the_piece_rule stays green when the #329 pass is deleted wholesale, so it reads as dead weight. It is the only bare-path guard against the neighbour-scoped spelling this fix originally took -- checked both ways. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The brace delimiter is ACCEPTED by 1.4.0's Constants and raises at parse time, not at assignment (measured 2026-08-03, matching what the kyusei row already said), and the trailing-marker rationale claimed more than anyone checked -- it is the shipped vocabulary that never writes a marker after the name, not every language. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The perf bullet described only the ten default shapes. A stage behind an opt-in Policy field needs a _POLICY_SHAPES row, and that table's two new guards -- the per-row reachability probe and the non-empty assertion -- are exactly the kind of thing a later reader has to be told about before touching the file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The header note and the inline comment disagreed. The note says v1 gives a shared delimiter pair to nickname, so the branch would fail its row if reached; the inline comment said the same spelling matched the row's Policy. Measured on "Baker (Johnson), Jenny": the branch's sentinel-add spelling gives nickname Johnson and empty maiden, and the row asserts the reverse, so the note was right. Also record the spelling that does match -- the bucket-move already used in tests/test_nicknames.py -- since that is what reviving the branch would need, and it is not obvious from the failing one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The facade runner's gate for these rows could never be true. It compared the row's nickname_delimiters against the unsubtracted default, but Policy's maiden-wins canonicalization removes the pair from nickname at construction, so a row routing parens to maiden differs from the default on BOTH delimiter fields -- and the nickname clause of `unexpressible` rejected it before the maiden clause got a say. The branch behind the gate would also have failed if reached. It ADDED parenthesis to the maiden bucket while nickname kept its own also-parenthesis-holding default, and v1 gives a shared pair to nickname: measured on "Baker (Johnson), Jenny", that yields nickname Johnson and empty maiden, the reverse of the row. Moving the pair instead -- pop from nickname, assign to maiden, the idiom tests/test_nicknames.py already uses -- yields maiden Johnson. Seven rows now run twice instead of once, all passing, and reverting the pop to the add fails all seven rather than skipping them. Two still skip and say why: one routes braces, which v1 rejects outright, and one adds fullwidth parens alongside ASCII. Swept the notes that recorded the old skip as a fact. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every delimited maiden row in the battery had a two-token clause, where "drop the clause's first token" and "drop all but the last" give the same answer. Two opposite mistakes therefore survived the whole suite: restricting the drop to a clause of exactly two tokens, and letting it eat the token after the marker. One three-token row kills both. The content-free row's note asserted in prose that the same clause inside a name keeps given/family and a maiden em dash. Nothing checked it, and widening the drop to take the clause's punctuation along with the marker left the suite green -- the bare form empties the parse either way, so only the with-a-name form separates the two readings. That form is now a row, and the note points at it instead of restating it. Both values measured against the tree and classified against real 1.4.0 through the bucket-move idiom (2026-08-03); maiden is the only field that differs in each. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
revise's docstring gained a three-way promise about maiden markers, of which only the mid-value leg was tested, and under a default Parser(). That left a mutant alive that is worse than a docs gap: making the sub-parse run on Parser() rather than self keeps the whole suite green, and revise then silently ignores the policy its own parser was built with -- "(née Jones)" comes back with the marker still on it. The new test covers the other two legs, both measured against the tree first, and dies alone under that mutation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The gate deciding a row's policy is expressible through v1 Constants had two silent failure modes, both of which read as a passing suite because pytest reports a skipped row exactly like a row nobody wrote. Reverting the _MAIDEN_PARENS shape to the unsubtracted nickname default -- the bug 7e605f3 just fixed -- pushes every parenthesis maiden row back into the skip count with nothing going red. The set of core-only rows is now asserted by id, so the revert fails. The header comment stops enumerating those ids, which it had already got wrong once (it named a row that exists nowhere in the tree); the assertion owns the list now. The other mode is a Policy field named in neither the rejection nor the translation: it is ADMITTED and then asserted under the facade's inherited default for that field. segment_scripts was in that state, invisible only because no row sets it. The gate now reads the fields off dataclasses.fields(Policy) against three declared tables, and a completeness test ties those tables to the dataclass, in the shape test_policy_patch_mirrors_policy_field_names already uses. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- release_log: keep the "maiden is the only field that differs" claim, drop "over some eighty thousand inputs". Nothing in the repo produces that number, or any number: the three corpora total 783 names and the differential harness runs the DEFAULT policy, where a non-empty maiden_delimiters is unreachable and none of this change is exercised. Substituting a bigger unreproducible count would be the same defect. - maiden_marker_delimited_unmarked_content said only its shape separates the clause-size test from the marker-tag test. A sibling row added in the same commit falsifies that: dropping the tag conjunct fails five tests across two shapes. Says what IS this row's alone instead -- no token in its clause is a marker at all. - maiden_marker_delimited_unaccented called itself the only TEST depending on 'nee'. True when written; 7e605f3 gave the row a second runner, so removing the entry now fails two tests, one row. - customize.rst said the drop needs "more than one word"; the code tests TOKENS (_policy.py and _group.py both say token). They diverge exactly where a reader would care -- "山田花子（旧姓佐藤）" is one token and two words, and keeps its marker. And test_benchmark's table guard no longer counts _POLICY_SHAPES' entries, which goes stale on the second one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The staleness sweep over this change's own words. _assemble's content test argued in prose that a maiden clause inside a NAME keeps its punctuation as the maiden value, and _group's clause rule argued that the drop takes the first token and no more; both now have a row, so both say which. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #331 +/- ##
=======================================
Coverage 98.47% 98.48%
=======================================
Files 41 41
Lines 2823 2832 +9
=======================================
+ Hits 2780 2789 +9
Misses 43 43 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #329.
parse("Jane Smith née Jones")gave maidenJones, butparse("Jane Smith (née Jones)")under a policy routing()to maiden gavemaiden
née Jones— the same relationship, two values, so a caller comparingmaidenacross a dataset saw a spurious mismatch between two spellings of oneperson's name. The Japanese form moves with it:
"山田 花子（旧姓 佐藤）"now givesmaiden
佐藤.The cause, stated correctly
classifydoes tag the marker — inside brackets as anywhere else. What wasmissing is the consuming:
extractrecords the clause as a span,tokenizegives the tokens cut inside it
Role.MAIDEN, andsegmentbuilds the mainstream from role-less tokens — so #274's rule, which walks
pieces, cannotreach a token that already carries a role.
(An earlier account of this said the marker was never tagged. That was wrong,
and correcting it in one place while it survived in five others is a recurring
theme of this branch — see below.)
The new pass is scoped to the clause, keyed on
state.extractedspansrather than on a maiden token's neighbours. Both halves are load-bearing:
Role.MAIDENis not proof of extraction (#274 sets it too), so a neighbour testfires on the bare path and eats the surname out of
"Jane Smith nee Nee Jones";and separate clauses are separate content, so
"Jane Smith (Nee) (Jones)"keepsmaiden
Nee Jones—Neeis a real surname, and a one-token clause is a namerather than a marker.
Scope
Policy.maiden_delimitersis empty by default, so this reaches only callerswho opted a pair in.
HumanNamereaches it through the v1 bucket move, sincev1 gives a pair held in both buckets to
nickname.One edge changes truthiness rather than a field:
"(née —)"returned maiden"née —"and is now falsy throughout, because dropping the marker leaves nothingcarrying an alphanumeric character.
"Jane Smith (née —)"is unaffected.Not fixed:
"山田（旧姓：佐藤）"still returns旧姓：佐藤. The fullwidth colonleaves marker and name a single token, so there is nothing to drop — that wants
the head-peel #317 tracks.
Verification
-Wcleanevery commit, not only at the tip
maidenis the only field that ever differs, measured across a generatedcensus rather than reasoned
bound, bisect index and per-clause scoping are each killed by named tests
Review history
Four independent reviewers over two rounds. The second round found four
mutation survivors the first round's fixes had left — including that no maiden
clause in the suite had more than two tokens, and that
revisesilently ignoredits own parser's policy. Both are now pinned.
Also corrected here: three claims #330 shipped about which stage does what, and
a facade-runner gate that had been structurally unsatisfiable since the
maiden-wins canonicalization landed, silently skipping seven case rows.
🤖 Generated with Claude Code