Swallowed-sense census, cycle 2: sequence-complete (B) and full-run in-text (C) shapes

[UniquePixels]

From: maintainer pattern-spotting during the doc-08 review, 2026-08-07 (option 2 chosen: run after the current sense-structure-repair cycle lands, reusing its S1 splitter + census/review/disposition machinery).

The doc-08 census (shape A: unnumbered sense, —2) in text, no 1) before it) surfaces only part of the Sefaria segmentation-failure space. Two further shapes measured 2026-08-07 against the committed corpus:

Shape B — 146 entries: structural sequence is a clean 1..n, but the last numbered sense's tag-stripped text carries —(n+1)) + whitespace — the continuation was swallowed and no existing detector fires (sequence doesn't break, doesn't start at 2, and the census skips numbered senses). Conservative measurement: last-sense/single-next-marker only; chains will deepen some rows. Known members overlap independent finds: D00919, D00077.

Shape C — 3 entries: the whole run lives in-text (1)…—2)…) with no structural numbering: A02061, B00516, U00483 (citation-noise filtered: 1) not preceded by alphanumeric/paren).

Done when: both shapes get the S3 treatment — committed literal censuses, generated eyes-on review doc(s) with doc↔census equality tests, maintainer decisions per row, splits/inserts encoded as rid-keyed repairs, and the S5 completeness gate extended to the new dispositions. Detector definitions above are the spec; calibration scripts live in the session that produced them and are trivially rebuilt from the definitions.

🤖 Generated with Claude Code

[coderabbitai]

Coding Plan

Summary

• Rebuild the Shape B and Shape C detectors directly from the ticket definitions. Use a shared tag-stripping and marker helper. Validate the detectors against the named example rids.
• Emit committed, byte-stable, rid-keyed literal censuses. Then generate human-readable review docs and lock them to the censuses with doc↔census equality tests.
• Capture per-row maintainer decisions. Encode the approved splits and inserts as additive rid-keyed repairs.
• Extend the S5 completeness gate so CI fails unless every Shape B and Shape C entry is repaired or explicitly dispositioned. Reuse the cycle-1 machinery where present. Use the existing scripts/validate-data.ts patterns as the fallback.

Design Choices

Design Choice 1: The Shape A (doc-08) splitter and census/review/disposition machinery is not present in this checkout

Options Considered:

1. Assume the cycle-1 machinery is present at execution time. Reuse its S1 splitter, census file format, review-doc generator, disposition store, and S5 gate. Mirror its file locations and formats exactly for Shape B and Shape C.
2. Build fresh detector, census, review, disposition, and gate components from the existing repo patterns (scripts/lib/, scripts/validate-data.ts, data/admin/annotations.json), without depending on cycle-1 artifacts.

Chosen Option: 1

Rationale: The ticket explicitly chose "option 2" (run after the current cycle lands, reuse its S1 splitter + census/review/disposition machinery), so the plan reuses cycle-1 components and mirrors their conventions. Where a component is absent at execution time, the follow-up agent must fall back to the equivalent repo pattern named in each task.

Design Choice 2: Detector definitions in the ticket are the authoritative spec

Options Considered:

1. Rebuild the Shape B and Shape C detectors from the ticket definitions and validate against the named example rids.
2. Locate and import the original calibration scripts.

Chosen Option: 1

Rationale: The original scripts are not in the repo, and the ticket says to rebuild from the definitions. The named example rids give a concrete validation set.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Detectors and committed literal censuses

This phase rebuilds the Shape B and Shape C detectors from the ticket definitions. It runs them over the truth-layer corpus and produces committed, literal census files. The census files are the machine-readable source of truth for all later phases.

Task 1: Add a tag-stripping and marker helper

Provide a shared helper to convert sense HTML to plain text and to find numbered markers, because both detectors operate on tag-stripped text.

• Add a helper module under scripts/lib/ (for example scripts/lib/sense-markers.ts).
• Add a function that strips HTML tags from a sense d string and returns plain text. Reuse the htmlparser2 dependency already used by scripts/lib/entry-html.ts rather than adding a new parser.
• Add a function that finds in-text numbered markers of the form N) and —N) in plain text.
• Add the citation-noise filter: a 1) marker counts only when it is not preceded by an alphanumeric character or a paren. Use this filter for Shape C.
• Keep this module dependency-light and side-effect free so tests can import it directly.

Task 2: Rebuild the Shape B detector

Detect entries where the structural sense numbers form a clean 1..n, but the last numbered sense text carries a swallowed —(n+1)) continuation.

• Add the Shape B detector under scripts/ (for example scripts/census-shape-b.ts), reusing the cycle-1 S1 splitter output if it is present.
• Read data/jastrow-part1.jsonl and data/jastrow-part2.jsonl line by line, matching the read pattern in scripts/validate-data.ts.
• For each entry, read the structural n labels in c.s[] and confirm they form a clean 1..n sequence.
• Strip tags from the last numbered sense d (Task 1.1). Detect a trailing —(n+1)) marker followed by whitespace.
• Apply the conservative rule: check the last sense only, and match a single next marker only. Do not follow deeper chains.
• Validate that D00919 and D00077 are detected.

Task 3: Rebuild the Shape C detector

Detect entries whose whole numbered run lives in-text, with no structural numbering.

• Add the Shape C detector under scripts/ (for example scripts/census-shape-c.ts).
• Read the truth-layer files with the same pattern as Task 1.2.
• Select senses that carry no structural n label but whose tag-stripped d text contains an in-text run 1)…—2)….
• Apply the citation-noise filter from Task 1.1 to the 1) marker.
• Validate that A02061, B00516, and U00483 are detected, and that the count is 3.

Task 4: Emit committed literal census files

Write the detector output to committed census files, one per shape.

• Write census files to the directory used by the cycle-1 Shape A census. If that directory is absent at execution time, use data/v2/ and note the location in the review doc.
• Use a literal, stable, rid-keyed format that mirrors the cycle-1 Shape A census (for example a JSON object keyed by id, each value holding the shape, the detected marker text, and the sense index). Use the data/admin/annotations.json rid-keyed structure as the fallback model.
• Sort keys deterministically so the committed files are byte-stable across reruns.
• Record the conservative-measurement note for Shape B (last-sense/single-next-marker only) in the census file metadata or an adjacent note.

🤖 Prompt for AI agents

Implement Phase 1: rebuild the Shape B and Shape C detectors from the ticket
definitions, run them over the truth-layer corpus, and produce committed literal
census files as the machine-readable source of truth for later phases.

First, add a shared helper module under `scripts/lib/` (for example
`scripts/lib/sense-markers.ts`). In this module, add a function that strips HTML
tags from a sense `d` string and returns plain text, reusing the `htmlparser2`
dependency already used by `scripts/lib/entry-html.ts`. Add a function that
finds in-text numbered markers of the form `N)` and `—N)` in plain text. Add a
citation-noise filter so that a `1)` marker counts only when it is not preceded
by an alphanumeric character or a paren; use this filter for Shape C. Keep the
module dependency-light and side-effect free so tests can import it directly.

Next, add the Shape B detector under `scripts/` (for example
`scripts/census-shape-b.ts`), reusing the cycle-1 S1 splitter output if it is
present. Read `data/jastrow-part1.jsonl` and `data/jastrow-part2.jsonl` line by
line, matching the read pattern in `scripts/validate-data.ts`. For each entry,
read the structural `n` labels in `c.s[]` and confirm they form a clean `1..n`
sequence. Strip tags from the last numbered sense `d` text and detect a trailing
`—(n+1))` marker followed by whitespace. Apply the conservative rule: check the
last sense only, and match a single next marker only; do not follow deeper
chains. Validate that D00919 and D00077 are detected.

Add the Shape C detector under `scripts/` (for example
`scripts/census-shape-c.ts`), reading the truth-layer files with the same
pattern. Select senses that carry no structural `n` label but whose tag-stripped
`d` text contains an in-text run `1)…—2)…`. Apply the citation-noise filter to
the `1)` marker. Validate that A02061, B00516, and U00483 are detected, and that
the total count is 3.

Finally, emit committed literal census files, one per shape. Write these files
to the directory used by the cycle-1 Shape A census; if that directory is
absent, use `data/v2/` and note the location in the review doc. Use a literal,
stable, rid-keyed JSON format keyed by `id`, with each value holding the shape,
the detected marker text, and the sense index, mirroring the cycle-1 Shape A
census or the `data/admin/annotations.json` structure as a fallback model. Sort
keys deterministically so the committed files are byte-stable across reruns.
Record the conservative-measurement note for Shape B
(last-sense/single-next-marker only) in the census file metadata or an adjacent
note.

Phase 2: Review docs, equality tests, and maintainer decisions

This phase turns the censuses into human-readable review docs. It adds tests that assert the review docs and the censuses stay in exact agreement. It captures the maintainer decision for each row.

Task 1: Generate eyes-on review docs

Produce human-readable review docs from the census files.

• Add a generator that reads the census files and writes review docs. Mirror the cycle-1 Shape A review-doc generator and its output location (fallback docs/v2/).
• Produce one row per census entry. Include id, hw, the current sense text, the detected marker, and the proposed split or insert.
• Generate the review doc content deterministically from the census so the equality test in Task 2.2 is meaningful.
• Cover both shapes, either as one doc with two sections or as one doc per shape, matching the cycle-1 convention.

Task 2: Add doc↔census equality tests

Add tests that fail when the review docs and the census files drift apart.

• Add a bun:test file under scripts/ (for example scripts/census-review.test.ts), following the describe/it/expect style of scripts/lib/entry-schema.test.ts.
• Assert that the set of rids in each review doc equals the set of rids in the matching census file, in both directions. Model the set-comparison on the seenIds map logic in scripts/validate-data.ts.
• Assert that each per-row field in the review doc (marker, sense index) matches the census value for that rid.
• Assert the fixed counts: Shape B = 146, Shape C = 3.

Task 3: Capture per-row maintainer decisions

Record the maintainer decision for each census row.

• Add a rid-keyed decisions store that mirrors the cycle-1 disposition format (fallback: a committed JSON file next to the censuses, keyed by id, each value holding the decision and any split/insert instruction).
• Support the decision values that cycle-1 uses (for example accept-split, insert-sense, no-change, defer). Reuse the exact vocabulary from cycle-1 if present.
• The maintainer fills the decisions; the plan provides the file and the schema, not the decisions themselves.

🤖 Prompt for AI agents

Implement Phase 2: turn the Phase 1 censuses into human-readable review docs,
add tests that lock the docs to the censuses, and capture the per-row maintainer
decision.

First, add a generator that reads the Shape B and Shape C census files and
writes review docs. Mirror the cycle-1 Shape A review-doc generator and its
output location, with a fallback location of `docs/v2/`. Produce one row per
census entry, including `id`, `hw`, the current sense text, the detected marker,
and the proposed split or insert. Generate the review doc content
deterministically from the census data so that the equality test is meaningful.
Cover both Shape B and Shape C, either as one doc with two sections or as one
doc per shape, matching the cycle-1 convention.

Next, add a `bun:test` file under `scripts/` (for example
`scripts/census-review.test.ts`), following the `describe`/`it`/`expect` style
of `scripts/lib/entry-schema.test.ts`. In this test file, assert that the set of
rids in each review doc equals the set of rids in the matching census file,
checked in both directions, modeling the set comparison on the `seenIds` map
logic in `scripts/validate-data.ts`. Assert that each per-row field in the
review doc (marker, sense index) matches the census value for that rid. Assert
the fixed counts: Shape B = 146 and Shape C = 3.

Finally, add a rid-keyed decisions store that mirrors the cycle-1 disposition
format. If that format is absent, fall back to a committed JSON file located
next to the censuses, keyed by `id`, with each value holding the decision and
any split/insert instruction. Support the same decision values that cycle-1 uses
(for example accept-split, insert-sense, no-change, defer), reusing the exact
vocabulary from cycle-1 if present. Provide only the file and the schema for
maintainer decisions; do not fill in the decisions themselves.

Phase 3: Rid-keyed repairs and completeness-gate extension

This phase encodes the approved splits and inserts as rid-keyed repairs. It extends the S5 completeness gate so every Shape B and Shape C entry is either repaired or explicitly dispositioned.

Task 1: Encode splits and inserts as rid-keyed repairs

Translate accepted maintainer decisions into rid-keyed repair records.

• Write repairs to the cycle-1 rid-keyed repair store and format (fallback model: the Record<id, ...> shape of data/admin/annotations.json).
• For Shape B, encode the repair as a split of the swallowed —(n+1)) continuation out of the last sense into a new structural sub-sense.
• For Shape C, encode the repair as the split of the in-text 1)…—2)… run into structural senses with n labels.
• Keep repairs keyed by id and additive, so they layer onto the truth layer without rewriting unrelated content. Preserve byte-stability of untouched entries, consistent with the "clean content is never rewritten" rule in scripts/lib/entry-html.ts.

Task 2: Extend the S5 completeness gate

Extend the completeness gate to cover the new dispositions.

• Extend the cycle-1 S5 gate. If that gate is absent at execution time, extend scripts/validate-data.ts, which is the live completeness gate.
• Add a check: every rid in the Shape B and Shape C censuses must have either a rid-keyed repair or an explicit maintainer decision. Fail CI with a file · id · detail message for any census rid that is neither repaired nor dispositioned.
• Reuse the non-zero exit and problem-reporting pattern already in scripts/validate-data.ts.
• Wire the new check into the existing bun validate:data (or the cycle-1 gate command) so CI runs it.

🤖 Prompt for AI agents

Implement Phase 3: encode the approved maintainer decisions as rid-keyed repairs
and extend the completeness gate to enforce full coverage of the Shape B and
Shape C censuses.

First, write repairs to the cycle-1 rid-keyed repair store and format. If that
store is absent, use the `Record<id, ...>` shape of
`data/admin/annotations.json` as the fallback model. For Shape B, encode the
repair as a split of the swallowed `—(n+1))` continuation out of the last sense
into a new structural sub-sense. For Shape C, encode the repair as a split of
the in-text `1)…—2)…` run into structural senses with `n` labels. Key all
repairs by `id` and keep them additive so they layer onto the truth layer
without rewriting unrelated content. Preserve byte-stability of untouched
entries, consistent with the "clean content is never rewritten" rule in
`scripts/lib/entry-html.ts`.

Next, extend the completeness gate to cover the new dispositions. Extend the
cycle-1 S5 gate if it is present; if it is absent, extend
`scripts/validate-data.ts`, the live completeness gate. Add a check that every
rid in the Shape B and Shape C censuses has either a rid-keyed repair or an
explicit maintainer decision. Fail CI with a `file · id · detail` message for
any census rid that is neither repaired nor dispositioned. Reuse the non-zero
exit and problem-reporting pattern already present in
`scripts/validate-data.ts`. Wire the new check into the existing `bun
validate:data` command, or the cycle-1 gate command, so CI runs it.

Research

This repository holds the Jastrow Dictionary data and tooling. The truth-layer entries live in data/jastrow-part1.jsonl and data/jastrow-part2.jsonl as minified JSONL with keys id, hw, and c.s[] (sense objects with d text and optional n number label). The raw upstream records live in data/raw/jastrow-part*.jsonl with rid and content.senses[]. Structural validation uses scripts/lib/entry-schema.ts, HTML allow-list checks use scripts/lib/entry-html.ts, and scripts/validate-data.ts drives the CI completeness gate (per-line parse, schema, global id uniqueness). Unit tests follow the bun:test pattern in scripts/lib/entry-schema.test.ts. The nearest existing rid-keyed record store is data/admin/annotations.json (Record<id, Annotation[]>).

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Implement Phase 1: rebuild the Shape B and Shape C detectors from the ticket
definitions, run them over the truth-layer corpus, and produce committed literal
census files as the machine-readable source of truth for later phases.

First, add a shared helper module under `scripts/lib/` (for example
`scripts/lib/sense-markers.ts`). In this module, add a function that strips HTML
tags from a sense `d` string and returns plain text, reusing the `htmlparser2`
dependency already used by `scripts/lib/entry-html.ts`. Add a function that
finds in-text numbered markers of the form `N)` and `—N)` in plain text. Add a
citation-noise filter so that a `1)` marker counts only when it is not preceded
by an alphanumeric character or a paren; use this filter for Shape C. Keep the
module dependency-light and side-effect free so tests can import it directly.

Next, add the Shape B detector under `scripts/` (for example
`scripts/census-shape-b.ts`), reusing the cycle-1 S1 splitter output if it is
present. Read `data/jastrow-part1.jsonl` and `data/jastrow-part2.jsonl` line by
line, matching the read pattern in `scripts/validate-data.ts`. For each entry,
read the structural `n` labels in `c.s[]` and confirm they form a clean `1..n`
sequence. Strip tags from the last numbered sense `d` text and detect a trailing
`—(n+1))` marker followed by whitespace. Apply the conservative rule: check the
last sense only, and match a single next marker only; do not follow deeper
chains. Validate that D00919 and D00077 are detected.

Add the Shape C detector under `scripts/` (for example
`scripts/census-shape-c.ts`), reading the truth-layer files with the same
pattern. Select senses that carry no structural `n` label but whose tag-stripped
`d` text contains an in-text run `1)…—2)…`. Apply the citation-noise filter to
the `1)` marker. Validate that A02061, B00516, and U00483 are detected, and that
the total count is 3.

Finally, emit committed literal census files, one per shape. Write these files
to the directory used by the cycle-1 Shape A census; if that directory is
absent, use `data/v2/` and note the location in the review doc. Use a literal,
stable, rid-keyed JSON format keyed by `id`, with each value holding the shape,
the detected marker text, and the sense index, mirroring the cycle-1 Shape A
census or the `data/admin/annotations.json` structure as a fallback model. Sort
keys deterministically so the committed files are byte-stable across reruns.
Record the conservative-measurement note for Shape B
(last-sense/single-next-marker only) in the census file metadata or an adjacent
note.
===============================================================================

Task: 2

Implement Phase 2: turn the Phase 1 censuses into human-readable review docs,
add tests that lock the docs to the censuses, and capture the per-row maintainer
decision.

First, add a generator that reads the Shape B and Shape C census files and
writes review docs. Mirror the cycle-1 Shape A review-doc generator and its
output location, with a fallback location of `docs/v2/`. Produce one row per
census entry, including `id`, `hw`, the current sense text, the detected marker,
and the proposed split or insert. Generate the review doc content
deterministically from the census data so that the equality test is meaningful.
Cover both Shape B and Shape C, either as one doc with two sections or as one
doc per shape, matching the cycle-1 convention.

Next, add a `bun:test` file under `scripts/` (for example
`scripts/census-review.test.ts`), following the `describe`/`it`/`expect` style
of `scripts/lib/entry-schema.test.ts`. In this test file, assert that the set of
rids in each review doc equals the set of rids in the matching census file,
checked in both directions, modeling the set comparison on the `seenIds` map
logic in `scripts/validate-data.ts`. Assert that each per-row field in the
review doc (marker, sense index) matches the census value for that rid. Assert
the fixed counts: Shape B = 146 and Shape C = 3.

Finally, add a rid-keyed decisions store that mirrors the cycle-1 disposition
format. If that format is absent, fall back to a committed JSON file located
next to the censuses, keyed by `id`, with each value holding the decision and
any split/insert instruction. Support the same decision values that cycle-1 uses
(for example accept-split, insert-sense, no-change, defer), reusing the exact
vocabulary from cycle-1 if present. Provide only the file and the schema for
maintainer decisions; do not fill in the decisions themselves.
===============================================================================

Task: 3

Implement Phase 3: encode the approved maintainer decisions as rid-keyed repairs
and extend the completeness gate to enforce full coverage of the Shape B and
Shape C censuses.

First, write repairs to the cycle-1 rid-keyed repair store and format. If that
store is absent, use the `Record<id, ...>` shape of
`data/admin/annotations.json` as the fallback model. For Shape B, encode the
repair as a split of the swallowed `—(n+1))` continuation out of the last sense
into a new structural sub-sense. For Shape C, encode the repair as a split of
the in-text `1)…—2)…` run into structural senses with `n` labels. Key all
repairs by `id` and keep them additive so they layer onto the truth layer
without rewriting unrelated content. Preserve byte-stability of untouched
entries, consistent with the "clean content is never rewritten" rule in
`scripts/lib/entry-html.ts`.

Next, extend the completeness gate to cover the new dispositions. Extend the
cycle-1 S5 gate if it is present; if it is absent, extend
`scripts/validate-data.ts`, the live completeness gate. Add a check that every
rid in the Shape B and Shape C censuses has either a rid-keyed repair or an
explicit maintainer decision. Fail CI with a `file · id · detail` message for
any census rid that is neither repaired nor dispositioned. Reuse the non-zero
exit and problem-reporting pattern already present in
`scripts/validate-data.ts`. Wire the new check into the existing `bun
validate:data` command, or the cycle-1 gate command, so CI runs it.

💡 Iterate on the plan with: `@coderabbitai` <feedback>

Example Feedback
- `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2.
- `@coderabbitai` For design choice 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!

[coderabbitai]

🔗 Related PRs

#19 - 🦠 fix(render): show unnumbered child senses [merged]
#26 - 🦄 new(pipeline): divergence audit (1.2) [merged]
#35 - 📖 doc(specs): fold body model into architecture [merged]
#36 - 🦄 new(pipeline): entry body model (§6.0) [merged]
#41 - 📖 doc(specs): sense structure repair design [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[coderabbitai]

⚠️ Possible Duplicate Issue(s)

• Data: flattened sense-breaks (inline —N)) in 236 entries #18

🔗 Related PRs

#19 - 🦠 fix(render): show unnumbered child senses [merged]
#26 - 🦄 new(pipeline): divergence audit (1.2) [merged]
#35 - 📖 doc(specs): fold body model into architecture [merged]
#36 - 🦄 new(pipeline): entry body model (§6.0) [merged]
#41 - 📖 doc(specs): sense structure repair design [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!