Skip to content

feat(db): line_dh — dibbur-hamatchil index - #20

Merged
Y-PLONI merged 2 commits into
otzariafrom
feat/line-dh-index
Aug 31, 2026
Merged

feat(db): line_dh — dibbur-hamatchil index#20
Y-PLONI merged 2 commits into
otzariafrom
feat/line-dh-index

Conversation

@palmoni5

@palmoni5 palmoni5 commented Aug 31, 2026

Copy link
Copy Markdown
Member

Resolves the DB side of the forum request: searching Rashi/Tosafot by dibbur hamatchil, and rendering dibburim as sub-headings.

Problem

A commentary's dibbur hamatchil is embedded in each line's content, and the TOC stops at daf/perek level (Rashi on Berakhot: 126 TOC entries for 3,265 lines). There is no way to search a commentary by its opening words, or to navigate a daf's comments by dibbur.

What this adds

line_dh(bookId, dhText, lineIndex)normalised text, not a hash (unlike line_ref), so the client can prefix-match while the user types, straight off the WITHOUT ROWID PK btree (dhText >= ?prefix AND dhText < ?prefixEnd), and render the rows as virtual sub-headings — without touching tocEntry/tocText.

Two shapes cover the corpus, one per book:

  • <b>…</b> prefix — Rashi on Tanakh, Mishnah commentaries, most Otzaria-sourced books.
  • Text before the first spaced dash — Sefaria's Talmud commentaries (עד סוף האשמורה הראשונה – שליש הלילה…), including the daf's first comment that ends with . instead of a dash.

False-positive defences (measured on a real DB)

Bold ≠ dibbur in general: 2.0M lines open with <b>, including discourse-word emphasis (והנה ×28K, אמנם ×10K), structural markers (מתני', גמרא, בא"ד), and whole-line-bold decorated headings (57K). Three layers keep them out:

  1. Book-dominance gate (the main one): a book is indexed only when one shape covers ≥40% of its content lines with ≥10 hits. Books that merely bold an occasional word or use an incidental dash never reach the threshold.
  2. Line guards: real commentary text must follow the dibbur (whole-line bold is a heading), length ≤100 chars, <h1><h6> excluded.
  3. Structural-marker stop-list, compared before quote-stripping — so the locator תוד"ה is dropped while the genuine dibbur תודה survives.

Parity

The stored value is DhKey.normalize() output; the Otzaria client must normalise the typed query with the same rules. Both sides pin to dh_key_fixtures.json (DhKeyParityTest here), same pattern as ref_key_fixtures.json.

Verification

  • Regression coverage for extraction, normalisation, base-text exclusion, atomic rollback, and manifest compatibility — passes.
  • Full build over a real 7GB seforim.db: 1,919,349 dibburim over 3,435 non-base/dependent books in 49s; pure base texts contribute zero rows.
  • buildLineDhIndex runs after every book-writing stage and is idempotent (rebuilds from scratch).

🤖 Generated with Claude Code

Review hardening

  • Rebased directly onto otzaria; duplicated line_ref commits were removed.
  • Pure base texts are excluded, while dependent works marked as base remain eligible through book_base_text.
  • Structural markers handle Hebrew, ASCII, doubled-apostrophe, and curly quote variants.
  • Rebuilds are transactional and roll back to the previous index on failure.
  • Schema-4 manifests now emit patchFormatVersion, matching the updater contract.

palmoni5 and others added 2 commits September 1, 2026 00:42
Rashi/Tosafot (and commentaries generally) cannot be searched or navigated
by their dibbur hamatchil: it is embedded in each line's content, and the
TOC stops at daf/perek level (Rashi on Berakhot: 126 TOC entries for 3,265
lines). This adds line_dh, mapping (bookId, dhText) to lineIndex, so the
client can prefix-match dibburim as the user types and render them as
virtual sub-headings — without touching tocEntry/tocText.

Two shapes cover the corpus, one per book: a <b>…</b> prefix (Rashi on
Tanakh, Mishnah commentaries) and the text before the first spaced dash
(Sefaria's Talmud commentaries). DhExtractor extracts conservatively —
short dibbur, real commentary after it, structural markers (מתני', גמרא,
בא"ד…) excluded — and BuildLineDhIndexCli indexes a book only when one
shape dominates its content lines (>=40%, >=10 hits), which keeps books
that merely bold an occasional word or use an incidental dash out entirely.

The stored value is normalised TEXT (DhKey, in core), not a hash like
line_ref: prefix search needs the text, and the (bookId, dhText, …) PK
btree of the WITHOUT ROWID table serves range scans directly. The client
must normalise the typed query with the same rules, so the normalisation
is pinned to dh_key_fixtures.json for cross-repo parity.

On a real v23 seforim.db: 1,935,274 dibburim over 3,451 books, built in
under a minute; Rashi on Berakhot covered 3,092/3,139 content lines.
buildLineDhIndex runs after every book-writing stage and is idempotent.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Y-PLONI
Y-PLONI force-pushed the feat/line-dh-index branch from da2ca1a to ce89be1 Compare August 31, 2026 22:18
@Y-PLONI
Y-PLONI changed the base branch from master to otzaria August 31, 2026 22:20
@Y-PLONI Y-PLONI closed this Aug 31, 2026
@Y-PLONI Y-PLONI reopened this Aug 31, 2026
@Y-PLONI
Y-PLONI merged commit fda728e into otzaria Aug 31, 2026
4 checks passed
@palmoni5
palmoni5 deleted the feat/line-dh-index branch August 31, 2026 22:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants