Skip to content

tanakh-v1: the Westminster Leningrad Codex, pointed and plain

Choose a tag to compare

@shaypal5 shaypal5 released this 18 Nov 16:58
3d9b234

Known issue (2026-11-19): HeOCR's evaluation now reserves four verse ranges for its pointed-print test set (eval_reservations_v5.json, sha256 8a869b0f…): Neh 3:1–15, Ezek 40:5–19, Ezra 2:1–28, and Neh 7:6–32. These files have 165 lines of them each, and later builds leave them out. See KNOWN_ISSUES.md for how to filter them. The assets are unchanged.

The Hebrew Bible, as the Westminster Leningrad Codex (WLC 4.20) gives it, cut into lines in two forms: pointed (S10b vowel points, with no cantillation or meteg) and plain. Built by htext tanakh build. This is v1, part of the htext-cc0 release (decision 0022).

Rights. The Leningrad Codex is about a thousand years old, and the WLC text is in the public domain. OSHB's README says "The text of the WLC remains in the Public Domain." Only the words are used. OSHB's lemma and morphology layers (CC BY 4.0) and its / morpheme dividers are not. The edition is WLC, not UXLC.

Biblical quotations. Verses are quoted in many printed Hebrew pages, HeOCR's evaluation pages among them. Run the training-time overlap check (CLEAN) against your evaluation set before training on this corpus.

Pins

  • Source: openscriptures/morphhb @ db6c5c9, wlc/.
  • Builder: htext @ 87fe9be. The card and files are committed at 3d9b234.
  • Wrap check and S10b length profile: hocrsyngen @ 88c61af.
  • Eval reservations: eval_reservations_v3.json, sha256 969618a8…. No biblical unit is reserved.

Credit: Westminster Leningrad Codex (public domain), via the Open Scriptures Hebrew Bible.

Corpus card: data/tanakh/v1/CORPUS_CARD.md (corpus_card.json).

Asset Lines Contents
tanakh_lines_niqqud_v1.txt 58,120 pointed
tanakh_lines_plain_v1.txt 58,074 plain
tanakh_line_index.v1.jsonl.gz 116,194 rows per line: subset, sha256, verse ref, part and parts

Check the assets against SHA256SUMS.

Signed: htext[claude/claude-code#shay-macbook-m4]@heocr/opensource