Skip to content

pby-v2: Project Ben-Yehuda public-domain Hebrew lines

Choose a tag to compare

@shaypal5 shaypal5 released this 18 Nov 02:21
926c36b

Known issue, superseded by pby-v3. Fifteen works with lines here (40,235 lines, none in il_us) are now held out by HeOCR's P-hist evaluation. See data/pby/v2/KNOWN_ISSUES.md for the counts and how to filter. The assets below are unchanged.

Public-domain Hebrew lines from Project Ben-Yehuda, built by htext pby build. This is v2, part of the htext-cc0 release (decision 0022). It supersedes pby-v1.

Pins

  • PBY dump: projectbenyehuda/public_domain_dump @ 5e4277f, VERSION 2026-03.
  • Builder: htext @ 39a4df8.
  • Wrap check and S10b length profile: hocrsyngen @ 88c61af.
  • Eval reservations: eval_reservations_v2.json, sha256 2abce5ea….
  • Parameters: as_of 2026, seed 15, no per-work cap. Genres: prose, article, memoir, letters, fables, poetry, reference and drama.

Credit: Project Ben-Yehuda volunteers.

Corpus card: data/pby/v2/CORPUS_CARD.md (corpus_card.json).

Asset Lines Contents
pby_lines_il_plain_v2.txt 12,318,857 public domain in Israel, plain
pby_lines_il_niqqud_v2.txt 1,680,295 public domain in Israel, pointed
pby_lines_il_us_plain_v2.txt 407,456 also public domain in the US, plain
pby_lines_il_us_niqqud_v2.txt 18,613 also public domain in the US, pointed
pby_line_index.v1.jsonl.gz 13,999,152 rows per il line: sha256, PBY work id, subset, genre, period, il_us

What changed since v1

  • Two more genres: reference and drama.
  • No cap on lines per work. The line index keeps the work id, so per-work shares are set at mix time.
  • 139 reviewed manual death years.
  • Eval-reserved works are left out whole.
  • PBY's inline verse-break / now ends a line.
  • A work with no lines has htext_release: null.
  • 25,282 works have lines.

The card, SHA256SUMS, the exclusion list and the gzipped rights index are in git under data/pby/v2/ at this tag. Verify the downloaded assets with:

sha256sum -c --ignore-missing data/pby/v2/SHA256SUMS

Licences. The outputs are CC0-1.0. The source texts are public domain, as PBY's dump licence states. pd_israel and pd_us are the project's reading of the law, not legal advice. See docs/pby_corpus.md.