Skip to content

pby-v3: Project Ben-Yehuda public-domain Hebrew lines

Latest

Choose a tag to compare

@shaypal5 shaypal5 released this 18 Nov 20:52
78d7e39

Known issue (2026-11-20): 13 works with lines in v3 are now reserved by HeOCR's evaluation (7,734 lines). See KNOWN_ISSUES.md. The Hugging Face and Zenodo copies leave them out.

Zenodo: 10.5281/zenodo.24384745 (all versions: 10.5281/zenodo.24384744). Hugging Face: HeOCR/htext-pby, without the lines that overlap HeOCR's evaluation text; the Zenodo record holds those same files.

Public-domain Hebrew lines from Project Ben-Yehuda, built by htext pby build. This is v3, part of the htext-cc0 release (decision 0022). It supersedes pby-v2.

Why v3. v3 is v2 rebuilt with HeOCR's eval_reservations_v4.json (sha256 c8842a90…) instead of eval_reservations_v2.json (sha256 2abce5ea…). v4 adds the P-hist eval hold-out and its top-up. The 31 reserved PBY works are left out whole. Fifteen of them had lines in v2 (40,235 lines); see data/pby/v2/KNOWN_ISSUES.md. Nothing else changed.

Pins

  • PBY dump: projectbenyehuda/public_domain_dump @ 5e4277f, VERSION 2026-03.
  • Builder: htext @ 1feab81.
  • Wrap check and S10b length profile: hocrsyngen @ 88c61af.
  • Eval reservations: eval_reservations_v4.json, sha256 c8842a90….
  • Parameters: as_of 2026, seed 15, no per-work cap. Genres: prose, article, memoir, letters, fables, poetry, reference and drama.

Credit: Project Ben-Yehuda volunteers.

Corpus card: data/pby/v3/CORPUS_CARD.md (corpus_card.json).

Asset Lines Contents
pby_lines_il_plain_v3.txt 12,278,848 public domain in Israel, plain
pby_lines_il_niqqud_v3.txt 1,680,290 public domain in Israel, pointed
pby_lines_il_us_plain_v3.txt 407,456 also public domain in the US, plain (same bytes as v2)
pby_lines_il_us_niqqud_v3.txt 18,613 also public domain in the US, pointed (same bytes as v2)
pby_line_index.v1.jsonl.gz 13,959,138 rows per il line: sha256, PBY work id, subset, genre, period, il_us

25,267 works have lines, all htext-cc0.

The card, SHA256SUMS, the exclusion list and the gzipped rights index are in git under data/pby/v3/ at this tag. Verify the downloaded assets with:

sha256sum -c --ignore-missing data/pby/v3/SHA256SUMS

Licences. The outputs are CC0-1.0. The source texts are public domain, as PBY's dump licence states. pd_israel and pd_us are the project's reading of the law, not legal advice. See docs/pby_corpus.md.