Repository navigation
Known issue (2026-11-20): 13 works with lines in v3 are now reserved by HeOCR's evaluation (7,734 lines). See KNOWN_ISSUES.md. The Hugging Face and Zenodo copies leave them out.
Zenodo: 10.5281/zenodo.24384745 (all versions: 10.5281/zenodo.24384744). Hugging Face: HeOCR/htext-pby, without the lines that overlap HeOCR's evaluation text; the Zenodo record holds those same files.
Public-domain Hebrew lines from Project Ben-Yehuda, built by htext pby build. This is v3, part of the htext-cc0 release (decision 0022). It supersedes pby-v2.
Why v3. v3 is v2 rebuilt with HeOCR's eval_reservations_v4.json (sha256 c8842a90…) instead of eval_reservations_v2.json (sha256 2abce5ea…). v4 adds the P-hist eval hold-out and its top-up. The 31 reserved PBY works are left out whole. Fifteen of them had lines in v2 (40,235 lines); see data/pby/v2/KNOWN_ISSUES.md. Nothing else changed.
Pins
- PBY dump:
projectbenyehuda/public_domain_dump@5e4277f, VERSION2026-03. - Builder:
htext@1feab81. - Wrap check and S10b length profile:
hocrsyngen@88c61af. - Eval reservations:
eval_reservations_v4.json, sha256c8842a90…. - Parameters:
as_of2026, seed 15, no per-work cap. Genres: prose, article, memoir, letters, fables, poetry, reference and drama.
Credit: Project Ben-Yehuda volunteers.
Corpus card: data/pby/v3/CORPUS_CARD.md (corpus_card.json).
| Asset | Lines | Contents |
|---|---|---|
pby_lines_il_plain_v3.txt |
12,278,848 | public domain in Israel, plain |
pby_lines_il_niqqud_v3.txt |
1,680,290 | public domain in Israel, pointed |
pby_lines_il_us_plain_v3.txt |
407,456 | also public domain in the US, plain (same bytes as v2) |
pby_lines_il_us_niqqud_v3.txt |
18,613 | also public domain in the US, pointed (same bytes as v2) |
pby_line_index.v1.jsonl.gz |
13,959,138 rows | per il line: sha256, PBY work id, subset, genre, period, il_us |
25,267 works have lines, all htext-cc0.
The card, SHA256SUMS, the exclusion list and the gzipped rights index are in git under data/pby/v3/ at this tag. Verify the downloaded assets with:
sha256sum -c --ignore-missing data/pby/v3/SHA256SUMSLicences. The outputs are CC0-1.0. The source texts are public domain, as PBY's dump licence states. pd_israel and pd_us are the project's reading of the law, not legal advice. See docs/pby_corpus.md.