Repository navigation
pby-v2: Project Ben-Yehuda public-domain Hebrew lines
Known issue, superseded by
pby-v3. Fifteen works with lines here (40,235 lines, none inil_us) are now held out by HeOCR's P-hist evaluation. Seedata/pby/v2/KNOWN_ISSUES.mdfor the counts and how to filter. The assets below are unchanged.
Public-domain Hebrew lines from Project Ben-Yehuda, built by htext pby build. This is v2, part of the htext-cc0 release (decision 0022). It supersedes pby-v1.
Pins
- PBY dump:
projectbenyehuda/public_domain_dump@5e4277f, VERSION2026-03. - Builder:
htext@39a4df8. - Wrap check and S10b length profile:
hocrsyngen@88c61af. - Eval reservations:
eval_reservations_v2.json, sha2562abce5ea…. - Parameters:
as_of2026, seed 15, no per-work cap. Genres: prose, article, memoir, letters, fables, poetry, reference and drama.
Credit: Project Ben-Yehuda volunteers.
Corpus card: data/pby/v2/CORPUS_CARD.md (corpus_card.json).
| Asset | Lines | Contents |
|---|---|---|
pby_lines_il_plain_v2.txt |
12,318,857 | public domain in Israel, plain |
pby_lines_il_niqqud_v2.txt |
1,680,295 | public domain in Israel, pointed |
pby_lines_il_us_plain_v2.txt |
407,456 | also public domain in the US, plain |
pby_lines_il_us_niqqud_v2.txt |
18,613 | also public domain in the US, pointed |
pby_line_index.v1.jsonl.gz |
13,999,152 rows | per il line: sha256, PBY work id, subset, genre, period, il_us |
What changed since v1
- Two more genres: reference and drama.
- No cap on lines per work. The line index keeps the work id, so per-work shares are set at mix time.
- 139 reviewed manual death years.
- Eval-reserved works are left out whole.
- PBY's inline verse-break
/now ends a line. - A work with no lines has
htext_release: null. - 25,282 works have lines.
The card, SHA256SUMS, the exclusion list and the gzipped rights index are in git under data/pby/v2/ at this tag. Verify the downloaded assets with:
sha256sum -c --ignore-missing data/pby/v2/SHA256SUMSLicences. The outputs are CC0-1.0. The source texts are public domain, as PBY's dump licence states. pd_israel and pd_us are the project's reading of the law, not legal advice. See docs/pby_corpus.md.