Repository navigation
v0.5.0
Hebrew G2P with the full decode stack, and pointed input that finally counts.
Vowelized (pointed) input
Niqqud in the input used to tokenize to [UNK] and leak the raw combining marks into the IPA. It is now handled in two modes:
niqqud="strip"(default) — points are dropped, so pointed text decodes byte-identically to the same text unpointed.niqqud="use"— points are read as evidence. Each vowel sign, dagesh qal and shin/sin dot is pinned into the exact-MAP energy as a hard constraint, so the model predicts only what the pointing leaves open.
g2p = G2P(niqqud="use")
g2p.phonemize("סֵפֶר") # sˈefeʁ book
g2p.phonemize("סַפָּר") # sˈapaʁ barber
g2p.phonemize("סָפַר") # sˈafaʁ counted
g2p.phonemize("סִפֵּר") # sˈipeʁ toldAll four are sˈefeʁ when stripping — the skeleton ספר cannot tell them apart. Qamats and shva stay ambiguous by design (qamats qatan shares its sign; shva na vs nah), and a constraint that would leave a letter with no legal reading is dropped, so sloppy pointing degrades to the normal decode. Niqqud does not mark stress, so stress stays the model's own call.
On 24 hand-checked pointed words: 24/24 vowels and consonants right (13/24 stripping), 19/24 including stress.
Decoding
- Exact-MAP cascade decode, on by default: consonant, vowel and stress decoded jointly under
E(c,v,s) = log P(c) + log P(v|c) + log P(s|c,v), with per-letter legality and "stress needs a vowel" as hard constraints. This removes the stranded-stress-mark class — a mark can no longer land on a letter with no vowel. - Attested-reading rescorer, on whenever
datastore.jsonsits beside the weights — which it now does, since the automatic download fetches it too. - Force lexicon —
G2P(..., lexicon={...})or a TSV path. - Long inputs past the encoder's 2,046-character window are split on sentence, then comma, then word boundaries, with a raw-Hebrew-in-output backstop.
Packaging
- The Hebrew number front end moved to
hebrew-num2wordsand is now a dependency rather than ~560 bundled lines. Output is unchanged and the API is re-exported. G2P()also acceptssession_options,repo_id,filenameandcache_dir.- Includes 0.4.0, which was never published: ONNX Runtime is no longer a hard dependency — install
renikud-plus[cpu]orrenikud-plus[gpu], or bring your own build.
Full notes in CHANGELOG.md.