Skip to content

v0.5.0

Choose a tag to compare

@maxmelichov maxmelichov released this 29 Aug 22:06
· 5 commits to main since this release

Hebrew G2P with the full decode stack, and pointed input that finally counts.

Vowelized (pointed) input

Niqqud in the input used to tokenize to [UNK] and leak the raw combining marks into the IPA. It is now handled in two modes:

  • niqqud="strip" (default) — points are dropped, so pointed text decodes byte-identically to the same text unpointed.
  • niqqud="use" — points are read as evidence. Each vowel sign, dagesh qal and shin/sin dot is pinned into the exact-MAP energy as a hard constraint, so the model predicts only what the pointing leaves open.
g2p = G2P(niqqud="use")
g2p.phonemize("סֵפֶר")   # sˈefeʁ   book
g2p.phonemize("סַפָּר")   # sˈapaʁ   barber
g2p.phonemize("סָפַר")   # sˈafaʁ   counted
g2p.phonemize("סִפֵּר")   # sˈipeʁ   told

All four are sˈefeʁ when stripping — the skeleton ספר cannot tell them apart. Qamats and shva stay ambiguous by design (qamats qatan shares its sign; shva na vs nah), and a constraint that would leave a letter with no legal reading is dropped, so sloppy pointing degrades to the normal decode. Niqqud does not mark stress, so stress stays the model's own call.

On 24 hand-checked pointed words: 24/24 vowels and consonants right (13/24 stripping), 19/24 including stress.

Decoding

  • Exact-MAP cascade decode, on by default: consonant, vowel and stress decoded jointly under E(c,v,s) = log P(c) + log P(v|c) + log P(s|c,v), with per-letter legality and "stress needs a vowel" as hard constraints. This removes the stranded-stress-mark class — a mark can no longer land on a letter with no vowel.
  • Attested-reading rescorer, on whenever datastore.json sits beside the weights — which it now does, since the automatic download fetches it too.
  • Force lexicon — G2P(..., lexicon={...}) or a TSV path.
  • Long inputs past the encoder's 2,046-character window are split on sentence, then comma, then word boundaries, with a raw-Hebrew-in-output backstop.

Packaging

  • The Hebrew number front end moved to hebrew-num2words and is now a dependency rather than ~560 bundled lines. Output is unchanged and the API is re-exported.
  • G2P() also accepts session_options, repo_id, filename and cache_dir.
  • Includes 0.4.0, which was never published: ONNX Runtime is no longer a hard dependency — install renikud-plus[cpu] or renikud-plus[gpu], or bring your own build.

Full notes in CHANGELOG.md.