Skip to content

v0.3.0 — annotate() offset fix (superseded by v0.3.1)

Choose a tag to compare

@ivrit ivrit released this 29 Jul 16:20
· 17 commits to main since this release

Superseded — use v0.3.1 instead.
This release contains a regression in annotate() (see Known issue below). Do not pin to it.

What shipped

annotate() now tokenizes with the document tokenizer, so every item carries its absolute
character offset. A surface form that recurs in a sentence is span-pooled at its own
occurrence instead of falling back to the first match:

"האיש שם את התיק שם ליד הדלת"   ("The man put the bag there, by the door")
2nd שם   before: pos=VERB   (pooled at the first שם)
         after:  pos=ADV    (correct — "there", not "put")

annotate() records also gained a start key, and gender-slash forms are reported with
their full surface (כותב/ת), matching lemmatize_text().

Known issue (fixed in v0.3.1)

annotate() passed the raw token text to the retriever. An inclusive-writing form like
כותב/ת is not a lemma-bank key, so retrieval landed on junk for those tokens:

כותב/ת -> ות""ת      חבר/ה -> חבר'ה

Only annotate() and the CLI were affected; lemmatize_text() was always correct.

Model weights

Unchanged. Pairs with HebArabNlpProject/shoshan revision e5839654041f812c96833f9f46ccfdeb50198d75.