v0.3.0 — annotate() offset fix (superseded by v0.3.1)
Superseded — use v0.3.1 instead.
This release contains a regression inannotate()(see Known issue below). Do not pin to it.
What shipped
annotate() now tokenizes with the document tokenizer, so every item carries its absolute
character offset. A surface form that recurs in a sentence is span-pooled at its own
occurrence instead of falling back to the first match:
"האיש שם את התיק שם ליד הדלת" ("The man put the bag there, by the door")
2nd שם before: pos=VERB (pooled at the first שם)
after: pos=ADV (correct — "there", not "put")
annotate() records also gained a start key, and gender-slash forms are reported with
their full surface (כותב/ת), matching lemmatize_text().
Known issue (fixed in v0.3.1)
annotate() passed the raw token text to the retriever. An inclusive-writing form like
כותב/ת is not a lemma-bank key, so retrieval landed on junk for those tokens:
כותב/ת -> ות""ת חבר/ה -> חבר'ה
Only annotate() and the CLI were affected; lemmatize_text() was always correct.
Model weights
Unchanged. Pairs with HebArabNlpProject/shoshan revision e5839654041f812c96833f9f46ccfdeb50198d75.