v0.3.1 — gender-slash retrieval fix
Current release. Supersedes v0.3.0.
pip install git+https://github.com/ivrit/shoshan.git@v0.3.1Fixed
annotate() retrieved inclusive-writing gender-slash forms on their raw surface
instead of the collapsed base. The lemma bank is keyed on real lemmas, and כותב/ת is not
one, so those tokens got junk lemmas:
before: כותב/ת -> ות""ת חבר/ה -> חבר'ה
after: כותב/ת -> כותב חבר/ה -> חבר
annotate() now feeds the model collapse_gender_slash(token) and reports the original
surface — the same split lemmatize_text() has always used, so the two paths agree on the
lemma as well as the token. Only annotate() and the CLI were affected.
Also in this line (from v0.3.0)
annotate() span-pools each token at its own character offset, so a form that recurs in a
sentence is read in its own context (שם "put" vs. שם "there"). Records carry a start key.
Model weights
Unchanged since v0.2.0. Pairs with HebArabNlpProject/shoshan revision
e5839654041f812c96833f9f46ccfdeb50198d75. Nothing to re-download.
Demo
https://huggingface.co/spaces/noamor/shoshan-demo runs this version.