Skip to content

v0.3.1 — gender-slash retrieval fix

Choose a tag to compare

@ivrit ivrit released this 29 Jul 16:21
· 16 commits to main since this release

Current release. Supersedes v0.3.0.

pip install git+https://github.com/ivrit/shoshan.git@v0.3.1

Fixed

annotate() retrieved inclusive-writing gender-slash forms on their raw surface
instead of the collapsed base. The lemma bank is keyed on real lemmas, and כותב/ת is not
one, so those tokens got junk lemmas:

before:  כותב/ת -> ות""ת       חבר/ה -> חבר'ה
after:   כותב/ת -> כותב        חבר/ה -> חבר

annotate() now feeds the model collapse_gender_slash(token) and reports the original
surface — the same split lemmatize_text() has always used, so the two paths agree on the
lemma as well as the token. Only annotate() and the CLI were affected.

Also in this line (from v0.3.0)

annotate() span-pools each token at its own character offset, so a form that recurs in a
sentence is read in its own context (שם "put" vs. שם "there"). Records carry a start key.

Model weights

Unchanged since v0.2.0. Pairs with HebArabNlpProject/shoshan revision
e5839654041f812c96833f9f46ccfdeb50198d75. Nothing to re-download.

Demo

https://huggingface.co/spaces/noamor/shoshan-demo runs this version.