You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
v3.8.20: Lemmatizer and German tokenizer fixes, thinc v10
Allow thinc v10, the continuation of thinc v8.3.x.
Support SudachiPy 0.7 for the Japanese tokenizer.
registry.<name>.get_all() now lists spaCy's registered functions and factories without having to load a pipeline first.
🔴 Bug fixes
Fix the rule-based lemmatizer preferring the last matching rule over the first, which since v3.8.8 gave wrong lemmas for many English verbs, e.g. "rides" → "rid", "hoped" → "hop", "sites" → "sit" (#13860, #3052). The fix for hyphenated words from #13816 is kept ("co-authored" → "co-author").
Keep German gender-inclusive colon forms like "Kund:innen", "Ärzt:in" and "eine:n" as single tokens (#14010). Other colons between letters are still split, e.g. "Achtung:Bitte". Packaged pipelines keep the tokenizer rules they were trained with, so this applies to blank German pipelines and newly trained ones.
Show the mecab-ko install message when natto can't load MeCab.
Handle Morpheme.split() returning an empty list in the Japanese tokenizer.