Skip to content

v3.8.20: Lemmatizer and German tokenizer fixes, thinc v10

Latest

Choose a tag to compare

@github-actions github-actions released this 21 Nov 12:07

✨ New features and improvements

  • Allow thinc v10, the continuation of thinc v8.3.x.
  • Support SudachiPy 0.7 for the Japanese tokenizer.
  • registry.<name>.get_all() now lists spaCy's registered functions and factories without having to load a pipeline first.

🔴 Bug fixes

  • Fix the rule-based lemmatizer preferring the last matching rule over the first, which since v3.8.8 gave wrong lemmas for many English verbs, e.g. "rides" → "rid", "hoped" → "hop", "sites" → "sit" (#13860, #3052). The fix for hyphenated words from #13816 is kept ("co-authored" → "co-author").
  • Keep German gender-inclusive colon forms like "Kund:innen", "Ärzt:in" and "eine:n" as single tokens (#14010). Other colons between letters are still split, e.g. "Achtung:Bitte". Packaged pipelines keep the tokenizer rules they were trained with, so this applies to blank German pipelines and newly trained ones.
  • Show the mecab-ko install message when natto can't load MeCab.
  • Handle Morpheme.split() returning an empty list in the Japanese tokenizer.