Skip to content

simplemma-2.0.0

Latest

Choose a tag to compare

@adbar adbar released this 12 Aug 13:48
· 3 commits to main since this release
22a19bd
  • New languages: Arabic (ar), Ancient Greek (grc), Hebrew (he) and Malayalam (ml) (#180)
  • All dictionaries rebuilt from fresh sources with reviewed correction lists: improved accuracy for many languages (#178, #180, #184)
  • New front-coded dictionary format shrinks the shipped data from 67 MB to 19 MB
  • More accurate rules covering more languages (#175)
  • Better affix handling: new clitic and apostrophe strategies, simpler greedy search, updated prefixes (#176)
  • New morpheme decomposition strategy, used for Tagalog and Indonesian (#180)
  • Faster, more precise tokenizer, and new split_sentences() function for sentence splitting (#177, #182)
  • Lower memory usage: shared dictionary cache and new low_memory flag (#178, #181)
  • Language detector fixes (#172)