v1.3.0

Summary

• Improving speed by 10~20x by using fasttext and langua only (remove langdetect), also using simple splitting logic instead of machine learning based wtpsplit without lowering the performance.
• Split languages other than Chinese, Japanese, Korean using space
• Split Chinese and Japanese based on budoux
  • using ja parser first to split out Japanese words
  • using zh parser to deal with kanji combine with Chinese characters
  • rule base merge including: merge substrings if both side of it are in same language and itself is very short.

What's Changed

• Close #1 by @DoodleBears in #2

New Contributors

• @DoodleBears made their first contribution in #2

Full Changelog: v1.2.0...v1.3.0