1.0.0

First the official release! 🎉

• Fixed the APIs for tokenizers and dictionaries
• Ready for production usage and long term support

Improvements (since 0.0.4):

• (default-dict) Improve number tokenization consistency
• (default-dict) More compact (3.4 MB)

Support Sudachi Dictionary

• Start supporting Sudachi integration and dictionary formats in kotori-sudachi
• Refactoring new tools and utilities

Major performance improvement

• Performance improvement with better data matching and lattice implementation
• Inner binary dictionary format (=> Smaller Jar file)

The first release (Fixed)

A simple Tokenizer implementation with MeCab's ipadic as the default dictionary.

The first release

A simple Tokenizer implementation with MeCab's ipadic as the default dictionary.