Skip to content

v0.1.96

Compare
Choose a tag to compare
@taku910 taku910 released this 17 Jun 16:55
d8711f5

Updates

  • Improves the performance of unigram training
  • Updated the nfkc normalization with the latest ICU module.
  • Stop handling zero-width-joiner string as whitespace.

New features

  • added new sampling algorithm without replacement.
  • added API for new sampling and perplexity calculation.
  • added allow_whitespace_only_pieces mode.