Skip to content

Releases: yishn/chinese-tokenizer

chinese-tokenizer v2.4.0

05 Jun 13:01
@yishn yishn
v2.4.0
abf208f

Choose a tag to compare

View all tags
chinese-tokenizer v2.4.0 Latest
Latest
  • Add position property to tokens
Assets 2
Loading

chinese-tokenizer v2.1.0

29 May 01:41
@yishn yishn
v2.1.0
cea7029

Choose a tag to compare

View all tags
chinese-tokenizer v2.1.0
  • Handle tokenization of non-Chinese text gracefully
Loading