add jieba tokenizer for Chinese

[vincascm]

add jieba tokenizer for Chinese

[valeriansaliou]

Hello there! Your code is being Chinese-specific, which is not fine by me in terms of code hygiene. Can you also explain the purpose of this PR? Not familiar with the Jieba tokenizer.

[vincascm]

@valeriansaliou

1. Chinese do not have a trivial word segmentation process . https://en.wikipedia.org/wiki/Text_segmentation

2. jieba is a widely used Chinese tokenizer. https://github.com/fxsjy/jieba#jieba-1

3. may add special feature for Chinese.

[sftblw]

It is common to use tokenizer in CJK, even in elasticsearch (google search of es + mecab tokenizer). As an example, NLP library spaCy requires external tokenizer installed in system for...

chinese	japanese	korean
jieba	mecab via fugashi	mecab-ko via natto-py

[dzcpy]

Have you guys found a solution for Chinese tokenizing?
Maybe we can borrow some ideas from https://github.com/tantivy-search/tantivy, like make the tokenizer configurable in order to support CJK languages?

[hajiuxbz]

Hope to merge

[rcy17]

One and a half years later...... Sincerely hope to merge.

[valeriansaliou]

Thanks for the PR, sorry it took so much time. Currently updating the jieba library to latest and refactoring code to make it an optional feature, as not all Sonic users need Chinese tokenization support (the library adds some size overhead).