add Chinese NLP support

[BenDerPan]

Add Chinese NLP implement code files , follow the step Tutorials

Description

Added the stopwords of Chinese. and make a spacy style structure of Chinese support.
I tried to make Chinese NLP data model, but something failed, more details see: #1209

Types of changes

• Bug fix (non-breaking change fixing an issue)
• New feature (non-breaking change adding functionality to spaCy)
• Breaking change (fix or feature causing change to spaCy's existing functionality)
• Documentation (addition to documentation of spaCy)

Checklist:

• My change requires a change to spaCy's documentation.
• I have updated the documentation accordingly.
• I have added tests to cover my changes.
• All new and existing tests passed.

[honnibal]

Thanks for this! I've replied on the other thread about the data issue.

One thing to note here is that the tokenizer exceptions mechanism won't work in the ordinary way, because the tokenizer is replaced by Jieba.

Here's what I'd like to happen for Chinese, given effort :). I think we should probably subclass the Tokenizer class, and have it segment each character into a token. However, it should group numbers and Latin-alphabet words into single tokens. It seems to me that a lot of Chinese text does include at least some non-Chinese tokens.

Once the text is segmented into tokens, we should be able to use either the parser or tagger classes to predict the token boundaries. We'd then use the merge() method to join them up.

[BenDerPan]

Thanks for your help, I will try to figure it out. Chinese language seems to be much more complicated.

[wilddylan]

If you need, I'll provide some corpus for what you do.

[BenDerPan]

@wilddylan That's great. thank you, my email: benderpan@163.com

[eromoe]

@BenDerPan I think you should know that spacy break tokenize and pos tagging in two pipeline steps, which make it very difficult to integrate Chinese tokenizer. I opened a issue #854

[howl-anderson]

This feature is very useful! Any progress on this PR now? @BenDerPan

[honnibal]

With Chinese support moving forward in the v2.1.0a0, I think it's time to finally get this merged. Sorry about the long delay on this --- but looking forward to finally produce some models, hopefully by September or October.

Thanks for your patience!

[ines]

See #2591!

[howl-anderson]

@honnibal @ines I already build a complete SpaCy model for Chinese at https://github.com/howl-anderson/Chinese_models_for_SpaCy, I am currently working on improving model performance. I am looking forward to working together to make an official Chinese model for SpaCy.