Identify Chinese

[dsewnr]

No description provided.

[valeriansaliou]

Hi! Not sure about this, can you check the stopwords before and see which variant of Chinese it implements? (Does it implement both?)

https://github.com/valeriansaliou/sonic/blob/master/src/stopwords/cmn.rs

[dsewnr]

Hi! Not sure about this, can you check the stopwords before and see which variant of Chinese it implements? (Does it implement both?)

https://github.com/valeriansaliou/sonic/blob/master/src/stopwords/cmn.rs

cmn.rs is implemented for Simplified Chinese.

[dsewnr]

There are two new files for both Chinese:

src/stopwords/csd.rs Chinese (Simplified)
src/stopwords/ctl.rs Chinese (Traditional)

[valeriansaliou]

Ah, thanks! Can you also PR https://github.com/greyblake/whatlang-rs with the added "CTL" ISO code? We depend on this library for languages mappings and trigrams language guessing, so I'd first need this library to support "CTL" before I can accept this PR.

[dsewnr]

"CMN" ISO code is include Simplified Chinese and Traditional Chinese.
I've merged Traditional Chinese stop words into cmn.rs.
whatlang-rs can detect both of Simplified Chinese and Traditional Chinese by "CMN" unicode range.

[valeriansaliou]

Ah that's perfect! Thanks, I'm accepting this now.