Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How well will this handle Chinese? #23

Open
benjiwheeler opened this issue Jun 27, 2019 · 1 comment
Open

How well will this handle Chinese? #23

benjiwheeler opened this issue Jun 27, 2019 · 1 comment

Comments

@benjiwheeler
Copy link

I know that Chinese does not have the same density of spaces as English and most languages; a Chinese character is more analogous to an English word than an English letter.

Would you expect your classifier to treat Chinese characters as letters, or as words?

@toonimoadi
Copy link

toonimoadi commented Oct 21, 2021

Depends on your tokenizer.
By default it will tokenize Chinese characters as letters, but you can easily modify it with the following tokenizer

bayes({
    tokenizer: function (text) { return text.replace(/\s/g, '').split('') }
})

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants