You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hello, thanks for this interesting project!
Currently in my nlp pipelines I'm using Jieba / Mecab as chinese / segmenter and japanese/tokenizer modules.
Is it safe to use FastHan as a replacement of Mecab han tokenizer?
Thank you!
The text was updated successfully, but these errors were encountered:
I'm not familiar with mecab, but I’m certain that fastHan cannot be used as japanese tokenizer. Because it was only trained on Chinese data samples, it can not even recognize japanese characters
Hello, thanks for this interesting project!
Currently in my nlp pipelines I'm using Jieba / Mecab as chinese / segmenter and japanese/tokenizer modules.
Is it safe to use FastHan as a replacement of Mecab han tokenizer?
Thank you!
The text was updated successfully, but these errors were encountered: