Tokenizer for Ja/Ko

[kination]

Hello~
I'm currently testing tokenizer with Japanese/Korean, but seems it is not working correctly.

Is there some working plan for this?

Thanks.

[kination]

In test code, when I add as

    let analyzed = analyzer.analyze("Zut, l’aspirateur, j’ai oublié de l’éteindre !");
    println!("{:?}", analyzed.tokens().map(|t| t.text().to_string()).collect::<Vec<_>>());
    println!("{:?}", analyzed.reconstruct().map(|(s, _)| s.to_string()).collect::<String>());

    let analyzed_kor = analyzer.analyze("안녕하세요, 오늘은 날씨가 매우 춥습니다!");
    println!("{:?}", analyzed_kor.tokens().map(|t| t.text().to_string()).collect::<Vec<_>>());
    println!("{:?}", analyzed_kor.reconstruct().map(|(s, _)| s.to_string()).collect::<String>());

result will be:

["zut", ", ", "l", "’", "aspirateur", ", ", "j", "’", "ai", " ", "oublie", " ", "de", " ", "l", "’", "eteindre", " !"]
"Zut, l’aspirateur, j’ai oublié de l’éteindre !"
["안", "녕", "하", "세", "요", ", ", "오", "늘", "은", " ", "날", "씨", "가", " ", "매", "우", " ", "춥", "습", "니", "다", "!"]
"안녕하세요, 오늘은 날씨가 매우 춥습니다!"

but in my mind, 3rd line should be

"안녕하세요", ",", "오늘은", "날씨가", "매우", "춥습니다!"

and it is same in japanese too...

[kination]

Follow-up by #49

[curquiza]

Japanese support was added, however not the Korean language support.
For people wanting the Korean support, please open a discussion in the product repo so that the product team is aware of your need 😄