Split longer word, rather than word by word

[minhtringo141]

Firstly, greatly appreciate your library, it's very useful and easy to to use. But when using i've a trouble,.In Vietnamese vocabulary, a meaning word sometimes includes more than 1 word. For example, sentence "I live in Ha Noi", i want "Ha Noi" will stand together after being split. Is there any way or any parameter to handle this case ? Best wishes !

[taku910]

Thank you for using sentencepiece.

spm_train --split_by_whitespace=false allows you to extract pieces crossing whitespaces. So, "Ha Noi" may be extracted if it appears frequently in the corpus.

However, my experience says that the whitespace constraint is reasonably useful to extract meaningful pieces.

[minhtringo141]

It works for me, thanks you so much ! Best wishes !