Unicode character cannot be truncated correctly

[Lw-Cui]

Unicode character such as Chinese character can't work.
Please fix it.

[Hrily]

I would like to fix this issue.

[tcorral]

Go ahead for it.

…

On Tue, 15 Aug 2017 at 18:37, Hrishi Hiraskar ***@***.***> wrote: I would like to fix this issue. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#9 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAw9IYLl6MSQkgKSm8bv8ikNJxBXiWI7ks5sYckxgaJpZM4Frf4M> .

[Hrily]

Okay :)

[Hrily]

Hie

The library works perfectly fine for Unicode characters

But, in case of Chinese, there is no defined word boundary as we have in other languages (the space)
An example I got was, the sentence
这是一个句子
can be split in following ways

• 这 | 是 | 一 | 个 | 句子
• 这是 | 一个 | 句子

I don't know the language, so I don't know what above two means.

So the inference is that, as the number of words can't be count due to non availability of word boundary, the library fails at Chinese characters.

The work around I could propose is to split on basis of number of characters in case of Chinese.
But I'm unsure of the effect it will have on readability of the last part.