Boundary-value is not tokenized properly

[saito400]

If tests/text_large.txt is given, I think 「その他 名詞,代名詞,一般,,,*,その他,ソノタ,ソノタ」 should be returned, but following tokens are returned.

そ 名詞,特殊,助動詞語幹,,,,そ,ソ,ソ
の 助詞,連体化,,,,,の,ノ,ノ
他 名詞,非自立,副詞可能,,,,他,ホカ,ホカ

[mocobeta]

😅

[mocobeta]

This happens only when the input text is larger than inner buffer size and the text has no punctuation (like this example.)
https://github.com/mocobeta/janome/blob/master/tests/text_large.txt
For now, I do not come up with good idea for this issue. Anyone has smart solutions or suggestions?

[saito400]

I wanted to fix this , but for now I have no idea...