Tokenize words, rather than wordparts

[zachmayer]

First of all, thanks for sharing such a useful too! I really like this library.

Second, I'm working on a non-translation task where I think I want to be workings with words, rather than word parts. Are there any settings I can use in sentencepiece to tend to favor longer word units?

[justinchiu]

I'm working on this in a fork -- is there any reason for the 64 character limit in the bpe trainer? Is it okay to increase that?

[taku910]

There are three parameters to control the piece length and shape.

--max_sentencepiece_length (default 16)
--split_by_unicode_script (default true)
--split_by_whitespace (default true)

When the sentence is tokenized with spaces, --split_by_whitespace=false may extract pieces crossing word boundaries.

In addition, I've increased the maximum value of --max_sentencepiece_length from 64 to 512 so we can extract longer pieces.

Thank you.