The CangjieTokenStream index is wrong

[WincerChan]

The problem is that when I want to create a snippet of tantivy query result, an error(byte index xx is out of bounds of xxxx) will be raised. Looks like an out-of-bounds access to the string(byte array).

This is my test code: https://gist.github.com/WincerChan/09a1777944914164f6a22b62c95ec564

cargo dependencies:

1. tantivy = "0.14.0"
2. cang-jie = "0.11.0"
3. jieba-rs = { version = "0.6.2", default-features = false }

After some works, I find the problem is token stream: https://github.com/DCjanus/cang-jie/blob/master/src/stream.rs#L30

The variable offset_from does not start from 0, but from current_word.len(), So there will be an out-of-bounds error at the last token.

If necessary, I can make a PR to fix it.

[DCjanus]

Thanks for your report, I'd like to fix this in a while, PR is welcome.

[WincerChan]

@DCjanus You are welcome. PR: #21