ClickHouse's text index provides the asciiCJK tokenizer (first added here, it was later renamed). It uses ICU to split Chinese, Japanese and Korean inputs into tokens.
The recent StringZilla 5.0 release added equivalent functionality (maybe even a drop-in replacement) to tokenize inputs, see the docs here. I'm pretty sure that @ashvardanian (main author of StringZilla) went great length to make the tokenization in StringZilla faster than in ICU.
We should test that out. Basically:
- bump the existing StringZilla submodule in ClickHouse to 5.0
- introduce a new tokenizer
asciiCJK_v2 based on asciiCJK but using Stringzilla's UTF8 segmentation (in case of success, the latter will be deprecated)
- do benchmarking ... lots of benchmarking to prove the new tokenizer has a tangible performance benefit
ClickHouse's text index provides the
asciiCJKtokenizer (first added here, it was later renamed). It uses ICU to split Chinese, Japanese and Korean inputs into tokens.The recent StringZilla 5.0 release added equivalent functionality (maybe even a drop-in replacement) to tokenize inputs, see the docs here. I'm pretty sure that @ashvardanian (main author of StringZilla) went great length to make the tokenization in StringZilla faster than in ICU.
We should test that out. Basically:
asciiCJK_v2based onasciiCJKbut using Stringzilla's UTF8 segmentation (in case of success, the latter will be deprecated)