Skip to content

v1.108.161 — BM25 tokenizer: Unicode word splitting + CJK character bigrams

Choose a tag to compare

View all tags
@jgravelle jgravelle released this 23 Jul 16:18
v1.108.161
b16090a

The BM25 tokenizer treated every non-ASCII character as a separator ([a-zA-Z0-9]{2,}), so CJK symbol names, summaries, and docstrings produced zero lexical tokens and accented Latin identifiers were mangled (café -> caf).

The tokenizer now splits on Unicode word boundaries and expands CJK runs (Hangul, Hiragana/Katakana, Han) into overlapping character bigrams — applied identically at index and query time, so bigram overlap is the match signal. Mixed-script tokens split cleanly; camelCase/snake_case splitting, stemming, abbreviation expansion, and pure-ASCII tokenization are unchanged.

Suite parity: jdocmunch-mcp v1.114.1 (issue #91 there) and jdatamunch-mcp v1.23.1 shipped the same fix today. No reindex needed — BM25 tokenizes stored index fields at scoring time.

Assets 10
Loading