FTS5 default tokenizer (`unicode61`) treats CJK runs as single tokens — `search 計算` returns 0 matches against text containing `計算する`, same for emoji and any non-whitespace-separated script

[Widthdom]

Summary

cdidx creates fts_chunks with FTS5's default tokenizer (unicode61), which separates tokens at Unicode-category boundaries (letter ↔ punctuation ↔ symbol). For Latin-script code that's fine — words are whitespace-separated. For Japanese / Chinese / Korean / emoji text, a whole multi-character run is one token because there's no separator between the codepoints.

Concrete consequence: search 計算 returns 0 matches in a file containing def 計算する(値):, because 計算する is a single FTS5 token and 計算 is a different (non-existent) token. The user has to know to reach for --fts "計算*" (FTS5 prefix-match syntax) — and even then, --fts is opt-in raw FTS5 syntax with all its escape pitfalls.

This is a global, silent failure mode for any codebase containing CJK text, including:

• Code with CJK identifiers (legal in Python, JS, C#, Java, etc.).
• CJK comments / docstrings (extremely common in Japanese / Chinese open-source projects).
• CJK content in test fixture strings.
• cdidx's own Japanese documentation (CLAUDE.md Japanese section, README Japanese section, CHANGELOG Japanese section). Searching cdidx's own repo for 日本語 returns 0 matches against any line that has 日本語 only as part of a longer run (e.g. 日本語ドキュメント shows up if you search the exact full token, not the substring).

Repro

mkdir /tmp/cjk && cd /tmp/cjk
cat > emoji.py <<'EOF'
def 計算する(値):
    return 値 * 2

def emoji_func():
    """Test emoji 🎉 in docstring"""
    return "😀"
EOF

/root/.local/bin/cdidx index .

# Substring search of a CJK token: 0 results
/root/.local/bin/cdidx search 計算
# No results found.

# Exact full-token search: works
/root/.local/bin/cdidx search 計算する
# emoji.py:1-9
#   def 計算する(値):
#   ...

# Emoji search: 0 results
/root/.local/bin/cdidx search "🎉"
# No results found.

# Workaround: --fts prefix-match syntax
/root/.local/bin/cdidx search "計算*" --fts
# emoji.py:1-9 ...

For comparison, the symbol-name LIKE path does find substrings, because symbols uses LIKE not FTS5:

/root/.local/bin/cdidx symbols 計算
# function   計算する     emoji.py:1-2

So the divergence is: symbols works for CJK substrings (LIKE), search doesn't (FTS5 token-exact).

Direct verification of the tokenizer cause

import sqlite3
conn = sqlite3.connect('/tmp/cjk/.cdidx/codeindex.db')
print(conn.execute('SELECT sql FROM sqlite_master WHERE name="fts_chunks"').fetchone()[0])
# CREATE VIRTUAL TABLE fts_chunks USING fts5(content, content='chunks', content_rowid='id')
#                                                                                       ^^^^^^^
#                                                                                       no tokenize=
# → defaults to unicode61

# Confirm 計算する is a single token (matches as 計算*, not as 計算):
list(conn.execute("SELECT highlight(fts_chunks, 0, '<<', '>>') FROM fts_chunks WHERE fts_chunks MATCH '計算*'"))
# [('def <<計算する>>(値): ...',)]   ← whole 計算する highlighted as one match

計算する is a single token because unicode61 has no rule that breaks adjacent CJK letters. FTS5 supports prefix matching (X*) but has no native "substring" mode for the unicode61 / simple tokenizers.

Suspected root cause (from reading the source)

src/CodeIndex/Database/DbContext.cs:466-472:

Execute(@"
    CREATE VIRTUAL TABLE IF NOT EXISTS fts_chunks USING fts5(
        content,
        content='chunks',
        content_rowid='id'
    )");

No tokenize= clause means SQLite picks unicode61 (its default). For CJK / emoji that's effectively "no token splitting".

Suggested direction

SQLite ships a trigram tokenizer (added in SQLite 3.34, July 2021, well before the .NET 8 + Microsoft.Data.Sqlite shipping cdidx uses) specifically for "non-whitespace-separated text" — it produces 3-character n-grams, supports substring MATCH queries natively, and is the standard answer for CJK / emoji search in FTS5.

Three reasonable shapes:

Option A — switch to trigram outright

CREATE VIRTUAL TABLE IF NOT EXISTS fts_chunks USING fts5(
    content,
    content='chunks',
    content_rowid='id',
    tokenize='trigram'
)

Trade-off: trigram indexes are 2-3× larger than unicode61 (each character contributes to multiple 3-grams). For a typical Latin-script repo this is a real cost. Substring search works for everything (including ASCII — search def would match default, defer, etc., which may be undesirable).

Option B — add a second virtual table for CJK-friendly fallback

Keep fts_chunks as unicode61 (fast for ASCII). Add a second fts_chunks_trigram with tokenize='trigram'. The search command tries fts_chunks first; if 0 hits AND the query contains non-ASCII / non-Latin codepoints, retry against fts_chunks_trigram. Larger schema, but semantics for ASCII users are unchanged.

Option C — auto-prefix queries containing CJK

In the literal-safe path, detect non-ASCII characters in the query and append * to enable FTS5 prefix matching (search 計算 → effectively search 計算*). Cheapest fix, no schema change. Doesn't fix true substring search (e.g. search 算する still wouldn't match 計算する), but covers the most common case (prefix of a CJK token).

I'd suggest Option C as a tactical fix (single-line change in the query path, no schema migration) and Option B as the proper long-term fix. Option A is too coarse — it changes ASCII semantics in a way users will notice.

Sibling observations

• --fts "計算*" works as a workaround but requires the user to know FTS5 syntax. Worth surfacing in search's "no results found" hint when the query contains non-ASCII characters: e.g. "Hint: query contains non-ASCII characters; FTS5 may need a prefix match. Try --fts \"<query>*\"."
• MCP search has the same limitation (same FTS5 table). AI consumers via Claude Code / Cursor / Windsurf can't search CJK substrings either.
• The MCP tool description currently doesn't mention this limitation, so the AI client has no way to know.

Real-world impact

Any non-Latin-script codebase. cdidx is targeted explicitly at multi-language and includes Japanese documentation, but its own Japanese docs are partially un-searchable from itself. AI consumers indexing real-world Japanese / Chinese / Korean projects today get silently degraded full-text search.

Scope

• src/CodeIndex/Database/DbContext.cs:466-472 — pick a tokenizer strategy (Option A/B), or leave the schema and implement Option C in the search path.
• (Option C) src/CodeIndex/Database/DbSearchReader.cs query-building — auto-append * to non-ASCII tokens in the literal-safe path.
• src/CodeIndex/Cli/QueryCommandRunner.cs — improve "no results" hint for non-ASCII queries.
• src/CodeIndex/Mcp/McpToolDefinitions.cs — document CJK behavior in the search tool description.
• tests/CodeIndex.Tests/DbReaderTests.cs — add CJK / emoji search coverage.
• README + CLAUDE.md design-decisions section — document the chosen tokenizer / fallback semantics.
• CHANGELOG.md — Fixed entry (English + Japanese).

Notes for reviewer

I lack a local .NET SDK in this cloud session, so I can't compile/test the change. The diagnosis was confirmed via Python sqlite3 directly against the produced DB — the tokenizer behavior is reproducible without rebuilding cdidx. CI / a reviewer with an SDK should pick a strategy and validate it against the existing search tests for ASCII regressions.

Related

• --path <pattern> is documented as a "pattern" but actually does literal substring containment — --path "*.py" returns 0 results, only --path ".py" works (no glob support, breaks every conventional CLI expectation) #195 — --path substring vs. glob pattern semantics. Sibling discoverability problem: tools that look like one thing but behave like another.
• search --snippet-lines N has no per-line character cap — a single hit in a minified/transpiled file (one 120 KB line) returns 120 KB of output even with --snippet-lines 1 #196 — search payload sizing. Same code path; both are AI-experience improvements.

Environment

• cdidx: v1.10.0 (install.sh).
• Platform: linux-x64 container.
• Filed from a cloud Claude Code session per CLOUD_BOOTSTRAP_PROMPT.md.

[Widthdom]

Starting work on this issue. Plan: implement Option C as a tactical fix — in the literal-safe search path, auto-append FTS5 prefix * to tokens that contain non-ASCII characters so queries like search 計算 or search 🎉 match CJK/emoji runs without requiring users to drop into raw --fts syntax. Will also add CJK/emoji coverage to DbReaderTests.cs, update the MCP search tool description, and note the behavior in README + CLAUDE.md.