Fix #198: auto-prefix CJK tokens in literal-safe FTS path - #438
Merged
Conversation
FTS5's default unicode61 tokenizer treats a run of adjacent CJK / Japanese / Chinese / Korean codepoints as a single token, so `cdidx search 計算` quoted the token as `"計算"` and returned 0 matches against any chunk containing `計算する`. The literal-safe query path now routes each token through a new `FormatFtsToken` helper that detects any codepoint above 0x7F and emits an FTS5 prefix phrase (`"計算"*`) instead of a bare phrase, so CJK substring queries hit without requiring `--fts` / `rawQuery`. Pure-ASCII tokens keep their existing exact-phrase semantics unchanged, so the fix is additive and non-breaking for Latin-script search. Added DbReaderTests coverage for CJK substring matches through both `Search` and `CountSearchResults`, plus an exact-full-token regression so the prefix fallback cannot silently widen a literal search. Updated the MCP `search` tool description and CLAUDE.md design-decisions section to document the CJK prefix behavior and the known emoji limitation (emoji substring search requires a schema-level tokenizer change — tracked as a follow-up to #198). Closes #198
The first pass of the #198 fix promoted any token containing a non-ASCII codepoint (`foreach c in token: if c > 0x7F`) to an FTS5 prefix phrase. That was too broad and was caught by codex review as an observable false positive: `cdidx search 'foo🎉'` not only matched `src/emoji.py` containing `foo🎉` but also leaked into `src/plain.py` which only contained `foo` / `foobar`. The cause is that FTS5's default unicode61 tokenizer drops emoji (symbol category) codepoints entirely during both indexing and query tokenization, so the sanitized query `"foo🎉"*` is parsed by FTS5 as `"foo"*` — a prefix search on `foo` that sweeps in every ASCII neighbor. The same class of over-widening applied to Latin-diacritic tokens (`naïve`, `café`), Greek, and Cyrillic, which unicode61 tokenizes normally. Promoting them to prefix would silently widen an exact literal search into a prefix search without the user asking for it. `FormatFtsToken` now routes tokens through `ContainsCjk`, which uses `Rune.EnumerateRunes()` to check explicit CJK Unicode blocks (Han / CJK Unified Ideographs + Extensions A–G + Compatibility / Radicals, Hiragana / Katakana + phonetic extensions, Hangul Syllables + Jamo + Compatibility Jamo, halfwidth Katakana) and excludes symbol categories (`OtherSymbol` / `MathSymbol` / `CurrencySymbol` / `ModifierSymbol`) up front so an emoji that happens to fall inside a CJK-adjacent block cannot trigger the fallback. The Halfwidth / Fullwidth Forms block is narrowed to the halfwidth-Katakana sub-range so fullwidth Latin / digits (`ＡＢＣ`, `０１２`) also keep exact-phrase semantics. Also added three negative regression tests: - `Search_CjkPrefixDoesNotMatchUnrelatedCjkTokens` — pins that `search 計算` hits the `計算する` file but not an unrelated `検索する` file, so the prefix fallback can only widen *to* tokens that start with the query codepoints. - `Search_EmojiMixedTokenDoesNotOverMatchAsciiPrefixNeighbors` — pins the exact bug codex caught: `foo🎉` must not drag in `foobar`. - `Search_LatinDiacriticTokenDoesNotWidenToPrefixSearch` — pins that `naïve` stays as exact phrase semantics and does not widen to `naïvety`. Updated CLAUDE.md (EN + JA), CHANGELOG.md (EN + JA), and the MCP `search` tool description to say "CJK token" instead of "non-ASCII" and call out the over-widening risk that motivates the narrowing. Refs #198.
Codex round 2 review caught two correctness gaps in the previous
round's tests and docs, plus one dead-code issue:
1. `Search_EmojiMixedTokenDoesNotOverMatchAsciiPrefixNeighbors` only
pinned that `foo🎉` did not widen to `foobar`, but CHANGELOG /
CLAUDE.md / the MCP description all claimed emoji-mixed tokens keep
"exact-phrase semantics". That claim is not achievable at the FTS
layer: unicode61 strips emoji codepoints on BOTH the index side and
the query side, so `foo🎉` is indexed and queried as the plain
token `foo` — meaning `search foo🎉` also matches a file containing
just `def foo():`. Callers who need strict emoji-bearing equality
must route through the `--exact` / `instr` path.
2. `Search_CjkQueryStillRespectsExactFullToken` only asserted the
positive case ("full-token CJK query finds full-token CJK content")
but its name, comment, and the CHANGELOG entry implied it also
pinned "CJK full token does not silently widen into a prefix
search". The implementation cannot honor that — CJK tokens take
the prefix path unconditionally, which is the whole point of the
fix (otherwise `search 計算` would regress back to 0 hits against
`計算する`). The trade-off needed to be written down, not pretended
against.
3. CJK Radicals Supplement (U+2E80..U+2EFF) and Kangxi Radicals
(U+2F00..U+2FDF) ranges in `IsCjkScript` were dead: every
codepoint in those blocks is Unicode category `OtherSymbol`, which
is rejected by the category guard earlier in the function.
Changes:
- `src/CodeIndex/Database/DbSearchReader.cs`: remove dead Radicals
ranges, document why they are excluded (unicode61 drops them).
- `tests/CodeIndex.Tests/DbReaderTests.cs`:
- Rename `Search_CjkQueryStillRespectsExactFullToken` →
`Search_CjkFullTokenQueryStillFindsExactFullToken` with an
explicit comment that this is a "does not regress" pin, not a
"narrows to exact" pin.
- Add `Search_CjkFullTokenQueryAlsoWidensToLongerTokens` to pin
the intentional widening so it cannot be silently reverted into
the #198 zero-hit repro.
- Rename `Search_EmojiMixedTokenDoesNotOverMatchAsciiPrefixNeighbors`
→ `Search_EmojiMixedTokenDoesNotPrefixWidenToAsciiNeighbors` with
a comment that this only guards against PREFIX widening, not
against the deeper FTS-equivalence fallback.
- Add `Search_EmojiMixedTokenFallsBackToAsciiToken_UseExactForStrict`
that pins both halves of the limitation: `foo🎉` and `foo` are
FTS-indistinguishable, and the `--exact` path is the only way to
tell them apart.
- `CHANGELOG.md` (EN + JA), `CLAUDE.md` (EN + JA), and
`src/CodeIndex/Mcp/McpToolDefinitions.cs` now document, rather than
overclaim:
- CJK prefix promotion is unconditional; strict equality requires
`--exact`.
- Emoji-mixed tokens are FTS-equivalent to their plain ASCII
counterpart; strict equality requires `--exact`.
- Radicals blocks are excluded because unicode61 drops them.
Full test suite still passes (1668 tests).
Refs #198.
) - Add CJK Unified Ideographs Extension H (U+31350..U+323AF, Unicode 15.0) and Extension I (U+2EBF0..U+2EE5F, Unicode 15.1) to IsCjkScript. Without Extension H, `search '𱍐'` returned 0 hits against content containing `𱍐abc` — the exact #198 zero-hit shape reproduced on a newer Unicode block. - Rewrite the FormatFtsToken summary comment: the previous wording claimed non-CJK tokens "intentionally keep exact-phrase semantics," which contradicts the rest of the documentation now that we acknowledge unicode61 drops emoji codepoints on both sides, making `foo🎉` and `foo` indistinguishable at the FTS layer. The new comment matches CHANGELOG, CLAUDE.md, and the MCP tool description. - Tighten CountSearchResults_IncludesCjkSubstringMatches from `>= 1` to exact `Equal(1, ...)` with a miss-file fixture so drift that inflates the count is caught too. - Add Search_FindsNonBmpCjkExtensionHSubstringInsideLongerToken using char.ConvertFromUtf32(0x31350) to pin surrogate-pair handling. - CHANGELOG Japanese section: "拡張 A–G" → "拡張 A–I" for parity with the English "Extensions A–I" entry. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…/ halfwidth Hangul (#198) Codex round 4 review found four remaining 0-hit repros even after Extensions H/I were added: `search 々`, `search 〇`, `search ﾱ`, and `search 𚿰` all returned zero against content like `々abc` / `〇abc` / `ﾱﾲﾳabc` / `𚿰abc`. Under unicode61 those codepoints are treated as word characters but were not in the CJK prefix fallback set, so the whole CJK run indexed as a single token and the bare phrase query could never match a longer token. Extend `IsCjkScript` in `src/CodeIndex/Database/DbSearchReader.cs`: - Han-script codepoints outside the CJK Unified Ideographs blocks — ideographic iteration mark 々 (U+3005), closing mark 〆 (U+3006), number zero 〇 (U+3007), Hangzhou numerals (U+3021..U+3029), and Hangzhou 10/20/30 + vertical iteration marks (U+3038..U+303B). - Kana Extended-B (U+1AFF0..U+1AFFF, Unicode 15.0). - Halfwidth range widened from U+FF65..U+FF9F (halfwidth Katakana) to U+FF65..U+FFDC (through halfwidth Hangul). U+FFDD..U+FFDF are unassigned and U+FFE0..U+FFEE are fullwidth currency/arrow symbols that unicode61 drops during tokenization, so stopping at U+FFDC is correct. Tests — `tests/CodeIndex.Tests/DbReaderTests.cs`: - Added four new CJK regressions pinning `search 々`, `search 〇`, `search ﾱ`, and `search 𚿰` against content containing `<codepoint>abc`. - Added a dedicated Extension I regression alongside the existing Extension H test so a future cleanup that drops either range would break its own dedicated test rather than silently regressing. - Fixed the stale "non-ASCII prefix-match fallback" comment on the `Search_FindsCjkSubstringInsideLongerToken` test to say "CJK-only" consistent with the predicate's actual scope. Docs — CHANGELOG (EN + JA), CLAUDE.md (EN + JA literal-safe search bullet), and the MCP `search` tool description now enumerate the CJK set explicitly (Kana + Kana Extended-B, CJK Unified Ideographs + Extensions A–I, Han-script codepoints outside the unified blocks, Hangul, and halfwidth forms through U+FFDC) so AI clients and humans can see at a glance what is / is not auto-upgraded to a prefix phrase, and explain why the halfwidth range stops at U+FFDC. Verified end-to-end: the locally built binary now returns hits for all four round-4 BLOCKING repro shapes (`search 々`, `search 〇`, `search ﾱ`, `search 𚿰`) against files containing `々abc` / `〇abc` / `ﾱﾲﾳabc` / `𚿰abc`. Full test suite: 1674 passed, 2 skipped (performance, as designed). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Codex round 5 review found two more 0-hit repro blocks left unaddressed by round 4's expansion: 1. Vertical kana repeat marks U+3031..U+3035 (〱 〲 〳 〴 〵). Unicode category Lm; unicode61 keeps them as word characters. Adjacent to the U+3038..U+303B range but not covered by it. 2. Bopomofo (U+3100..U+312F) and Bopomofo Extended (U+31A0..U+31BF) — Mandarin zhuyin and minority-dialect zhuyin extensions, category Lo. unicode61 keeps them as word characters, so `search ㄅ` returned 0 against `ㄅabc`. Extend `IsCjkScript` in `src/CodeIndex/Database/DbSearchReader.cs` to include both classes. The Han-script branch now enumerates U+3031..U+3035 alongside the existing individual 々 / 〆 / 〇 / Hangzhou / U+3038..U+303B lines — kept explicit rather than widening to the full U+3000..U+303F block so we don't sweep in ideographic space / brackets / dots that unicode61 already drops. Tests — `tests/CodeIndex.Tests/DbReaderTests.cs`: - Search_FindsVerticalKanaRepeatMarkInsideLongerToken (〱 U+3031) - Search_FindsBopomofoInsideLongerToken (ㄅ U+3105) - Search_FindsBopomofoExtendedInsideLongerToken (ㆠ U+31A0) Each test indexes `<codepoint>abc` and queries `<codepoint>` through the normal FTS path, so removing the corresponding range makes the test fail. Docs — CHANGELOG (EN + JA), CLAUDE.md (EN + JA), MCP `search` tool description now enumerate Kana Supplement / Kana Extended-A / Small Kana Extension (U+1B000..U+1B16F, which the code already covered but the prose did not mention — round 5 review flagged this as a non-blocking doc/code mismatch), Bopomofo + Bopomofo Extended, and vertical kana repeat marks so AI clients can see the full CJK scope at a glance. Verified with the locally built binary: `search 〱`, `search 〵`, `search ㄅ`, and `search ㆠ` all now return hits against files containing the respective `<codepoint>abc` forms. Full test suite: 1677 passed, 2 skipped (performance). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Codex review round 6 identified six more historical East Asian Unicode blocks whose codepoints are Lo / Lm (kept as word characters by the default unicode61 tokenizer) and that were not covered by `IsCjkScript`, reproducing the #198 zero-hit shape on `search <cp>` against `<cp>abc` content: - Yi Syllables U+A000..U+A48F (Nuosu syllabary, BMP, Lo) - Tangut U+17000..U+187FF (Western Xia, non-BMP, Lo) - Tangut Components U+18800..U+18AFF (non-BMP, Lo) - Khitan Small Script U+18B00..U+18CFF (non-BMP, Lo) - Tangut Supplement U+18D00..U+18D8F (non-BMP, Lo) - Nüshu U+1B170..U+1B2FF (women's script, non-BMP, Lo) `IsCjkScript` in `DbSearchReader` now covers those blocks so tokens containing them take the FTS5 prefix-phrase path (`"<cp>"*`) and `search ꀀ` / `search 𗀀` / `search 𘴀` / `search 𛅰` find content containing `<cp>abc`. Yi Radicals (U+A490..U+A4CF) stay excluded — every codepoint there is Unicode category `OtherSymbol` and is dropped by unicode61, so prefix fallback cannot help them anyway. Added four DbReader regression tests pinning Yi Syllables, Tangut (lower bound of the merged U+17000..U+18D8F range), Tangut Supplement (upper bound, doubling as a Khitan / Tangut Supplement pin), and Nüshu. Updated CHANGELOG, CLAUDE.md, and the MCP `search` tool description (EN + JA) to enumerate the new blocks and explain why Yi Radicals is excluded. Verified with the locally built binary against a temp fixture containing `<cp>abc` files for each block: all four queries return hits. All 1681 tests pass (+4 vs round 6). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Round 8 codex review follow-ups: 1. Split the merged 0x17000..0x18D8F range into four Unicode-block envelopes — Tangut (U+17000..U+187FF), Tangut Components (U+18800..U+18AFF), Khitan Small Script (U+18B00..U+18CFF), Tangut Supplement (U+18D00..U+18D8F) — so each block has its own predicate branch and its own regression test. A future narrowing of any single block no longer drags the others. 2. Add the individual iteration / annotation codepoints from the Ideographic Symbols and Punctuation block that unicode61 keeps as word characters — U+16FE0 (Tangut iteration mark, Lm), U+16FE1 (Nüshu iteration mark, Lm), U+16FE3 (Old Chinese iteration mark, Lm), U+16FE4 (Khitan Small Script filler, Mn), and U+16FF0..U+16FF1 (Vietnamese Chu Nom alternate reading marks CA / NHAY, Mc). U+16FE2 (Old Chinese hook mark, Po) is deliberately excluded because unicode61 drops it during tokenization. Without these, historical Han-based text containing iteration marks reproduces the #198 0-hit shape on a different codepoint class. 3. Document that `OtherNotAssigned` (Cn) is intentionally NOT added to the top-level category exclusion. .NET's Unicode tables lag the Unicode Consortium's releases, so codepoints assigned in Unicode 15.1 — for example Extension I (U+2EBF0..U+2EE5F) — still report `Cn` on .NET 8. Excluding `Cn` would silently regress real CJK codepoints just because the runtime has not caught up. Small unassigned trailing holes inside block envelopes (e.g. Tangut U+187F8..U+187FF) are harmless because unicode61 drops unassigned codepoints from indexed content, so no prefix-promoted query can spuriously match them. Added five new regression tests: Tangut Components (𘠀 U+18800), Khitan Small Script (𘬀 U+18B00), Tangut Iteration Mark (𖿠 U+16FE0, Lm), Khitan Small Script Filler (𖿤 U+16FE4, Mn), and Vietnamese Chu Nom Alternate Reading Mark CA (𖿰 U+16FF0, Mc). The three Ideographic Symbols and Punctuation tests span the Lm / Mn / Mc Unicode categories so no single annotation-category regression can silently slip past, and the four Tangut-family tests each exercise their own predicate branch so a future block-merge refactor breaks its own dedicated test first. Full suite: 1686 pass / 0 fail / 2 skip on net8.0. Closes #198.
Round 8 code review flagged that the "unicode61 drops unassigned codepoints from indexed content" rationale was empirically false: searching for reserved codepoints like U+187F8, U+18CD6, U+18D09 returns content containing those codepoints through the locally built binary, proving unicode61 keeps them as word characters. The actual reason the block-envelope predicate works: - .NET 8's Unicode tables lag the Unicode Consortium's releases, so Extension I (U+2EBF0..U+2EE5F, Unicode 15.1) is reported as OtherNotAssigned despite being assigned. - unicode61 ships its own (older, different) Unicode tables and keeps reserved codepoints in CJK-like blocks as word characters. - Therefore allowing Cn through the block envelopes produces correct search behavior. Rewrote rationale comments/docs in DbSearchReader.cs, CHANGELOG.md (EN+JA), CLAUDE.md (EN+JA), and McpToolDefinitions.cs (EN+JA). Predicate behavior is unchanged. Tests: 1686 pass / 0 fail / 2 skip on net8.0. Refs #198 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #198:
cdidx search 計算returned 0 matches against content containing計算するbecause SQLite FTS5's defaultunicode61tokenizer keeps adjacent CJK codepoints as a single token, so the literal-safe phrase"計算"never matched the longer indexed token."計算"*) instead of bare phrases, so they match the longer CJK tokensunicode61produces. Non-CJK tokens (Latin-diacritic, Greek, Cyrillic, emoji-mixed) keep bare-phrase behavior to avoid silently widening matches or leaking through emoji stripping. Strict equality on CJK substrings remains available via--exact/instr.unicode61drops those symbol codepoints at tokenization time regardless. Emoji-only substring search still needs a tokenizer-level schema change (tracked as a FTS5 default tokenizer (unicode61) treats CJK runs as single tokens —search 計算returns 0 matches against text containing計算する, same for emoji and any non-whitespace-separated script #198 follow-up).DbSearchReader.IsCjkScriptsplits non-BMP CJK-adjacent blocks into per-block branches so each block can be independently verified, and regression tests exercise the previously-failing計算case plus each new block-envelope branch.OtherNotAssignedinclusion —.NET 8's Unicode tables lag the Unicode Consortium's releases (Extension I reports asCndespite being assigned in Unicode 15.1), andunicode61ships its own older tables that empirically keep reserved block-tail codepoints as word characters. AllowingCnthrough the block envelopes therefore preserves thesearch <cp>→"<cp>abc"invariant. Truly unreachable codepoints still produce zero matches because FTS5 drops them on its own query-side tokenizer.CHANGELOG.md,CLAUDE.md, and the MCPsearchtool description updated in English and Japanese.Test plan
dotnet build— 0 warnings, 0 errors.dotnet test— 1686 pass / 0 fail / 2 skip on net8.0.search 計算against計算する(the original FTS5 default tokenizer (unicode61) treats CJK runs as single tokens —search 計算returns 0 matches against text containing計算する, same for emoji and any non-whitespace-separated script #198 repro) and each per-block branch (Tangut, Tangut Components, Khitan Small Script, Tangut Iteration Mark, Khitan Small Script Filler, Vietnamese Reading Mark CA, Extension I).5306b73).dotnet ./src/CodeIndex/bin/Debug/net8.0/cdidx.dll) —search 計算returns the expected計算するchunk.Closes #198
Co-Authored-By: Claude Opus 4.7 noreply@anthropic.com