From 9e7360ab4ea7cb183016b5ece622ac9bd489c501 Mon Sep 17 00:00:00 2001 From: Widthdom Date: Sat, 18 Apr 2026 05:57:36 +0900 Subject: [PATCH 1/9] Fix #198: auto-prefix CJK tokens in literal-safe FTS path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit FTS5's default unicode61 tokenizer treats a run of adjacent CJK / Japanese / Chinese / Korean codepoints as a single token, so `cdidx search 計算` quoted the token as `"計算"` and returned 0 matches against any chunk containing `計算する`. The literal-safe query path now routes each token through a new `FormatFtsToken` helper that detects any codepoint above 0x7F and emits an FTS5 prefix phrase (`"計算"*`) instead of a bare phrase, so CJK substring queries hit without requiring `--fts` / `rawQuery`. Pure-ASCII tokens keep their existing exact-phrase semantics unchanged, so the fix is additive and non-breaking for Latin-script search. Added DbReaderTests coverage for CJK substring matches through both `Search` and `CountSearchResults`, plus an exact-full-token regression so the prefix fallback cannot silently widen a literal search. Updated the MCP `search` tool description and CLAUDE.md design-decisions section to document the CJK prefix behavior and the known emoji limitation (emoji substring search requires a schema-level tokenizer change — tracked as a follow-up to #198). Closes #198 --- CHANGELOG.md | 2 ++ CLAUDE.md | 4 +-- src/CodeIndex/Database/DbSearchReader.cs | 29 +++++++++++++++- src/CodeIndex/Mcp/McpToolDefinitions.cs | 2 +- tests/CodeIndex.Tests/DbReaderTests.cs | 42 ++++++++++++++++++++++++ 5 files changed, 75 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index bcf1b87f37..6d11bc85ea 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### [Unreleased] #### Fixed +- **CJK queries no longer return 0 results against longer FTS tokens (#198)** — `fts_chunks` is created with FTS5's default `unicode61` tokenizer, which has no rule for splitting runs of adjacent CJK / Japanese / Chinese / Korean codepoints, so a multi-character CJK word like `計算する` is stored as a single FTS token. The literal-safe search path used to wrap every whitespace-separated query token in `"..."` phrase quotes without any prefix designator, so `cdidx search 計算` became `MATCH '"計算"'` and returned 0 matches against any chunk containing `計算する`, even though the text was clearly present — including against cdidx's own Japanese documentation (CLAUDE.md / README / CHANGELOG Japanese sections). `SanitizeFtsQuery` now routes each token through a new `FormatFtsToken` helper that detects any codepoint above `0x7F` (CJK, full-width, Cyrillic, Greek, etc.) and emits the token as an FTS5 prefix phrase (`"計算"*`) instead of a bare phrase, so `search 計算` now finds `def 計算する(値):` while pure-ASCII queries keep their existing exact-phrase semantics unchanged. The same path is shared by `search`, the `--count` fast path (`CountSearchResults`), and the MCP `search` tool, so CLI and MCP consumers get the fix together. Added `DbReaderTests` coverage for CJK substring matches via both `Search` and `CountSearchResults`, and a "still respects exact full token" regression test so the prefix fallback cannot silently widen a full-token literal search into a prefix search. Known follow-up (out of scope for this fix): emoji substring search (`search 🎉`) is still 0-result because the default `unicode61` categories `L* N* Co` exclude `S*` (symbols) entirely, so emoji are dropped by the tokenizer before any query ever runs. That requires a schema-level tokenizer change (issue #198 Option A/B) and is tracked separately. Affected: `src/CodeIndex/Database/DbSearchReader.cs`, `tests/CodeIndex.Tests/DbReaderTests.cs`. Closes #198. - **Wrapped C# class / struct / interface / enum headers now preserve their base list and `where` clauses in `symbols.signature` (#382)** — `SymbolExtractor` previously sliced `SymbolRecord.Signature` for C# type declarations from the single physical line the declaration regex matched, so headers whose base list or generic constraints wrapped onto following lines — e.g. `public sealed class Foo\n : BaseFoo, IBar, IBaz\n where T : class, new()\n{` — truncated to `public sealed class Foo` and silently lost every `:` base, interface, and `where T : ...` clause. Downstream consumers that read the signature to recover the base type (the planned `ParseCSharpBaseType` path in #257, future type-position reference extraction for #256, any lint or `impact` / `analyze_symbol` flow that wants to answer "what does this class extend?" from the index alone) degraded to "no base type" for every wrapped-header C# class on every large library that uses formatter-wrapped headers (ASP.NET Core, Roslyn, EF Core, xUnit, `Microsoft.Extensions.*`, etc.). The C# signature path now extends the existing multi-line-signature mechanism (already used for properties) to type declarations: a new `TryFindCSharpTypeHeaderExtent` helper scans forward from the header start using `LexCSharpLine` so `{` / `;` inside string literals, comments, and verbatim / raw strings are ignored, tracks paren depth (C# 12 primary constructor parameter lists) and bracket depth (attributes on type or generic parameters) so a nested `{` / `;` can never short-circuit the scan, terminates at the first body-opening `{` or primary-constructor / forward-declaration `;` at zero depth, and caps lookahead at 64 lines to stay safe on unterminated input. A dedicated `BuildCSharpTypeHeaderSignature` assembles the header slice by concatenating the relevant `[startColumn, endColumn)` span from each physical line with a literal `'\n'` between lines, then runs the combined text through a single-pass `SanitizeCSharpTypeHeaderSlice` that strips `//` line comments (terminated at the next `'\n'`) and `/* ... */` block comments, collapses runs of Code-mode whitespace (including the newlines between lines) into a single space, and preserves String / Verbatim / Raw / Char literal contents verbatim — including multi-line raw / verbatim strings, which keep every internal line break and leading indentation instead of being `Trim()`ed and space-joined. The sanitizer also tracks C# interpolation holes with an explicit frame stack: `$"..."`, `$@"..."`, `@$"..."`, `$"""..."""`, and `$$"""{{expr}}"""` each push a String / Verbatim / Raw frame marked `Interpolated`, and a hole opens only when the `{` run is at least as long as the number of `$` prefixes (with `{{` and `}}` still honored as literal-brace escapes in non-raw interpolated strings). Inside a hole, the sanitizer re-enters Code mode and matching nested braces before returning to the outer literal, so `$"{@"a b"} c"` now keeps the inner verbatim literal's double-space intact and the outer literal content ` c` is preserved instead of the previous collapse to `a b` + ` c`. The helper still produces signatures like `public sealed class Foo : BaseFoo, IBar, IBaz where T : class, new()` for plain wrapped headers, and literal-bearing declarations like `record Foo(string S = "a b")`, `@"C:\tmp\ spaces"`, `"""a b"""`, multi-line raw-string primary-constructor defaults, and interpolation holes containing nested verbatim strings all keep every internal space and newline exactly as written. Same-line headers, single-line bodies, and the existing multi-line property path are unchanged. Added `SymbolExtractor` regression coverage for wrapped class + base list + `where` clauses, wrapped interface + multi-base + `where`, wrapped record primary constructor + wrapped base list + `where`, wrapped struct + `IEquatable>` + `where T : IComparable`, wrapped enum + underlying type `: byte`, a same-line-class pin, dedicated wrapped-header + `//` line comment and wrapped-header + `/* ... */` block comment pins so comment-stripping cannot regress, wrapped primary-constructor-header pins for regular string, verbatim string, and raw string defaults so the lexer-aware collapse cannot silently rewrite literal whitespace, a multi-line raw-string default pin so newlines and per-line indentation inside multi-line raw strings cannot be destroyed by cross-line collapsing, and a nested-verbatim-inside-interpolation-hole pin so interpolation holes inside wrapped base-list arguments cannot exit the outer string too early and collapse inner literals. `BuildCSharpTypeHeaderSignature` also trims a trailing `'\r'` from every sliced line before appending the inter-line `'\n'`, so CRLF-terminated sources (Windows CI with `autocrlf=true`, files saved from Visual Studio, etc.) produce the same signature as LF-terminated sources instead of carrying OS-dependent `"\r\n"` separators between lines; a dedicated CRLF-content regression test pins that normalization. Affected: `src/CodeIndex/Indexer/SymbolExtractor.cs`, `tests/CodeIndex.Tests/SymbolExtractorTests.cs`. Closes #382. - **Regression lock for out-of-range numeric query options (#161)** — `QueryCommandRunner` already rejects invalid `--limit` / `--depth` / `--before` / `--after` / `--start` / `--end` / `--snippet-lines` values with exit code 1, but the existing entrypoint coverage only exercised the non-parseable `"nope"` path. The original bug was specifically about valid integers (`0`, `-5`, `-1`) that parse successfully but fail the positive / non-negative constraint — those used to silently fall back to defaults and emit real results on stdout with exit 0, so callers that branch on exit code alone treated the default-backed output as authoritative. Added a dedicated regression theory `QueryEntrypoints_OutOfRangeNumericOptionsFailClosed_Issue161` that drives `RunSymbols`, `RunSearch`, `RunImpact`, and `RunExcerpt` with the exact repro values from the issue (`--limit 0` / `--limit -5`, `--snippet-lines 0`, `--depth -1`, `--start 0` / `--start -5`, `--end 0`, `--before -1`, `--after -1`) and pins three invariants at once: exit code is `UsageError`, stderr carries the expected `--