Summary
FormatFtsToken() (DbSearchReader.cs:49-61) applies the FTS5 prefix-match * suffix only to tokens that contain a CJK code point. Latin-script tokens with diacritics (café, naïve, Ångström) and emoji-bearing tokens never receive prefix promotion, even when a user clearly intends a prefix search. The same asymmetry interacts badly with Unicode normalization: a café indexed as NFC (one code point) does not match a query typed as NFD (cafe + combining acute), and there is no user-visible hint that --exact or explicit normalization is required.
Where
src/CodeIndex/Database/DbSearchReader.cs:49-61 (FormatFtsToken)
Suggested approach
(1) Decouple prefix-promotion from the CJK heuristic: promote any token whose final code point is not a token-boundary (per unicode61 defaults) — covers Latin-diacritics and emoji uniformly. (2) Apply NFC normalization to both indexed content and query tokens at the FTS layer (Unicode NFC is the conventional storage form). (3) When the query token contains a diacritic-prone codepoint and the database was indexed without folding, emit a one-time stderr hint suggesting the --lang/--exact mode that will match. (4) Add a regression test that searches café* against a corpus containing café_au_lait and asserts a match. (5) Cross-link with #198 (CJK tokenization) — same tokenizer layer.
Summary
FormatFtsToken()(DbSearchReader.cs:49-61) applies the FTS5 prefix-match*suffix only to tokens that contain a CJK code point. Latin-script tokens with diacritics (café,naïve,Ångström) and emoji-bearing tokens never receive prefix promotion, even when a user clearly intends a prefix search. The same asymmetry interacts badly with Unicode normalization: acaféindexed as NFC (one code point) does not match a query typed as NFD (cafe+ combining acute), and there is no user-visible hint that--exactor explicit normalization is required.Where
src/CodeIndex/Database/DbSearchReader.cs:49-61(FormatFtsToken)Suggested approach
(1) Decouple prefix-promotion from the CJK heuristic: promote any token whose final code point is not a token-boundary (per
unicode61defaults) — covers Latin-diacritics and emoji uniformly. (2) Apply NFC normalization to both indexed content and query tokens at the FTS layer (Unicode NFC is the conventional storage form). (3) When the query token contains a diacritic-prone codepoint and the database was indexed without folding, emit a one-time stderr hint suggesting the--lang/--exactmode that will match. (4) Add a regression test that searchescafé*against a corpus containingcafé_au_laitand asserts a match. (5) Cross-link with #198 (CJK tokenization) — same tokenizer layer.