Summary
cdidx's case-sensitivity policy for query content is partly documented and partly not. --exact mode using instr() (DbSearchReader.cs:295-296) is documented as case-sensitive. Default FTS5 search is case-insensitive for ASCII (FTS5 unicode61 tokenizer default), but the docs (USER_GUIDE.md:338) don't say what happens for accented Latin characters (é vs É), CJK (mostly case-irrelevant), or Unicode special cases (Turkish dotted/dotless I, German ß ↔ SS). The result: searches behave inconsistently across scripts, and users can't reason about what to expect.
Where
src/CodeIndex/Database/DbSearchReader.cs:295-296 (exact mode)USER_GUIDE.md:338 (case sensitivity statement)
Suggested approach
(1) Document the full case-sensitivity matrix in USER_GUIDE: default mode → unicode61 tokenizer (lowercase fold for ASCII + Latin-1 supplement; unaccented; CJK normalized); --exact → byte-exact instr(); --regex (if it exists) → as the regex engine handles. (2) Surface a --case-sensitive / --case-insensitive flag for the default mode that switches between unicode61 and unicode61 remove_diacritics 0 case_sensitive 1 configurations (FTS5 supports the toggle at index time, but a runtime override using LIKE or instr() would also work). (3) Add fixtures and regression tests for: ASCII case insensitivity, accented Latin, German ß/SS, Turkish I/ı, CJK identity. (4) Cross-link with #1546 (path case sensitivity) — sibling concern at the FS layer. (5) Cross-link with the just-filed #1772 (kind taxonomy) — same family of "what does the tool actually do?" docs.
Summary
cdidx's case-sensitivity policy for query content is partly documented and partly not.
--exactmode usinginstr()(DbSearchReader.cs:295-296) is documented as case-sensitive. Default FTS5 search is case-insensitive for ASCII (FTS5 unicode61 tokenizer default), but the docs (USER_GUIDE.md:338) don't say what happens for accented Latin characters (évsÉ), CJK (mostly case-irrelevant), or Unicode special cases (Turkish dotted/dotless I, German ß ↔ SS). The result: searches behave inconsistently across scripts, and users can't reason about what to expect.Where
src/CodeIndex/Database/DbSearchReader.cs:295-296(exactmode)USER_GUIDE.md:338(case sensitivity statement)Suggested approach
(1) Document the full case-sensitivity matrix in USER_GUIDE: default mode → unicode61 tokenizer (lowercase fold for ASCII + Latin-1 supplement; unaccented; CJK normalized);
--exact→ byte-exactinstr();--regex(if it exists) → as the regex engine handles. (2) Surface a--case-sensitive/--case-insensitiveflag for the default mode that switches between unicode61 andunicode61 remove_diacritics 0 case_sensitive 1configurations (FTS5 supports the toggle at index time, but a runtime override usingLIKEorinstr()would also work). (3) Add fixtures and regression tests for: ASCII case insensitivity, accented Latin, German ß/SS, Turkish I/ı, CJK identity. (4) Cross-link with #1546 (path case sensitivity) — sibling concern at the FS layer. (5) Cross-link with the just-filed #1772 (kind taxonomy) — same family of "what does the tool actually do?" docs.