Skip to content

symbols --exact: fold non-ASCII casing via a stored Unicode-normalized name key #86

Description

@Widthdom

Context

Follow-up from the codex adversarial review of #81. `symbols --exact` (and MCP `symbols` `exact: true`) currently compares names with `s.name = @q COLLATE NOCASE`, backed by the new `idx_symbols_name_nocase` index. SQLite's built-in `NOCASE` collation is ASCII-only:

```
sqlite> SELECT 'Ä' = 'ä' COLLATE NOCASE, 'SearchSymbols' = 'searchsymbols' COLLATE NOCASE;
0|1
```

For projects with non-ASCII identifiers (Swift, Rust, Java, Kotlin, Scala, Go 1.18+ with Unicode identifiers, etc. — any \\w-tolerant symbol extraction already indexes them), --exact Ä will silently miss an indexed symbol named ä. This is a silent false-negative specifically on the path the caller opted into for precision, which is worse than the LIKE substring default.

The current behavior is documented (help text, MCP tool description, README EN/JP, CHANGELOG) as ASCII NOCASE so users are not misled, but the underlying limitation is worth fixing.

Proposal

Replace the COLLATE NOCASE exact-match strategy with a Unicode-aware indexed folded-name lookup:

  1. Add a stored name_folded column to symbols (additive, nullable for legacy rows).
  2. Populate it at write time via a stable invariant fold (e.g. .NET string.ToLowerInvariant() plus string.Normalize(NormalizationForm.FormKC)).
  3. Add CREATE INDEX IF NOT EXISTS idx_symbols_name_folded ON symbols(name_folded).
  4. In exact-match mode, fold the query in .NET with the same function and compare s.name_folded = @qFolded.
  5. Keep the read-path graceful for legacy DBs: if name_folded is absent or NULL, fall back to the current COLLATE NOCASE path so older indexes (or partial reindexes) do not break.
  6. Add regression tests with non-ASCII identifier/casing pairs (Ä / ä, İ / i edge case, combining-character forms) and a CLI/MCP smoke test.

Why this matters

  • Fulfills the --exact contract for all indexed symbol names, not just ASCII.
  • Keeps the existing performance win (new index still supports O(log n) exact lookup per name).
  • Aligns MCP and CLI behavior with AI client expectations when working on multilingual or non-English codebases.

Scope

  • Additive column + index (non-breaking).
  • Opportunistic migration on open (mirror the existing TryMigrateForRead / EnsureColumn pattern).
  • Reindex required to populate name_folded on existing DBs, but the fallback keeps queries working meanwhile.

Out of scope

Reference

  • Codex adversarial review finding, 2026-04-13: "`symbols --exact` silently fails for non-ASCII identifier casing (src/CodeIndex/Database/DbSymbolReader.cs:120-129)".

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions