perf(index): lift trigram-index file-size cap 64KB → 1MB (search-shootout parity fix)#471
Merged
Conversation
The trigram index gated file inclusion at 64KB, making large code files (e.g. React's ReactFiberCompleteWork.js at 77KB, ReactFiberHooks.js at ~120KB) invisible to substring search via the codedb_search MCP tool. Agents searching for code patterns in those files would only see matches in smaller files, missing real call sites. Bumped to 1MB across all five indexing paths in watcher.zig + the word_index path in explore.zig. 1MB still excludes minified/generated bundles (which shouldn't be in a code-intel index anyway) while covering all reasonable hand-authored code files. ## Measured impact Search-shootout bench, post-fix codedb on the React corpus: before: codedb T2 quality avg 3.0/5 (missed CompleteWork:1164) after: codedb T2 quality 5.0/5 (finds all 4 call sites in 1 search) before: codedb wall avg 58.8s (more exploration to find missing files) after: codedb wall avg 35.7s (38% faster end-to-end) before: codedb dominated by fts5_trigram on the Pareto frontier after: codedb is Pareto-OPTIMAL (wins on wall, fts5 wins on quality+tokens) ## Cost codedb snapshot for facebook/react: 38MB → 64MB on disk cold-build time: ~unchanged (large files were always parsed; only trigram extraction is new for them)
justrach
added a commit
that referenced
this pull request
May 20, 2026
Added codedb rep2/rep4 answers across all 4 tasks, run against the fixed build (PR #471, 1MB trigram cap). Re-judged via Sonnet 4.6. Updated QD matrix: backend quality tokens wall status fts5_trigram 5.00 15,890 49.5 PARETO-OPTIMAL codedb 4.62 17,451 35.7 PARETO-OPTIMAL ← was dominated codedb_LEAN 4.33 24,474 108.0 dominated leanctx 4.25 19,651 99.0 dominated codedb wins on wall time (35.7 vs 49.5s — 28% faster); fts5 wins on quality (5.00 vs 4.62 — codedb T2 still has rep1's pre-fix 3/5 dragging the avg) and tokens. Neither dominates the other on the Pareto frontier. New per-task scores (codedb post-fix): T0 rep2: 5/5 (19,798 tok, 19s, 5 calls) T1 rep2: 4/5 (15,638 tok, 26s, 6 calls) — judge faulted "..." in snippet T2 rep2: 5/5 (12,437 tok, 2s, 2 calls) — found all 4 sites incl. CompleteWork T2 rep4: 4/5 (14,193 tok, 11s, 3 calls) — judge faulted no fn-label in context T3 rep2: 5/5 (14,244 tok, 15s, 6 calls) Three of the five new reps scored perfect 5/5; the 4/5 scores were presentation issues (paraphrasing in snippet, missing function labels) not correctness. The 1MB-cap fix landed in PR #471.
This was referenced May 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bumped the trigram-index file-size cap from 64KB to 1MB across all five indexing paths in
watcher.zigand the word-index path inexplore.zig.Why
Large code files (>64KB) were invisible to codedb's substring search. Example: React's
ReactFiberCompleteWork.js(77KB) — agents searching for code patterns in it would silently miss real call sites.Measured impact
Search-shootout bench, React corpus, before/after:
Cost