Context
Raised independently by the local code review on #92 and by Greptile on #92. Follows #90.
#92 reads local-path file sizes lazily at search time. That fixes the O(N) re-read on every cached load, but it leaves two gaps in file_chars telemetry:
- The size describes the file as it is now, not as it was when the returned chunk was indexed. During the MCP freshness cooldown, or with an explicitly loaded index whose source changed,
saved_chars = file_chars - snippet_chars drifts. A file grown past MAX_FILE_BYTES since indexing contributes 0.
- A git index reloaded from cache (
source_id is a URL) and a moved local source both report file_chars: 0 forever, because there is nothing to read lazily.
Proposal
Persist the per-file UTF-16 size map in the on-disk index at build time, alongside chunks.json / manifest.json, behind an INDEX_SCHEMA_VERSION bump so stale caches rebuild.
save() writes the map captured at index time (the indexer already reads every file, so this is free).
load_from_disk restores it; no search-time I/O at all.
- Collapses the
FileSizes empty / captured / lazy split and the Mutex memo into a single captured map.
- Sizes then always match the indexed content, and git-from-cache and moved-source both work.
Notes
Context
Raised independently by the local code review on #92 and by Greptile on #92. Follows #90.
#92 reads local-path file sizes lazily at search time. That fixes the O(N) re-read on every cached load, but it leaves two gaps in
file_charstelemetry:saved_chars = file_chars - snippet_charsdrifts. A file grown pastMAX_FILE_BYTESsince indexing contributes 0.source_idis a URL) and a moved local source both reportfile_chars: 0forever, because there is nothing to read lazily.Proposal
Persist the per-file UTF-16 size map in the on-disk index at build time, alongside
chunks.json/manifest.json, behind anINDEX_SCHEMA_VERSIONbump so stale caches rebuild.save()writes the map captured at index time (the indexer already reads every file, so this is free).load_from_diskrestores it; no search-time I/O at all.FileSizesempty / captured / lazy split and theMutexmemo into a single captured map.Notes
.please/docs/references/semble.mdand the relevant ADR.