perf(rag): cache chunk embeddings by content hash, skip redundant re-embeds

[JSONbored]

Part of #4325.

Context

src/review/rag-index.ts's cron-triggered indexRepo (the full re-index job, :266-317) unconditionally re-chunks and re-embeds EVERY indexable file in every registered repo on every cycle — unlike reindexChangedPaths (:336), which is correctly scoped to only the paths that actually changed. Since bge-m3 embeddings are deterministic (same input text → same vector), this means the GPU spends real compute re-embedding files whose content is byte-identical to the last cycle, which is typically the vast majority of files in any repo between cron ticks.

Requirements

• Store a per-chunk content hash (e.g. alongside the existing chunk-text row, or as a new column) computed the same way for every indexing path.
• Before calling embedTexts (src/review/rag.ts:294) for a chunk, check whether its content hash already has a stored, valid embedding — skip the embed call and reuse the existing vector when it matches.
• src/selfhost/redis-cache.ts already has claim/release and dedup primitives for exactly this shape (webhook-delivery dedup) — reuse that pattern rather than inventing a new caching mechanism, OR do the hash check directly against the existing Postgres/D1 chunk-storage table if that's simpler and avoids a Redis round-trip per chunk. Pick whichever fits the existing RagInfra/upsertChunks shape with the least new surface area — this issue should record the choice and why.
• Must not change retrieval behavior or output — this is purely an internal optimization; a repo's retrieved context before/after this change must be identical for identical input content.
• Measure and record: embedding-call volume before/after on a real repo's full re-index cycle (should show a large drop for a repo with mostly-unchanged content between cycles).

Deliverables

• Content-hash-based skip implemented in the full re-index path (rag-index.ts's indexRepo)
• Unit tests: a chunk with a matching hash skips embedTexts entirely; a chunk with a changed/new hash still embeds; a corrupted/missing cached vector falls back to re-embedding (fail-safe, never silently skips a chunk that has no real vector)
• Before/after embedding-call-count measurement on a real repo, recorded in the PR description

Expected outcome

A measurable reduction (targeting >80-90% on typical repos) in redundant GPU embedding calls during scheduled full re-index cycles, with zero change to retrieval quality or output.

Effort

M

[linear-code]

JSO-3313

[JSONbored]

Already resolved — same-day as filing. indexRepo's unconditional re-embed was fixed by #4395, with a follow-up correctness fix in #4405 (a changed file was starved by the cap-check ordering).

The shipped approach didn't match either option proposed in this issue's Requirements: instead of a Redis-based dedup or a new sha256 content-hash column, it reuses git's own blob SHA (already free on the tree API response) as the content-hash equivalent, stored on a new repo_chunks.blob_sha column (migration 0128_repo_chunks_blob_sha.sql) and compared via getStoredChunkMeta (src/review/rag.ts:307-319) before the fetch/chunk/embed work in indexRepo (src/review/rag-index.ts:281-354, skip logic at :317-322). This is less new surface area than either proposed option — no new hashing utility, no Redis round-trip per chunk.

Test coverage: test/unit/rag-index.test.ts's describe("indexRepo: embedding cache (#4365)…") block covers matching-hash skip, changed-hash re-embed, and the missing-SHA fail-safe reprocess path — satisfying this issue's test deliverables.

One real nuance for the record: the fix is file-granularity, not per-chunk as this issue's Requirements literally specified — a single changed line in a large file still re-embeds every chunk in that file. Given the >80-90% target was aimed at the "vast majority of files unchanged between cron ticks" case (already fully solved), and there's no current evidence the remaining per-chunk-within-a-changed-file waste is materially costly, I'm not opening a speculative follow-up for it — if a specific repo's re-index cost shows this mattering, that's a narrower, evidence-backed issue at that point.

Closing as resolved.