Content-hash dedup to avoid wasting embed quota on retries?

[okureanthonytonny-commits]

Hello, i hit a case where a failed embed job re-embedded everything from scratch, including chunks that already succeeded before it failed. vectorize() doesn't check content hashes before re-embedding.
Would a SHA-256 dedup check be welcome, or is there a reason it's not there already?

[lfnovo]

@okureanthonytonny-commits yes, the waste is real, and there's no deliberate reason it's missing. I checked embed_source (commands/embedding_commands.py): it deletes the source's existing embeddings, generates embeddings for all chunks in batches, and inserts them in one go at the end. So when a batch fails halfway, nothing has been written yet, and the retry pays again for every batch that had already succeeded.

Two ways to fix it, and I'd like your take before picking:

1. Persist per batch and resume. Insert each batch as soon as it's embedded, and on retry embed only the chunks that are still missing. This needs no new storage and fixes the retry case directly.
2. Content-hash cache (your proposal). Store a hash per chunk and reuse an existing embedding when the hash matches. It also helps when a source is re-processed with mostly unchanged text, but the key must include the embedding model: switching models has to re-embed everything (that's what "rebuild embeddings" is for), and a hash on text alone would silently reuse vectors from the old model.

My lean is (1) first, since it's smaller and covers the case you hit; (2) can come later if re-processing unchanged sources turns out to be common. Do you want to implement it? If so, tell me which approach, and I'll open the issue with that scope and assign it to you.

Status: exploring. Problem verified; waiting on approach and builder.

[okureanthonytonny-commits]

Thanks for confirming. I'd like to take option 1 (persist each batch as it's embedded, and on retry, embed only the missing chunks). It fixes my retry case, and the hash cache can be a follow-up.

One limitation: resuming is only safe if the embedding model hasn't changed since the partial run started.. because source_embedding doesn't record which model produced a vector, so there's nothing to check that against today. For the first version I'd treat that as a documented limitation (changing the embedding model means a full re-embed of that source), unless you'd rather I add a model field to source_embedding as part of this.

Happy for you to open the issue scoped to option 1 and assign it to me.