fix(rag-worker): clear stale doc embeddings before upsert to prevent orphaned vectors

[lakshayyy10]

Problem

Fixes #603

The three ingestion pipelines (file / text / URL) in rag-worker.ts generate Qdrant point IDs with crypto.randomUUID(), upsert the points, and then update the doc record in MongoDB. These are separate network calls with no cleanup between attempts.

Because docs with status pending or error are re-polled, any retry of a failed or partially-completed run (e.g. Qdrant upsert succeeded but the Mongo updateByVersion failed) re-embeds the doc and upserts a full duplicate set of chunks under fresh random IDs. The vectors from the previous attempt are never removed, so duplicates accumulate on every retry — bloating the collection and polluting RAG search results with duplicate/stale chunks.

Fix

• Add a deleteDocEmbeddings(projectId, sourceId, docId) helper using the same filtered-delete pattern the worker already uses in runDeletionPipeline.
• Call it immediately before the Qdrant upsert in all three pipelines, making doc processing idempotent — a retry now replaces the previous attempt's points instead of stacking on top of them.
• Refactor runDeletionPipeline to reuse the helper (removes the duplicated inline filter).

Why delete-before-upsert rather than deterministic point IDs

Deterministic IDs (e.g. hashing docId + chunk index) would make retries overwrite in place, but if a doc is later re-processed and yields fewer chunks, the leftover higher-index points would still linger. The scoped filtered delete handles every case (retry, re-process, content change) and reuses an existing pattern in the codebase.

Testing notes

• Verified the filter shape matches the existing deletion paths (projectId + sourceId + docId must-clauses).
• esbuild parse check passes on the modified file.
• Happy to add a worker-level test if you can point me at the preferred harness for this script.

[lakshayyy10]

Ready for review whenever you get a chance — happy to adjust the approach or add a worker-level test if you can point me at the preferred harness for this script. cc @ramnique