chunksync

Catch un-deduped vector-store writes by reading your ingestion code — no vector store, no network, no run.

Every RAG pipeline eventually re-runs its ingestion job: a cron re-trigger, a redeploy, a retried batch after a flaky embedding call. If the write that pushes chunks into the vector store has no stable, content-derived id, that re-run doesn't refresh the index — it duplicates it. The same passage now exists under two ids, then three, then ten. Retrieval quality degrades quietly: near-duplicate hits crowd out the answer, relevance scores get diluted, and nothing ever throws an error. You just notice the RAG answers getting worse.

chunksync reads the source with Python's AST and flags the write call itself, before it ever touches a real index:

$ chunksync examples/risky_ingest.py

CS  CS001 examples/risky_ingest.py:15:4  Vector-store upsert with no dedup key (re-running this will duplicate embeddings).
        index.upsert(vectors=vectors)
        ↳ Pass id= a content hash (e.g. hashlib.sha256(text.encode()).hexdigest()) so re-ingesting the same content updates the existing vector instead of duplicating it.
CS? CS002 examples/risky_ingest.py:44:8  Vector-store upsert keyed on a fresh uuid4() with no content-hash or existence check nearby.
        collection.upsert(ids=[str(uuid.uuid4())], embeddings=[chunk])
        ↳ A random id changes every run, so a re-ingested chunk gets a new id instead of overwriting its old vector. Derive the id from a content hash, or check existence first.

1 file(s) · 3 blocker(s) · 1 warning(s)

Exit code 1 on a confirmed missing dedup key, so it drops straight into CI before the ingestion job ever ships.

What it flags

• CS001 (blocker) — a vector-store write with no id-like argument at all: index.upsert(vectors=[...]), collection.add(documents=..., embeddings=...) with no ids=, vectorstore.add_documents(docs), Weaviate's .data.insert(...) with no uuid=.
• CS002 (warning) — an id argument that's a bare uuid.uuid4() with no dedup guard anywhere else in the file. It looks like an id, but it's random — re-ingesting the same chunk still mints a new vector instead of overwriting the old one. Promoted to a failure under --strict.

Recognized write shapes: index.upsert(, collection.upsert(, .add_documents(, .add_texts(, vectorstore.add(, Pinecone's upsert(vectors=[...]), Weaviate's .data.insert(, Chroma's collection.add(, and a generic .insert( when the receiver's name is vector-ish (embedding_table.insert(...)).

Why static

There's no way to unit-test "did this duplicate a vector" without an actual vector store, real embeddings, and two runs of the pipeline — expensive to set up and easy to skip under deadline. chunksync catches the shape of the bug — no dedup key on the write — the moment the ingestion code is written, on the pull request, with nothing to configure and nothing to run.

Install

pip install chunksync

Usage

chunksync ingest/                 # scan a directory
chunksync pinecone_job.py         # scan a file
cat pinecone_job.py | chunksync -  # or read stdin
chunksync ingest/ --strict        # treat CS002 warnings as failures too
chunksync ingest/ --json          # machine-readable output

In CI

- run: pipx run chunksync ingest/ --strict

Exit codes: 0 clean · 1 a blocker (CS001), or any finding under --strict · 2 usage error.

Precision & suppression

Static analysis can't run the ingestion job twice and diff the index, so chunksync is deliberately conservative: only a tight set of known vector-store-shaped calls is considered at all (a plain cart.add(item), dict.update(...), or .insert() on an unrelated object is never flagged), and an id argument that isn't an obvious hash or a bare uuid4() is left alone rather than guessed at. When it's still wrong, silence that line:

index.upsert(vectors=vectors)  # chunksync:ignore

Or skip a path entirely with a .chunksyncignore file (one path substring per line, same mechanism as .gitignore):

vendor/
third_party/

Scope

chunksync proves the absence of a dedup key on the write call it can see — it can't prove an id argument built from an opaque helper function or a **kwargs unpack is (or isn't) a hash, and in those ambiguous cases it stays quiet rather than risk a false positive. It also can't verify an id is actually derived from the chunk's content, only that it looks like a hash (hashlib.sha256(...)) or an obviously random uuid4(). That's the trade-off for a zero-config, zero-runtime gate: fewer false positives, at the cost of missing a dedup key disguised behind logic it can't see into.

License

MIT © Jay Tank