vector-forget

Hard-delete and verify erasure of a data subject from your vector store.

When a GDPR / CCPA "right to be forgotten" request lands, store.delete() is not enough — most vector databases soft-delete: the vectors vanish from search while the raw bytes stay on disk, reconstructible. (See Ghost Vectors, arXiv:2606.18497 — deleted embeddings recovered at high rates from raw index files.) A single-subject delete also never trips the background compaction/vacuum that would actually reclaim them.

vector-forget does the two things a plain delete doesn't:

1. Hard-delete — forces the physical reclaim (e.g. pgvector REINDEX + VACUUM) so the ghost vectors are actually gone, not just hidden.
2. Verify + report — proves it with the backend's own residue signal and hands you a structured ForgetReport.

⚠️ Alpha / v0.1 — pgvector only. Qdrant, Weaviate and framework integrations are deliberately deferred (see Non-goals).

Install

pip install "vector-forget[pgvector]"

Requires Python 3.9+ and Postgres with the vector extension. The physical-residue proof (the wedge) additionally needs pgstattuple and table-owner / MAINTAIN rights — i.e. a self-hosted / owned Postgres. On managed Postgres it still deletes and verifies visibility, but the residue gate degrades honestly to UNVERIFIED-managed (see below).

Quickstart (pgvector)

import psycopg
from vector_forget import stamp, forget, verify
from vector_forget.pgvector import PgVectorStore

# 0) connect to your Postgres (self-hosted, for the full residue proof)
conn = psycopg.connect("postgresql://user:pass@localhost:5432/mydb")
store = PgVectorStore(conn, table="documents", subject_col="subject_id")

# 1) at INGEST — tag each row so it can be forgotten later (one-line shim)
metadata = stamp(metadata, subject_id="user_42")   # wrap your existing add_documents/insert

# 2) on an ERASURE request — hard-delete + force physical reclaim + verify
report = forget("user_42", store)
print(report.to_json(indent=2))
print("erased & verified:", report.ok)             # True only on visibility PASS + VERIFIED

# safe on a hot prod DB — delete + verify only, no REINDEX/VACUUM (residue stays PENDING)
report = forget("user_42", store, force=False)

# standalone proof pass, no delete
report = verify("user_42", store)

What it honestly claims — and does not

ForgetReport carries this boundary in the report itself (not just the docs):

Scoped to the live store(s) you pointed it at, at the time it ran. Does not cover backups/snapshots, replicas, WAL/logs, other stores (caches, a second DB, object storage, logs), embedding-provider-side copies, or data baked into fine-tuned model weights. It is a removal report, not a signed certificate of erasure.

On managed Postgres (RDS/Neon/Supabase) the physical-residue check may be unavailable → the report says residue: UNVERIFIED-managed rather than claiming more than it can prove. Under-claiming is the design rule.

Package-provenance attestations (PEP 740) emitted by the PyPI publish pipeline attest the wheel's build origin — they are not an erasure attestation and never sign a ForgetReport. Don't conflate a signed package with a signed proof of deletion.

Residue states

residue	meaning
VERIFIED	dead tuples reclaimed + index rebuilt (not "prior bytes overwritten")
PENDING-lock-timeout	reclaim hit the bounded lock wait — transient, retry
PENDING-<pin>	dead tuples pinned by a long txn / replication slot / prepared xact
UNVERIFIED-managed	managed PG / least-privilege role blocked the reclaim
UNVERIFIED-index-not-rebuilt	a vector index left invalid (ghost nodes may survive)
UNVERIFIED-dead-tuples-remain	tuples remained, no transient pin found

report.ok is True only on visibility PASS and residue == VERIFIED.

Operational caveats

• force_reclaim can wait on long-running transactions. REINDEX INDEX CONCURRENTLY must wait for older snapshots to finish — correct Postgres behaviour, but on a busy DB it could otherwise block indefinitely. The reclaim connection therefore runs with a bounded lock_timeout (default 5 s, reclaim_lock_timeout_ms; 0 disables). On timeout the run degrades to residue: PENDING-lock-timeout — a transient, retryable state, not the permanent UNVERIFIED-managed — and forget() returns instead of hanging. A timed-out REINDEX CONCURRENTLY can leave an INVALID index behind; drop it (DROP INDEX …) or a later run will flag it as UNVERIFIED-index-not-rebuilt.
• Retry is an opt-in poll: forget(..., retries=N, backoff_ms=M) re-attempts the reclaim while the residue is a transient PENDING-* (e.g. the blocking txn clears between attempts). It never turns an UNVERIFIED-* or a visibility FAIL into a pass, and total added wait is bounded.
• Zero-risk mode: force=False does delete + verify only (no REINDEX/VACUUM), so it can never block on a lock. It reports residue: PENDING (compaction not performed).
• reclaim_statement_timeout_ms (default 0 / off) is an extra hard cap on reclaim runtime. Leave it off unless you want to bound total work — unlike lock_timeout (which targets waits), it can cancel a legitimately long VACUUM/REINDEX.

Non-goals

• Not a signed erasure certificate / crypto-shred / cryptographic proof — ForgetReport is deliberately an unsigned removal report.
• Not machine-unlearning of fine-tuned weights — out of a library's reach; PII baked into model weights is explicitly out of scope.
• Not a hosted service or daemon — library only.
• v0.1 is pgvector-only — Qdrant, Weaviate, and framework (LangChain/LlamaIndex) integrations are deferred.

Status

Alpha (v0.1).

License

MIT © privatebydefault