v0.2.1 — migration blast-radius artifacts

Migration blast-radius artifacts

retrieval-fairness diff can now produce a PR-friendly inventory of every chunk that loses or gains exposure during an embedder or chunking migration.

retrieval-fairness diff --baseline before.json --candidate after.json \
  --blast-radius migration-blast-radius.md --blast-corpus corpus.jsonl

Added

• deterministic Markdown and CSV blast-radius artifacts
• newly-dark and rescued chunk lists with baseline/candidate frequency and delta
• optional, output-only corpus text enrichment via --blast-corpus
• 240-character Markdown text bound by default (--blast-text-limit 0 disables it)
• lossless CSV output via --blast-format csv
• public Python API: BlastRadiusEntry, BlastRadiusReport, build_blast_radius, and save_blast_radius

Real-data validation

On the BEIR NQ lexical → dense migration, the artifact reports:

• 1,155 newly-dark chunks
• 929 rescued chunks
• 2,084 actionable rows

Full verification: 134 tests passed, 3 optional integrations skipped, 87.92% coverage, all Python 3.9–3.13 CI checks, CodeQL, dependency audits, wheel/sdist tests, and the 1M benchmark passed.

Full changelog: v0.2.0...v0.2.1

v0.2.0 — semantic identity and reproducibility

Trust and reproducibility milestone

• Schema v3 separates workload/corpus ID, content, revision, order, and FAISS mapping identities.
• Strict same-content gates detect semantic input changes while allowing row reorder and embedder migration.
• Typed credential-safe provenance records runtime, adapter, metric, normalization, search, and model details.
• Compact summaries omit raw data and bound Lorenz output to 512 points by default.
• One-pass frequency calculations reduce the 1M summary benchmark to 1.62 seconds on the audit runner.
• Deterministic tie policies and checksum-bound atomic FAISS manifests prevent unstable or mislabeled results.
• CLI coverage, validation, sdist self-tests, Ruff, Pyright, CodeQL, Dependabot, and optional dependency audits are strengthened.

Legacy schema v1 and v2 baselines remain readable.

See CHANGELOG.md and quality verification.

v0.1.1 — trusted baselines and correctness fixes

v0.1.1 — trusted baselines and correctness fixes

v0.1.0 — exposure audit for vector search / RAG

First public release.

What it does: shows what share of your vector corpus queries actually reach and what share is never retrieved (dark matter — the antihub inventory of your index); measures exposure concentration (Gini), hub capture; regression-diffs embedder/chunking migrations; gates them in CI.

Highlights

• Metrics: coverage %, coverage of reachable ceiling, dark matter %, Gini, hub capture, Lorenz curve
• qrels lost gold: cross-check dark matter against relevance judgments — which relevant chunks are never surfaced
• diff / gate: per-chunk migration deltas (newly-dark / rescued), CI gate with strict mode and validated thresholds
• synth: antihub self-query audit when you have no query logs
• Dashboard: self-contained HTML (Lorenz, histogram, PCA corpus map)
• Adapters: InMemory, FAISS, pgvector, Qdrant
• Case study: BEIR NQ 260k chunks — coverage 11.7% of corpus = 88.3% of the workload-reachable ceiling, 905 relevant chunks lost in dark matter

Why dark matter is structural, not a bug: hubness in high-dimensional NN search (Radovanović et al., JMLR 2010) — see README.

Install: pip install retrieval-fairness (once PyPI publish completes).