mindex 2.0.0

⚠️ REQUIRED — THIS IS A REBUILD, NOT AN UPGRADE

Every index must be rebuilt from scratch. The vectors are from a different model;
nothing migrates them.

# 1. the database lineage restarts — a pre-v2 database is REFUSED at startup, by name
rm ~/.local/share/mindex/mindex.db          # or wherever [database].path points
#    the Docker test stack needs `down -v` once

# 2. the embedder is no longer shipped. Install one that speaks /v1/embeddings —
#    deploy/embedder/ has a ~200-line reference server, llama.cpp and vLLM recipes,
#    and the measured throughput that tells them apart. Install it OUTSIDE the checkout.

# 3. config keys changed, and stale ones now fail at startup rather than being ignored
#    [model].name -> [model].id (a registry id: qwen3-embedding-0.6b | -4b | -8b)
#    removed: [qdrant].{dense_prefetch_limit,sparse_prefetch_limit,fusion_limit}
#             [indexing].sparse_min_weight

# 4. rebuild, then drop the _v1/_v2 Qdrant collections the startup log names
mindex-index --force --root <each project>

Runbook: docs/claude/qdrant.md. [slicer].max_chunk_tokens also moves 512 → 364,
which is part of what the rebuild produces.

FOR HUMANS

Search got a lot better, and the reason is that we finally measured it. mindex's
retrieval had never been evaluated: the numbers in its documentation came from a one-off
that no longer existed, over a question set that existed nowhere. This release ships
bench/ — a pre-registered harness whose ground truth is each project's own Sphinx
documentation resolved by AST against the source tree, so no model and no LLM ever touches
the answer key — and then acts on what it found.

The shipped v2.0.0 server against the shipped 1.2.0 server, same corpus, same queries:
nDCG@10 0.3549 → 0.4563, Δ +0.1014, 95% CI [+0.0832, +0.1190], p = 0.0001 at n = 1115.
One corpus, and the harness says so itself.

Three retrieval components were deleted, and the measurement is why. mindex used to run
BGE-M3's three heads: a dense vector, a sparse one fused in by reciprocal rank fusion, and
a late-interaction (ColBERT) rerank on top. Measured against each other:

• RRF scored below the single dense leg it fused (0.4164 against 0.4448).
• The sparse leg was worth +0.004 with both confidence intervals through zero, once the
  dense leg was a 2026 encoder. Against BGE-M3's own weak dense head it had been worth
  +0.015 — it was compensating, not contributing.
• The rerank significantly harmed long queries (−0.016, p = 0.023) and was never
  established either way on short ones, a comparison this corpus is underpowered for by
  3×. It cost 99.6% of Qdrant's storage: 838 MB per segment against 2.6 MB dense.

So the pipeline is now one dense vector and one Qdrant query. The embedder was the lever;
the extra heads were not.

Switching models is cheap now, and switching sizes is nearly free. The model is a
registry entry — three Qwen3-Embedding sizes today (0.6B / 4B / 8B), one shared tokenizer
— and collections are named per (project, model). Changing [model].id writes a new
collection and holds the old one, so switching back is instant reuse rather than a
rebuild. Because all three sizes tokenize identically, changing size is mindex-index --vectors-only: re-embed the stored chunks, no re-slicing, no symbol work.

mindex stops shipping an embedder, and starts checking which one answered. The vendored
BGE-M3 server existed for one reason — nothing general returned three heads at once — and
one head makes it deletable. What replaces it is a contract: /v1/embeddings,
/v1/models, /health. But a general server brings a general problem, so
GET /v1/models is now a handshake: a server that answers and names a different model
is refused at startup, every response row is checked against the registry's width, and
GET /health re-checks it. None of that was checked before — a wrong embedder behind the
right URL indexed in silence.

Choose the serving stack by the indexing number, not the protocol. They differ by an
order of magnitude for identical vectors: this repository reindexes in 51 s through the
reference torch server and 410 s through llama.cpp, while query latency is 16 ms
against 30 ms. Numbers, method, and the three traps that cost real debugging (bf16 vs fp16
NaN, token-budget batching, empty_cache() corrupting output on ROCm) are in
deploy/embedder/README.md.

WHAT THIS RELEASE DOES NOT CLAIM

Stated here because a benchmark you cannot check is a press release.

• Two corpora, one language, one query shape. django and scikit-learn, both Python,
  both query sets documentation prose. Every other corpus in the plan is declared and
  unrun.
• The v2-vs-v3 number is one corpus and one comparison, and it is a system comparison
  — the embedder, the chunk window and the tokenizer move together, so none of the gain is
  attributable to one of them.
• The model that ships is not the one the model comparison selected.
  granite-embedding-english-r2 was statistically indistinguishable and cheaper; Qwen3 was
  chosen for multilingual queries and its one-tokenizer size ladder, neither of which this
  harness measures.
• The 364 chunk window is exploratory: a sweep rather than the pre-registered
  experiment, on one corpus, measured under the previous tokenizer.
• "TOST" and "Holm–Bonferroni" are not what the code does — the non-inferiority check
  is a confidence interval clearing a margin, and the family-wise correction was applied by
  hand in one family and nowhere else. Both are relabelled and disclosed.
• The noise floor was measured on the query path only, so cross-index comparisons —
  which includes the chunk window — have no measured floor.

Every one of these is in bench/FINDINGS.md's opening section, and every deviation from
the pre-registration is a dated row in bench/PROTOCOL.md §11.

FOR MACHINES

• GET /config publishes embedding_dim, min_chunk_tokens, max_chunk_tokens;
  model_id is the canonical registry id, not an HF repo.
• GET /version and mindex_build_info report 2.0.0; build_info now carries a
  model_id label.
• Collections are {guid_simple}_{slug}_v3. DELETE /projects/{guid} drops every model's.
• POST /index accepts vectors_only (re-embed stored chunks) beside symbols_only;
  the two are mutually exclusive, 400 validation.index_modes_exclusive.
• Four error codes joined the published catalogue in the OpenAPI description:
  index.file_in_flight, auth.route_not_configured,
  validation.index_modes_exclusive, research.invented. A test now walks every variant
  against that document.
• project_files gains chunker_id and embedded_model_id; both join the unchanged-file
  predicate, so flipping the model self-heals like a version bump.
• Nothing changed in the /research SSE contract, /search's request or response
  shape, or the authorization model. A 1.2.0 client works against a 2.0.0 server.

ARTEFACTS

mindex-index and mindex-watch for Linux, Windows and macOS (Intel and Apple silicon);
the mindex server for Linux x86-64; the VS Code .vsix; a Docker image
(ghcr.io/silencespeakstruth/mindex:v2.0.0) and the reference embedder
(server.py + requirements.txt + its unit) — the last two are new, because a release
that says "use Docker" while publishing no image, and ships a server that cannot start
without an embedder it does not include, is not a complete release.

Each archive carries a .sha256 sidecar.