v2.0.0: stable BM25 term IDs, atomic writes, doctor
Requires a full reindex. After upgrading, run
reindex(full=True)from a session. A plain reindex skips unchanged chunks and leaves the old sparse encoding in place — which is how a corrupt index survived the documented remedy.
Coming from 0.x, migrate the graph first with
python -m rawthink_mcp.migrate --guess-domains --heal-dangling --apply.rawthink-doctortells you what is still pending.
Why this release exists
BM25 term IDs were positions in a sorted vocabulary. One new token sorting early shifted every ID after it, so sparse vectors written before a reindex stopped matching queries encoded after it.
Nothing failed. RRF kept returning good results off the dense side, so search felt fine while half the retrieval system scored noise.
Measured on a real vault of 1,841 chunks:
| before | after | |
|---|---|---|
| sparse MRR | 0.1596 | 0.6259 |
| sparse nDCG@10 | 0.1851 | 0.6320 |
| sparse hit@10 | 5/19 | 14/19 |
| dense MRR | 0.6842 | 0.6842 |
Dense staying identical is what makes the sparse delta attributable to the fix.
A second finding
The measurement that confirmed the bug also surfaced something the audit had not predicted: 40 of 69 sessions in that vault had never been indexed. The files were on disk, search could never find them, and nothing said so. After indexing the sessions the evaluation needed, dense MRR went 0.6842 → 0.9474 and sparse 0.6259 → 0.8395.
The stale index was costing more than the encoding bug. Both were invisible, and together they read as one vague sense that search could be better. rawthink-doctor now checks for exactly this.
New
rawthink-doctor— nine checks with a fix line for each failure: vault, graph schema, BM25 state format, Qdrant, index coverage, vector dimension, Ollama, MCP registration.- Concurrent writes are safe. Graph mutations take a file lock and a thread lock. Measured before the fix, with three processes writing: one writer's twelve updates all vanished, silently.
- Graceful shutdown releases the Qdrant directory lock and the graph file lock on SIGINT/SIGTERM. POSIX only — Windows terminates rather than signalling.
- 104 tests, no Docker and no Ollama required, running on 3.10–3.13 in CI.
index_vault(force=True), so a full reindex actually re-encodes.
Changed
Term IDs are a hash of the token; format_version is 2 and legacy state is rejected rather than loaded. BM25 state lives with the vault instead of the package directory. Diagnostics go to stderr — they were going to stdout, which is the JSON-RPC channel, so the record of what an indexing run skipped never arrived anywhere. Graph search matches prefixes, not substrings. open_nodes is a pure read. get_session_chunks pages.
Reading
docs/postmortem-bm25-term-drift.md — how a defect stayed invisible for months, and what made it visible.
Full detail in CHANGELOG.md.