feat(core): measure retrieval, then fix it (BM25F + eval harness)

[siracusa5]

Summary

Retrieval was the weakest part of ContextCake and nobody could tell, because the ranker was unmeasurable. This adds the measurement, then fixes what it exposed.

recall@1   0.263 -> 0.895
recall@5   0.500 -> 1.000
mrr        0.348 -> 0.947
conflict   1.000 -> 1.000

Reproduce with npm run eval. Also records the #44 release gate and stops the release workflow from publishing past an open one.

Why this matters

Everything the cascade is good at — section merge, provenance, dated conflicts — is downstream of the agent finding the right concept. It was finding the right concept 26% of the time.

conflict was already 1.000 before the change. That is the honest read on the engine: the cascade and the conflict surfacing work, and always did. Retrieval was the broken half, and it was broken badly enough to hide the working half.

The four commits, in order

Commit	What
refactor(core)	Extract retrieval into search.mjs, behavior-identical. Prerequisite: mcp-server.mjs parses argv and can process.exit() at load, so it could not be imported, so the ranker could not be scored.
test(core)	The eval — golden question set over a committed 3-layer corpus. Baseline lands at 0.263.
feat(core)	BM25F over Porter-stemmed tokens. Baseline moves to 0.895.
ci	docs/release-gates.md plus a workflow step that blocks a release while a gate is open.

What was actually wrong

Three causes, isolated by the eval rather than guessed at:

1. Substring, not token, matching — the largest single effect. The old scorer looked for the query word I with indexOf and found the i inside identifier, in, with. Any natural-language question handed the longest document a large free score, which is why decisions/service-stack won nearly everything regardless of what was asked.
2. No IDF — a word in every document counted as much as a word in one.
3. No length normalization — repetition beat precision.

Aggregation across layers also changed from sum to best-layer: summing let a concept three layers happen to mention outvote the one document that answers the question. Measured separately — BM25 alone reaches 0.842 recall@1; best-layer aggregation takes it to 0.895.

On not cheating the benchmark

• The stemmer is the published Porter algorithm, not a suffix list written next to these questions. search.test.mjs pins it against Porter's own reference vocabulary.
• Porter's real gaps are recorded, not patched. rebalancing and rebalance genuinely do not meet; a test asserts that, so nobody "fixes" it into a corpus-fitted stemmer later.
• Questions with true synonym gaps (keep vs retained) stay in the set as the honest ceiling of lexical ranking.
• CLAUDE.md warns against tuning the stemmer or field boosts against the golden set — add questions first, then tune.

Release gate (#44)

Separate concern, same branch because it is one sitting's work.

PR #38 deferred three hosted acceptance checks to #44 as pre-release gates. 0.1.0, 0.2.0 and 0.3.0 all shipped since, all with account sync reachable: packaging throws without a hosted Supabase config, the release workflow supplies one, and AccountPanel renders unconditionally. No flag.

docs/release-gates.md splits the checks into the ones apps/desktop/test/ already discharges in CI (callback state, cancel/retry races, offline sign-out, the entire scrub/quarantine model) and the three that need a packaged app, a second Mac, and the hosted database. Those three remain open and need @siracusa5 — the OAuth flow needs real credentials.

This PR will block app-v0.3.1 until that gate is closed. That is intended.

Test plan

• npm test — 15 engine suites plus the eval, exit 0
• node --test packages/core/tests/search.test.mjs — 13 tests
• Extraction commit verified green before any ranking change
• Regression gate verified in both directions: a deliberately degraded scorer exits 1 with RETRIEVAL REGRESSED; restored, exits 0
• fixtures/mcp-tools-baseline.json unchanged — tool schemas are byte-identical
• Release-gate grep verified both ways (open blocks, closed passes)
• check-dco.sh origin/main HEAD — 4 commits signed off
• The three manual Pre-release validation: hosted OAuth and settings sync #44 checks — @siracusa5, steps in docs/release-gates.md

Notes

• Engine stays dependency-free; the stemmer and BM25 are plain Node.
• Retrieval remains O(corpus) per query, as before — unchanged, not addressed here.
• q07 ("how long do we keep logs?" against a corpus that says retained) still misses at rank 1. No stemmer bridges that; it is the number to beat if embeddings ever earn their dependency.

🤖 Generated with Claude Code