fix(benchmark): Ollama-backed QA harness + correct Recall@k scoring, retract prior figures

[salishforge]

Two related pieces of benchmark work. Refs #47.

1. QA harness runs on local Ollama (qwen3.5:cloud)

BENCHMARK_PLAN.md already documented an Ollama path, but the code hardcoded api.openai.com and threw without OPENAI_API_KEY. Now real, and the default: BENCHMARK_LIMIT=10 npm run benchmark:longmemeval-qa costs $0 and needs no key. New benchmarks/lib/llm.ts is the single transport for both LLM roles; switching to OpenAI is a base-URL change.

Comparability is enforced, not trusted. Only a gpt-4o* judge follows the paper protocol. With any other judge the runner warns, the report replaces "directly comparable to leaderboard entries" with an explicit not-comparable notice, and every manifest records judgeModel + paperProtocolJudge. Previously that claim was emitted unconditionally — switching the default would have auto-generated false credibility.

Also fixes 21 pre-existing type errors that made the QA harness uncompilable despite being marked "✅ Implemented" — it cannot ever have run. Notably ingest.ts declared allSessions inside the per-instance loop and read it after (a runtime ReferenceError), and the scripts referenced config.limit/offset/cleanup, which don't exist. Adds a request timeout; there was none, so one stalled call could hang a 500-question run.

2. Recall@k was miscomputed — figures retracted

evaluate.ts called recallAtK(ids, answers, ids.length). Passing the array's own length as k meant slice(0, k) never truncated: R@1, R@3, R@5 and R@10 were all the same number, computed over the entire retrieved set.

Not marginal — consolidation packs many sessions per row, so "top 5 rows" can hold hundreds of sessions. The published 93.2% R@5 actually meant "a gold session appeared anywhere among all sessions inside the top 5 rows."

The surrounding comment shows k was disabled deliberately — the author treated k as rows, not sessions. That's a definitional divergence from LongMemEval, so the fix is definitional:

Metric	Meaning	Use
recallAtKSessions	gold session among first k distinct sessions by rank	LongMemEval's definition — headline
recallAtKRows	gold session anywhere in top-k rows	MemForge-native, strictly ≥ sessions

Both are reported with sessionsPerRow, so the packing advantage is visible rather than baked into one number. With CONSOLIDATION_INNER_BATCH_SIZE=1 they converge — asserted in tests.

Tests where there were none

tests/benchmark-metrics.test.ts, 17 tests. The scorer had zero coverage, which is how a wrong headline number shipped and survived. The central guard — gold at rank 6 must miss at k=3 — I verified returns 1 under the old implementation and 0 under the new.

Retraction

Removed from the README badge/status/table, ROADMAP, benchmarks/README, funding.json (crawled by funding platforms), and RESULTS.md, which now explains exactly what the number measured. The PHASE_5_PLAN CI gate enforcing 93.2% is suspended — a gate defending a retracted number is worse than no gate. Figures return with the corrected re-run.

Gate	Result
benchmarks tsc	0 errors (was 21)
root type-check / lint	clean
test:benchmark-metrics	17/17
test:http / integration / epistemic-confidence	43 / 24 / 33
Ollama judge + reader smoke	verified against live daemon

🤖 Generated with Claude Code

https://claude.ai/code/session_011xCqQo49d3CEbn6oEvb3Ru