v0.36.0-beta — the calibration echo was a tell, and headroom turned out to be conditional
Two arcs, one release
1 · The calibration echo was a tell — six corpora moved
equalise_echo exists so gold and its distractors carry the same number of woven filler terms; otherwise gold is identifiable by having fewer commas. On the seven verticals calibrating per shape it was handed a literal 0.0, so gold's clause was sized at echo 0 while its distractors were woven at the real knob — gold averaged 1.00 terms against filler's 3.56 (episodic), 3.17 (prospective), 2.75 (semantic), 1.93 (arithmetic).
Direction of the error: the affected corpora were EASIER for a lexical retriever than designed, so 10 of the 12 shapes that moved went up.
| vertical | corpus_sha256 |
|
|---|---|---|
arithmetic |
535f4ed01b92 → 8de66481d5b8 |
filler only |
conjunction |
a949006ce182 → 9455d2dcb0b0 |
filler only |
forgetting |
be14b81ae4e2 → 2aaa0a41fc92 |
filler only |
semantic |
d5eff38bc74e → 630ea487e7dc |
filler only |
episodic |
bfb35552ec82 → cefa9956c0a8 |
🔴 35 of 50 questions differ |
prospective |
39f205b72294 → dd86f46ff1c4 |
🔴 redraw, 50 → 72 questions |
bitemporal |
abf2f3f43219 → cdc27b225033 |
🔴 5 of 60 answers survive |
🔴 Compare on corpus_sha256, never on question_id. All ids survive on every vertical; on the last three the questions behind them do not. procedural, temporal and workingmemory are byte-identical.
2 · ⭐ Published headroom is conditional on (BM25, K=5), and nobody had said so
realised_coverage has called itself a floor proxy a stronger retriever will exceed from the start. By how much had never been measured.
| configuration | shapes below the 0.15 discrimination floor |
|---|---|
| BM25, K=5 — what we publish | 2 of 35 |
| dense, K=5 | 8 of 35 |
| BM25, K=10 | 9 of 35 |
| dense, K=10 (45% of a median haystack) | 21 of 35 |
Measured, not predicted (367 calls): the V9 arm re-run with cosine-over-embeddings in place of BM25 — same documents, budget, prompt and judge. 8 of 8 at-risk shapes fell below the floor, five reaching V9 exactly 1.000; 4 of 4 controls held, moving 0.000–0.200.
✅ The survivors are coherent: multi-hop, ordering and windowed constructs. A query describing a join resembles no single link, so no retriever scoring documents independently gets a set right by similarity.
Every sidecar now carries probes.retriever_sensitivity — per shape, both retrievers, discriminates_under_dense. No corpus_sha256 moved for it.
Also in this release
calibrate_pinnednow applies the band ratchet per shape, not only to the vertical mean — a mean gate inside the mechanism that exists because means hide collapses.check_tag_has_changelog_section.py:v0.35.0-betashipped with no CHANGELOG section, and the reason it was ungated was an unmeasured claim that CI read the file. Nothing did.- Two out-of-band exemptions retired because the shapes came back into band; the scored-on-the-reader declaration is now checked in both directions.
Verification
- all ten corpora reproduce byte-for-byte from their committed generators
- 1190 Memory tests × net8.0/net9.0/net10.0, 10207 core, 19/19 CI checks green
- 33 of 36 shapes rank two systems at (BM25, K=5), each exception declared with a two-way drift check
Full detail in CHANGELOG.md and docs/findings/MEASUREMENT_STATUS.md §88.35–§88.40.