Skip to content

v0.36.0-beta — the calibration echo was a tell, and headroom turned out to be conditional

Choose a tag to compare

@joslat joslat released this 13 Sep 16:14
· 24 commits to main since this release
bed4f26

Two arcs, one release

1 · The calibration echo was a tell — six corpora moved

equalise_echo exists so gold and its distractors carry the same number of woven filler terms; otherwise gold is identifiable by having fewer commas. On the seven verticals calibrating per shape it was handed a literal 0.0, so gold's clause was sized at echo 0 while its distractors were woven at the real knob — gold averaged 1.00 terms against filler's 3.56 (episodic), 3.17 (prospective), 2.75 (semantic), 1.93 (arithmetic).

Direction of the error: the affected corpora were EASIER for a lexical retriever than designed, so 10 of the 12 shapes that moved went up.

vertical corpus_sha256
arithmetic 535f4ed01b92 → 8de66481d5b8 filler only
conjunction a949006ce182 → 9455d2dcb0b0 filler only
forgetting be14b81ae4e2 → 2aaa0a41fc92 filler only
semantic d5eff38bc74e → 630ea487e7dc filler only
episodic bfb35552ec82 → cefa9956c0a8 🔴 35 of 50 questions differ
prospective 39f205b72294 → dd86f46ff1c4 🔴 redraw, 50 → 72 questions
bitemporal abf2f3f43219 → cdc27b225033 🔴 5 of 60 answers survive

🔴 Compare on corpus_sha256, never on question_id. All ids survive on every vertical; on the last three the questions behind them do not. procedural, temporal and workingmemory are byte-identical.

2 · ⭐ Published headroom is conditional on (BM25, K=5), and nobody had said so

realised_coverage has called itself a floor proxy a stronger retriever will exceed from the start. By how much had never been measured.

configuration shapes below the 0.15 discrimination floor
BM25, K=5 — what we publish 2 of 35
dense, K=5 8 of 35
BM25, K=10 9 of 35
dense, K=10 (45% of a median haystack) 21 of 35

Measured, not predicted (367 calls): the V9 arm re-run with cosine-over-embeddings in place of BM25 — same documents, budget, prompt and judge. 8 of 8 at-risk shapes fell below the floor, five reaching V9 exactly 1.000; 4 of 4 controls held, moving 0.000–0.200.

✅ The survivors are coherent: multi-hop, ordering and windowed constructs. A query describing a join resembles no single link, so no retriever scoring documents independently gets a set right by similarity.

Every sidecar now carries probes.retriever_sensitivity — per shape, both retrievers, discriminates_under_dense. No corpus_sha256 moved for it.

Also in this release

  • calibrate_pinned now applies the band ratchet per shape, not only to the vertical mean — a mean gate inside the mechanism that exists because means hide collapses.
  • check_tag_has_changelog_section.py: v0.35.0-beta shipped with no CHANGELOG section, and the reason it was ungated was an unmeasured claim that CI read the file. Nothing did.
  • Two out-of-band exemptions retired because the shapes came back into band; the scored-on-the-reader declaration is now checked in both directions.

Verification

  • all ten corpora reproduce byte-for-byte from their committed generators
  • 1190 Memory tests × net8.0/net9.0/net10.0, 10207 core, 19/19 CI checks green
  • 33 of 36 shapes rank two systems at (BM25, K=5), each exception declared with a two-way drift check

Full detail in CHANGELOG.md and docs/findings/MEASUREMENT_STATUS.md §88.35–§88.40.