v0.26.0-beta — Bitemporal, Temporal, and a retracted conclusion
Pre-release
Two new verticals, and a retraction
Seven verticals now. Bitemporal and Temporal join the five, and both cleared the consuming project's independent probe before this tag.
The retraction comes first, because it matters more than the features
0.25.0-beta shipped a guide note concluding that four of five verticals "cannot measure retrieval quality", on the strength of V1 − V8 ≈ 0. That was wrong, and it is corrected in place.
V8 puts the entire haystack in context. So V1 − V8 ≈ 0 says distractors do not confuse a reader who already has everything — it says nothing about whether selecting the right sessions matters, and no real system dumps a haystack into context. It selects k, and selecting badly is far worse than either arm.
V9 — accuracy under a k-limited reference retrieval (top-K_ref from the same BM25 retriever the calibration gate uses) is the arm that was missing:
| Vertical | V1 gold-only | V8 whole haystack | V9 BM25 top-K | headroom (V1 − V9) |
|---|---|---|---|---|
| Arithmetic | 0.94 | 0.84 | 0.32 | +0.62 |
| Forgetting | 1.00 | 1.00 | 0.57 | +0.43 |
| Episodic | 0.96 | 1.00 | 0.60 | +0.36 |
| Prospective | 0.98 | 0.96 | 0.68 | +0.30 |
| Bitemporal | 1.00 | 0.98 | 0.80 | +0.20 |
| Temporal | 1.00 | 1.00 | 0.82 | +0.18 |
| WorkingMemory | 1.00 | 1.00 | 0.88 | +0.12 |
Every vertical has substantial retrieval headroom. V1 − V9 is the headroom number; V1 − V8 keeps its narrow reading as an interference cost and is documented as not a headroom number.
TypedMemEval-Bitemporal (60 questions)
Valid time (when a fact was true) against transaction time (when the record learned it). They diverge only after a retroactive correction, and a single-clock store cannot represent the difference — so its ceiling here is structural, not a matter of retrieval quality.
V1 60/60 · pair-flip 30/30 · V2 60/60 · V3 60/60. Pair-flip 30/30 means every pair's two clocks give different answers, so answering the valid-time question when asked the transaction-time one is wrong, not imprecise.
A design prediction the probe refuted: we expected a large interference cost and measured 59/60. That is a better property — V1 ≈ V8 ≈ 1.0 means the corpus holds neither reasoning ambiguity nor retrieval difficulty, so a system failing the transaction arm can only be failing because its store cannot represent when it learned a thing.
TypedMemEval-Temporal (50 questions)
The order events occurred, against the order they were mentioned. Sessions narrate retrospectively and anchor each event to another by a stated relation, so the timestamps record when a thing was said rather than when it happened — a system that sorts by date gets a confident, checkable, wrong answer.
V1 50/50 · V3 30/30 · V8 50/50. Zero questions are answerable by date-sorting and no answer contains a digit, both enforced by generator checks rather than review.
Also
Episodic's interference cost is −0.04: two participant-attribution questions fail on gold alone and succeed on the whole haystack, so V1 is not a strict ceiling for attribution shapes.
Warning
Arithmetic's difficulty bands are inverted. V8 by band reads 0.33 / 0.76 / 1.00 / 1.00 / 1.00 — the band labelled easiest is where the answer model fails two questions in three. Do not cite them.