v0.34.0-beta — Procedural, the tenth vertical

The TypedMemEval taxonomy closes. Procedural ships at 80 questions and headroom +0.80 — the largest in the family and fully reachable — bringing the family to 565 questions across 36 shapes in 10 verticals. Every memory type the consuming engine ships is now covered.

Purely additive. All nine existing corpora are byte-identical, verified by diffing v0.33.0-beta rather than asserted. Nothing a current consumer runs changes.

Procedural

Corpus d431a7fc9ac5. Recall only — whether a system remembers a procedure it was told across sessions, not whether it can execute one or improves at one.

shape	what only it tests
step-order	an order that must hold; violating it is an error, not a wrong answer
precondition	a constraint that is neither a step nor a value, reached through a second hop
amended-step	one element of a sequence replaced, the rest intact
retired-step	a position removed from a sequence — not a fact invalidated

V1 80/80 · V2 80/80 · V3 80/80 · V6 70/72 · V8 80/80 · V9 16/80 · headroom +0.80 · interference 0.0

The defect worth reading, and it was ours

The first build measured V6 77/80. That number was flattering. step-order's dependency sessions state the adjacent pairs of a 4-chain, so their transitive closure is the gold order — and filler stated only isolated pairs. The gold chain was the only complete order in the haystack, which made the membership session, the shape's entire second hop, redundant on all twenty questions. 14 of 60 membership-drop samples had reproduced the gold order verbatim; only 3 were condemned, the rest rescued by the resolution grader saying declined.

The arm reported four numbers before a true one — 60/80 → 77/80 → 68/80 → 51/52 → 70/72 — and each moved for a different reason, only the last moving the corpus. Three rules came out of it:

• A chance floor absent is not a floor of zero. It collapses the required-hit threshold to 1, condemning components that are load-bearing against a guesser firing at 1−(2/3)³ = 0.70. That is the understating direction, which is exactly why it survived a probe and a review.
• An undecidable result is not a pass, and a smaller denominator is not a better score. 51/52 reads better than 68/80 while saying nothing about a quarter of the corpus.
• A structural claim needs competition, not declaration. The generator now refuses a corpus whose rival chains are missing, short, or overlapping.

step-order realises 0.400 coverage against a 0.50 band floor and is declared out of band rather than tuned back in — a proxy traded for a measurement, which is what ADR-028 licenses. Removing the rivals would restore the number and delete the construct.

Also in this release

• Judge calibration re-measured over 257 cases (was 230): 0.988 / 0.992 / 0.992 across three runs, Procedural 27/27 in all three, prompt fingerprint unchanged.
• Name-collision audit extended to Procedural's seven entity banks, which the instrument had never covered — 206 names tested (was 171); procedural:STEPS, the ordered bank and the only one the harm model puts at risk, returned zero collisions.
• A dead generator branch fixed that had made the precondition frame gold-only.
• AGENTEVAL_CALIBRATION_ONLY gives the live judge arm a smoke stage that asserts nothing about agreement.

Verification

1181/1181 on net8.0, net9.0 and net10.0 · corpus reproduces byte-identically from its committed generator · probe, calibration-search and separability self-tests green · published tables regenerated from the sidecars.

Known limitation

Eight precondition questions are undecidable on V6 — their haystacks carry only two distinct conditions, so k=2 leaves no hit threshold at three samples. V6 scope is 72 of 80.

ADR-029 §§10–14 carry the full reasoning.

v0.33.0-beta — eight corpora moved; every shape is now scored

Cut for AgentMemory's C-D full-family run. It carries the two fixes named as blocking, neither
of which was in any published package before now.

	0.32.0-beta	this release
forgetting	7fe6e166dbf1	be14b81ae4e2
prospective	a570b890a5b9	39f205b72294 — due-window answer key

Breaking — eight of nine corpora changed bytes

Only bitemporal is unchanged (abf2f3f43219). question_id sets are stable across the move, so
compare on corpus_sha256, never on the question id. revision now carries the sha
(v5+<sha12>) and is gated by a test, so the metadata is load-bearing.

Two are retrieval-control changes rather than content changes: Conjunction grows 50 → 65
(ADR-029's declared cost for the new conditional-branch shape) and WorkingMemory holds H
constant at 60 non-gold sessions on every rung.

Every shape is now scored by something

32 shapes — 29 on retrieval headroom, 3 on a declared other axis with its own bar. Three shapes
were previously skipped in silence, one of them 30% of its vertical.

• V10/V11 abstention arms for questions with no gold. forgetting/never-known: 15/15 and 15/15.
• paired_arms at vertical level — forgetting 0.4667 @ 3.68 sd and prospective 0.6316 @
  5.878 sd had never been published.
• conjunction/conditional-branch — across the 470 prior questions, not one required resolving
  a conditional. V1 15/15, V9 0/15, headroom 1.00 fully reachable.
• chance_floor / v9_above_chance — 71 questions across 6 shapes name their own candidates
  and none had published a floor beside their accuracy arms.

Fixed — including a defect that shipped in 0.32.0-beta

72 distractors that answer any question asked. Forgetting's parity filler stated a value with no
setup naming the noun, so "I settled on Marloe Basic." is about nothing NAMED. 29 of 50 questions,
including 13 of the 15 haystacks whose premise is that no answer exists. No existing arm could
see it — they all ask whether the model produced THE GOLD, and this produces something else. 72 → 0.

WorkingMemory's ladder measured context volume and called it distance. H = distance + 1, and
BM25 is position-blind. 2 of 5 rungs discriminated; now 5 of 5.

V6 read 20/35 on Forgetting where it is 20/20, and its scope was a hardcoded two-vertical list —
now a per-question and per-component declaration, 85 → 200 questions, family 164/165.

prospective/due-window's answer key was wrong, not merely misleading: V8 4/18 → 16/18.

Known defects and declared residuals

arithmetic V6 49/50 (one miscount, explained rather than tuned away); two V6 results that are
undecidable rather than passing; prospective/seed-carry-over mixes two populations (0.60 and
0.1429) and publishes the split; closed_choice_k has two pinned limits; WorkingMemory gave up
volume sensitivity to isolate distance.

Full disclosure in CHANGELOG.md, derived by diffing the previous tag rather than written from
memory. ADR-028 §§13–20, ADR-029 §9.

Verification

1153/1153 on net8.0, net9.0 and net10.0; 9447/9449 on AgentEval.Tests; all nine corpora reproduce
byte-for-byte from their committed generators, in CI.

v0.32.0-beta — five corpora redrawn, and the collision is silent

Breaking — five corpora changed bytes, and the collision is SILENT

temporal, bitemporal, conjunction, episodic and forgetting all moved corpus_sha256.
arithmetic, prospective and semantic did not — their sidecars changed; their corpora did not.

vertical	corpus_sha256	hops
temporal	31d26e60fd21 → 9b83da0cd8ea	2
conjunction	c62ef4773597 → 9f6a0f37a506	3
bitemporal	f5b384d7f0ff → abf2f3f43219	1
episodic	2c6000a6912e → 542da3fa1767	1
forgetting	ba759097b9bd → 7fe6e166dbf1	1

Read this before comparing any number to a 0.31.0-beta baseline. All five redrawn corpora keep
100% identical question_id sets with 0 byte-identical items, and corpus_id
(agenteval-typedmemeval-<vertical>-v5) and revision (v5) do not move either. corpus_sha256
is the only field that distinguishes the two corpora.

In bitemporal, 27 of 60 items keep the exact same question text with a different gold answer
(tme-bit-003: same question, gold Northolt Bay → Kelsford); only 4 kept both. A consumer keying
a cache, leaderboard row or regression baseline on question_id will silently mis-grade rather
than fail loudly. Compare on the sha, never on the corpus id, revision or question id.

temporal additionally lost its entire G=1 stratum — the 20 single-gold questions are gone,
redistributing to G=3 (5→11), G=4 (5→12), G=5 (5→12). Retrieval controls change in kind, not only in
value.

Added — two shapes that could not rank anything now can

• temporal/occurrence-order: headroom 0.05 → 0.75, now the strongest shape in its vertical. It
  asked about two adjacent chain events, so the single link between them stated the answer outright
  while the question handed BM25 both rare names — a lexical lookup in the vertical whose premise is
  that narration order must be followed. It now asks the two ends, so every link is necessary.
  V9 19/20 → 5/20; gold 1 → 3–5 sessions; 0 of 20 questions have any single session naming both
  asked events, against 20 of 20 before. V1 held at 20/20.

• bitemporal/belief-at-instant: headroom 0.11 → 0.31, and on pairs 0.167 → 0.556 against a
  scaled floor of 0.254 it previously missed. Only two sessions per haystack named the asked subject;
  same-subject, other-month distractors now compete (median 2 → 5, exactly K_REF).

• conjunction/order-then-value: V9 15/15 → 7/15 — the second-largest per-shape V9 move in the
  release. It was saturated under BM25 and could not discriminate retrievers at all.

• Paired-arm discrimination. Bitemporal's 30 pairs have different correct answers and disjoint
  gold, so per-question scoring averages the capability away. pair_headroom ships with
  pair_floor_scaled and pair_separation_sd: a shape must clear both, because the floor assumes
  independent arms and they measurably are not, while the standard-error separation assumes nothing
  about correlation.

• A per-shape coverage band gate, which immediately found three shapes outside ADR-026's
  [0.50, 0.90] that nothing had reported.

• --dry-run for the probe runner — every case through the real code path against a stub: no API
  calls, no credentials, nothing written. It found a real bug on its first execution.

• --recalibrate on all nine generators. Calibration is now an authoring step, not a build step:
  a plain regeneration rebuilds at the echo the sidecar records and reproduces byte for byte.
  Previously a corpus was a function of (generator, seed, search algorithm), so changing the search
  silently desynchronised every committed corpus from the generator that produces it.

Fixed — previously published numbers that were wrong

Per-shape realised coverage never described the shipped corpus. calibrate_per_shape recorded
each shape's value mid-search, while later shapes' knobs were still 0, then rebuilt with all knobs
set.

shape	published	actual
arithmetic/delta	0.7767	0.7350
arithmetic/duration	0.6528	0.6250
conjunction/alias-then-count	0.3444	0.3356
semantic/co-reference	0.6889	0.7222
prospective/not-yet-true	0.6667	1.0000 — saturated
prospective/due-later-reminder	0.75	0.50
prospective/due-window	0.2222	0.4352
prospective/expiring-validity	0.8333	0.6667

semantic's echo never moved, so this was long-standing and independent of the calibration work.

The guide asserted something false. It read "V1 − V9 is the headroom a better retriever can
capture." It is not — a real retriever returns gold plus what else it ranks highly, so its ceiling
is V8, not V1. headroom_reachable had been in the sidecar since 0.31 and the word "reachable"
appeared zero times in the docs. prospective/due-window publishes 0.94 of which 0.17 is
reachable.

The calibration search was fixed twice. Bisection assumed a monotone most shapes violate; the
sweep that replaced it was itself a regression on cliff-shaped curves, returning coverage 0.000
on arithmetic/delta where bisection returned 0.735. A sweep locates and bisection resolves;
both are needed. See ADR-028 §12.

Three sidecars had lost fields they carried at 0.31.0-beta — structure.retrieval_ceiling and
structure.scaffolding_dependence on bitemporal, episodic and forgetting. Regenerating those
corpora dropped them correctly (the values described the old bytes) and nothing re-ran the stampers.
Restored, with no sha change.

Known defects shipping in this release

• prospective/due-window's answer key is wrong. Class parity requires filler to use gold's own
  construction, and this shape asks a set-membership question whose membership criterion is that
  construction — so filler reminders falling inside the window satisfy the question and are not
  gold. The reference model is marked wrong for being right. A fix is written and measured
  (V8 4/18 → 13/18, reachable headroom 0.17 → 0.44) but is not in this release; it cannot land
  while the separability tell below is unfixed. Read its published 0.94 / 0.17 as mis-keyed, not
  hard.

• prospective/not-yet-true is saturated at coverage 1.0 — BM25 returns gold for every question.
  Its headroom of 0.1667 is one question out of six.

• A measured, unfixed separability tell. Gold's first assistant turn is longer than every
  distractor's in 8–10 of 50 prospective questions against 4.1 expected by chance, and 6 of 50 in
  episodic. Diagnosed in tools/diagnose_padding_asymmetry.py: gold's first assistant turn starts
  45 and 36 characters shorter than filler's in those two verticals, so it takes the most
  whole-sentence padding steps and the last one overshoots. semantic, temporal and bitemporal
  have comparable base texts and show no tell.

• 61 of 75 measured real-world name collisions remain unremediated, declared in
  tools/name-collision-audit.json. The guard scans only temporal and conjunction, and only for
  the 14 remediated names.

• forgetting/never-known is checked by nothing. Its 15 questions have no gold, so every validity
  arm reports 0/0 and both discrimination assertions silently continue.

Changed — defaults and CI a fork inherits

• TypedMemEvalOptions.JudgeMaxOutputTokens 512 → 1500, the value the shipped judge calibration
  was measured at. ExternalBenchmarkOptions.JudgeMaxOutputTokens is unchanged at 256, so LongMemEval
  judge budgets are untouched.
• llm-integration-tests.yml default deployment gpt-4o-mini → gpt-5.5.
• A release-blocking gate in release.yml, between dotnet pack and dotnet nuget push, verifies
  every embedded corpus against the working tree — 9 corpora × 3 frameworks. A release can now fail
  after packing; that is this gate, not a packaging outage.
• corpus-reproducibility.yml regenerates all nine corpora and compares bytes, triggered by changes
  to tools/gen_*, typedmemeval_common.py or the corpora.
• New test classes: TypedMemEvalPackagedCorpusTests, TypedMemEvalNameCollisionTests,
  TypedMemEvalCoverageBandTests, TypedMemEvalDiscriminationTests, TypedMemEvalJudgeBudgetTests.
  Several read from disk, so the suite is no longer runnable outside a full repo checkout.

Sidecar schema

• Added: coverage.echo_by_shape, coverage.per_shape_realised, and per shape
  headroom_perfect_selector, headroom_reachable, limited_by, discriminates, plus pair_*
  fields where a shape has arms.
• Removed, and the absence is the disclosure: probes.empty_completion_disclosure,
  probes.empty_rate_scope and probes.no_answer_captured are conditionally emitted. With zero empty
  completions on the current corpora there is nothing to disclose. Read their absence as "none",
  not as "not measured".

Added

• Every shape now publishes what KIND of hard it is, and a gate on whether it can rank anything.
  Implements ADR-028 §3a and §3e.

  V1 − V9 is what a perfect selector buys — a retriever returning gold and nothing else. A real
  retriever returns gold plus whatever else it ranks highly, so it cannot beat having everything:
  its ceiling is V8, not V1. Where those diverge the published headroom is unreachable, and a
  consumer reading it buys retrieval work that cannot help.

  shape	headroom (perfect)	reachable	limited by
  prospective/due-window	0.94	0.17	reasoning
  semantic/co-reference	0.40	0.27	retrieval
  arithmetic/delta	0.90	0.90	retrieval

  `d...

v0.31.0-beta — Three fixes, and five defects it declares rather than hides

Nine verticals, 470 questions. Three fixes, five instruments — and a changelog that names five defects this release does not fix.

Temporal recency could not rank anything

It scored 15/15 at V1, V8 and V9 — the only shape in the family on which no two systems could be told apart. It asked about the last three events in the chain, so gold was two adjacent links and the answer was a single transitive step over two sessions that named the asked events outright.

Guessability was ruled out before the construct was touched: the answer was first-named 6/15, middle 3/15, last 6/15, all at chance. The shape was sound and simply too easy on both halves.

It now asks about events spanning the chain.

	before	after
recency V9	15/15	6/15
vertical V9	42/50	30/50
headroom	0.16	0.40

Gold grows from 2 links to count − 1 and stays minimal: on A<B<C<D<E asked over {A, C, E}, dropping any intermediate link removes a transitive step the answer needs.

The whole vertical was redrawn. A seeded generator redraws everything downstream of any change to the draw sequence, so all 50 questions differ — not the 15 that were targeted. Question ids are stable across that redraw, which makes an id-keyed join succeed while comparing different questions. Temporal sha a6c10b3d… → 31d26e60…; controls reset.

Prospective: firing semantics are now required, not optional

The consuming project ran the corpus with ProspectiveFiring and ValidTime=Current both dark and scored 49/50. Every shape named the thing, which hands a similarity retriever the words of the session it needs while the harness supplies "today" and the corpus supplies the due date — in-context arithmetic no memory feature is needed for.

The new due-window shape names nothing: several reminders whose only distinguishing property is when each falls due, and an answer that is a set whose membership changes with the as-of instant.

	before	after
V8 interference cost	0.00	0.28 — the family's first real one
V9	34/50	23/50
headroom	0.32	0.54

Prospective sha 6ddd3e9b… → a570b890…; also a full redraw, same id caveat.

Semantic gained a judge body — and the shared preamble did not change

Semantic shipped without a body deliberately, but that left it the only vertical with nothing to settle the preamble's abstained/missed line, which states the distinction as uncertainty versus denial and then illustrates abstained with "I have no record of that" — a denial.

Editing the preamble was tried, measured and reverted: it converted the family's canonical genuine-refusal cases to Missed across four verticals and dropped agreement 0.983 → 0.966. The rule that survives is a vertical body supersedes the shared preamble.

All 26 Semantic cases were enumerated before a line of the body was written, and 19 carry a declared route asserted independently of the outcome — so a template reaching the right label by the wrong reasoning fails rather than passing quietly. Semantic is now 26/26 in all three runs; family agreement 0.987 (0.991 / 0.987 / 0.996).

What this release admits about itself

Five defects ship declared and unfixed, in the changelog, with their directions stated. A release that carries wrong published numbers silently is the failure mode these corpora exist to catch.

V2 cannot judge closed-choice questions. Its reject line is an observed rate of 0.20; the chance floor is 0.50 at k=2. 71 of 470 questions enumerate their own alternatives, and 69 carry a V2 pass that records the reference model abstaining, not the corpus being non-inferable. A pure guesser is rejected with probability 0.989. Flattering direction.

The same floor sits under published arm counts. Chance-corrected: prospective V9 14/21 (0.67) → 0.33, temporal 24/35 (0.69) → 0.45, episodic 14/15 → 0.87. Headroom is a difference and largely cancels; the absolute counts do not.

Forgetting's coverage is diluted. Published mean_realised 0.670; over questions that have gold, 0.529, against a band floor of 0.50. Its 15 never-known probes are G=0 by design and realised_coverage returns a constant 1.0 for them, so 30% of the published mean measures nothing — and the echo calibration optimised that. Queued as a declared corpus revision.

Two shapes are out of band and were declared nowhere: episodic/list-order 0.275 (a reference retriever surfaces a quarter of its gold, so V9 there measures retrieval failure rather than reasoning) and forgetting/still-valid 0.467.

85.7% of the family by character is equalisation padding, uniform across all nine (82.4%–88.5%). Measured to be load-bearing, not waste — removing it puts 12 features over the 0.75 separability bar where 0 are now — so no redesign is proposed. Arithmetic additionally carries ledger voice (a bare common-noun subject carrying a value directly) at 77.5% of value-bearing sentences and 0.0% everywhere else.

Five instruments, none of which costs a model call

tool	what it answers
validate_v2_chance_floor.py	which passes a coin-flipper would also have earned
measure_signal_density.py	what fraction of the text is content at all
measure_padding_value.py	whether the scaffolding earns its keep
validate_coverage_population.py	whether a published mean counts items it cannot measure
make_padding_free_variant.py	controlled diagnostic cells for a consumer's extractor

Corpus verification

vertical	sha256
arithmetic	2feda94be7e8c026	unmoved
bitemporal	f5b384d7f0ff9c0f	unmoved
conjunction	c62ef477359749c1	unmoved
episodic	2c6000a6912e5ef0	unmoved
forgetting	ba759097b9bd1fd3	unmoved
prospective	a570b890a5b92c3b	moved
semantic	de0b1c22521198de	unmoved
temporal	31d26e60fd21d00e	moved
workingmemory	7e04e4cb1717cbbf	unmoved

All nine: probes.status = run, probed_corpus_sha256 == shipped sha, reference deployment gpt-5.5. Judge prompt fingerprint b1d3f7216032ba27…, calibration 230 cases over 3 runs.

Independently verified read-only by the consuming project's pre-tag probe before this tag was cut.

v0.30.0-beta — Two new verticals, and one shape that cannot discriminate

Nine verticals, 470 questions. Two new ones, both built controls-first with a measured baseline before any judge template was written.

TypedMemEval-Conjunction — questions no single memory type can answer

A fact of type A must be resolved and an operation of type B applied to it. Retrieving either half is necessary and neither is sufficient, so a stack strong on one type and weak on the other scores like a stack weak on both — which is what a per-type score cannot show.

shape	join	n
value-then-count	Semantic current-value + Arithmetic count	20
alias-then-count	Semantic co-reference + Arithmetic count	15
order-then-value	Temporal order + Semantic current-value	15

V1 49/50   V2 50/50   V3 49/50   V8 50/50   V9 18/50   headroom 0.62

Read the shapes, never the mean:

shape	V9	headroom
alias-then-count	1/15	0.93
value-then-count	2/20	0.85
order-then-value	15/15	0.00

order-then-value is saturated under BM25 and cannot discriminate retrievers at all. The 0.62 mean is carried entirely by the other two — the mean-satisfiable-by-averaging defect one level up, at headroom rather than coverage, declared in the corpus rather than left inside an average.

The first vertical with genuinely mixed gold (35 arithmetic+semantic, 15 semantic+temporal), so a per-type denominator is computable. ADR-027 §10's instruction not to inherit the parts' certifications proved load-bearing: its own V7 caught two gold-only constructions the parent verticals' passes would have papered over.

TypedMemEval-Semantic — resolution, not recall

§2.1 refused plain-fact Semantic as saturated by construction. These three shapes share what plain recall lacks — retrieving the evidence is necessary and not sufficient: current-value (an attribute replaced k times), co-reference (a fact asked under a different designation), source-attribution (which conversation a belief came from).

V1 50/50   V9 34/50   headroom 0.32

It ships with no judge body, and that is the finding — 0.958 across three runs on the shared preamble alone. Bitemporal and Temporal each needed one because each genuinely collided with the preamble; Semantic does not collide.

The judge now reports the route it took

Checking only the final label cannot distinguish a template that discriminates properly from one that suppresses a label outright. question_asks is emitted in the judge's JSON at zero extra calls and asserted independently of the outcome, so reaching the right answer by the wrong route fails the build.

Alongside it: a per-vertical judge floor of 0.80 on both the recorded result and the live arm — a family-wide threshold is satisfiable by averaging, which is how Bitemporal once sat at 0.750 behind a green 0.946. Falsification-verified.

Family agreement 0.991 over 224 calibration cases, eight of nine verticals at 1.000.

Also

• Temporal gained a judge body — the second and last vertical falling through to StandardBody. Measured against a baseline taken before the body existed.
• bench typedmemeval --evidence-detail references|content — Full capture was unreachable because the command passed options: null unconditionally. Every layer beneath was correct; the caller never set it.
• tools/validate_factgrain_axis.py — the fact-grain difficulty axis cannot be validated against our own arms: the only measure that predicts V9 misses is derived from the same BM25 that V9 is. Committed as a runnable script so the negative result is reproducible.
• Per-item gold type labels in the sidecar — corpus_sha256 covers the whole corpus JSON, so putting them in the extension would have invalidated every probe record.

Verifiable without spending anything

Only semantic (de0b1c22521198de) and conjunction (c62ef477359749c1) are new bytes. Every other corpus is byte-identical to 0.29.0-beta, including bitemporal (f5b384d7f0ff9c0f). Empty rate 0.0 on every arm of every vertical, and probed_corpus_sha256 == corpus_sha256 throughout.

Full changelog: https://github.com/AgentEvalHQ/AgentEval/blob/v0.30.0-beta/CHANGELOG.md

v0.29.0-beta — A mean that hid a destroyed shape, and silence that was still a verdict

Every headline number here is re-measured against gpt-5.5, not carried forward. All four items are instrument defects — the corpus designs were sound; the things measuring them were not.

A vertical mean is satisfiable by averaging

Arithmetic calibrated to 0.700 — dead on target, gate green, 985 tests passing — while its four shapes sat at:

shape	before	after	echo	n	V9
count	0.857	0.827	0.1250	14	7/14
delta	0.947	0.777	0.3125	10	1/10
duration	0.083	0.653	0.1250	12	5/12
sum	0.894	0.818	0.1875	14	6/14

A convention clause collapsed duration's lexical retrievability and the single echo knob compensated, loosening the other three until the average returned. count at 0.857 was not improving — it was being paid for. Calibration is now per shape, and the gate holds the band within every shape.

The search was also stopping at the first in-band rung rather than converging on a target, so stamped difficulty was set by grid placement instead of intent. Running the new per-shape gate red-first found ten shapes across five verticals outside the band — none caused by this work, none previously visible, because nothing had ever looked below a vertical mean. Pinned as a ratchet.

Silence was still a verdict — in both directions at once

0.27's retry cut V3's empty rate 78.2% → 3.1%, but frequency is not accounting: the residue still scored, as a PASS on V2/V3/V6 and a FAILURE on V1/V8/V9. V3 passes when an ablated context fails to reproduce the answer, and an empty completion reproduces nothing — so the arm whose whole job is proving gold is load-bearing was certifying validity the evidence did not support.

The ceiling had been sized on censored data: every recorded empty carried reasoning_tokens exactly equal to the 8,000 cap. Replayed uncensored: 153 / 7,677 / 14,639 — the cap sat almost exactly on the median. And the retry ladder was the real constraint: at ×3 from 900, two retries topped out at 8,100, so raising the cap alone could never have reached 14,639.

Re-probed on the corrected instrument — 0.0% empty on all six arms, unmeasured_no_answer null:

V1 50/50   V2 50/50   V3 49/50   V6 49/50   V8 50/50   V9 19/50   headroom 0.62

Bitemporal had no judge template — and the first fix overfitted where nothing could see it

Bitemporal shipped in 0.26.0-beta falling through to StandardBody, which is the two words "Grade this answer." The shared preamble defines premature as asserting as already true something gold says has not happened yet, and bitemporal golds justify themselves with exactly that sentence — so the judge graded the justification clause rather than the value. Agreement 0.750 / 0.792 / 0.792, never measured before, because the published 0.983 covered 120 cases from the five verticals that existed on 2026-08-15.

The first fix said premature "will essentially never apply" here. It scored 24/24 — and was overfitted: the set contained zero Premature-labelled bitemporal cases, so a rule suppressing Premature could not be penalised by the only instrument watching. Four negative controls were added, graded blunt to subtle; the overfitted version scores 0 of 4 on them.

The shipped fix is a question-type discriminator — asks which value → wrong; asks whether a correction had been made → premature. Bitemporal 28/28 in all three runs, family 0.983 over 172 cases, shared preamble untouched, blast radius measured across all seven verticals.

Also

• The judge-calibration record could describe a subset of the set it validates. It read cases: 120 while the set held 168 — Bitemporal and Temporal both route to StandardBody, already inside the fingerprint, so adding them changed nothing and fired nothing. The gate now requires the recorded count to equal the live set and every vertical to carry a per-vertical entry.
• The BM25 calibration gate read its acceptance band out of the artifact it was grading.
• The probe cache flushed every fifty calls, so an interrupted ten-minute window banked nothing. Now every ten.

Verifiable without spending anything

• corpus_sha256 == 2feda94be7e8c026…, probed_corpus_sha256 equal to it
• coverage.per_shape_realised — four keys, every value in [0.50, 0.90]
• probes.status == "run", reference_deployment == "gpt-5.5"
• every arm in empty_rate_by_arm at rate == 0.0
• headroom_over_lexical_retrieval == 0.62

Arithmetic question text changed, so duration-bearing controls downstream reset. duration's V8/V9 are also structurally depressed independently — median 6 required sessions against 4 elsewhere, mean ceiling 0.917, 6 of 12 questions above K_ref=5 — so part of its V9 is unreachable by any ranker at k=5.

Full changelog: https://github.com/AgentEvalHQ/AgentEval/blob/v0.29.0-beta/CHANGELOG.md

v0.28.0-beta — The empty-rate gate understated every arm it measured

0.27.0-beta shipped a per-arm empty-response statistic so that "a reasoning deployment burned its completion budget" would fail at authoring time instead of being found forensically. The instrument itself was mis-measuring, in two ways, and both understated the defect it exists to expose.

The two defects

A fallback that borrowed a real arm's identity. Arm tokens were parsed as v + digits, with return "v1" on no match. v9strip is a real arm, not a malformed v9, so its 700 calls were filed under v1: v1's denominator read 920 against a true 220, v9strip's empties landed in v1's numerator, and v9strip had no row and therefore no ceiling — it cleared the gate by not being in it.

Judge grades pooled into probe denominators. Judge calls are a different population, and a healthy one (0 empty of 1246), so including them mechanically dragged every rate down.

Corrected figures (probe answers only)

arm	published in 0.27.0-beta	corrected
v3	258/387 (66.7%)	258/330 (78.2%)
v6	182/861 (21.1%)	182/675 (27.0%)
v9	8/212 (3.8%)	8/110 (7.3%)
v1	4/920 (0.4%)	0/110 (0.0%)
v9strip	not measured	4/352 (1.1%)

V9's true rate breaches the 5% ceiling and always did — pooling 102 judge grades into 110 probe calls is the only reason it read as passing. It is now a ratchet entry: visible, and able only to shrink. Its direction is conservative (silence scores as a failure on V9), so the published retrieval ceiling remains a lower bound.

Every error runs the same way — understated, never overstated. No conclusion flips: V3/V6 remain uncitable pending re-run, by a wider margin than first stated.

Hardening

The gate now asserts the recorded arm set equals a C# list, rather than checking only that present arms are under their ceilings — the same pass-by-absence defence the V7 separability test uses. An unattributable key becomes unknown, which the gate fails on, instead of borrowing a real arm's identity.

Blast radius

Corpus bytes are untouched. Only the metadata stamp changed, so every probed_corpus_sha256 still matches and no probe re-run was required. Recomputed offline from the same call cache via --restamp-empty-rates-from-cache, which reuses the runner's own attribution rather than reimplementing it.

Consumers pinned to 0.27.0-beta need no re-run and no data migration — the corpora they load are byte-identical. Upgrade to correct the published statistic, not to change results.

Verification

Falsified three ways (removing an arm, injecting an unknown bucket, regressing a rate — each fails the gate). 985/985 memory tests pass. Six attribution cases added to the CI self-test. Stamped data re-derived by an independent parse across all 7 corpora.

Also fixes a latent CI flake: a LongMemEval percent-rendering guard asserted "5000" appeared nowhere in a transcript that prints a random temp path, and failed on a hex coincidence.

v0.27.0-beta — Evidence coverage, and four defects it uncovered

The instrument

QuestionEvidenceDiagnostics gains RequiredEvidenceSessionCount, RequiredEvidenceSessionsRetrieved and RequiredEvidenceSessionsInAnswerContext.

Every gold diagnostic before this was an at-least-one check over Retrieved, which is adequate only when a single session carries the answer — for a question assembled from four, one-of-four and four-of-four both reported GoldSessionPresent: true. And AnswerContext, the references actually supplied to the answer model, carried no gold analysis at all. Retrieval could rank every required session highly while a downstream context budget dropped most of them, with nothing in the envelope to show it.

The gap between the two new counts localizes a failure to a boundary rather than a subsystem. It is session-based rather than text-based, so it needs no evidence content and works under EvidenceCaptureMode.References with no privacy implication.

RequiredEvidenceSessionsInAnswerContext is null, never 0, when no answer-context reference carries a session ID — uninstrumented and starved must not read alike.

Four defects this uncovered, worst first

V3 and V6 count silence as a PASS. 258 of 387 V3 calls (66.7%) and 182 of 861 V6 calls (21.1%) returned no content. V3 passes when a gold-ablated context fails to reproduce the answer, and an empty completion cannot reproduce anything. This direction is anti-conservative — it certifies validity the evidence does not support — and V3 is the probe that establishes gold is load-bearing. V3 and V6 results are uncitable until re-run. V1, V2 and V7 are unaffected.

V8 and V9 count silence as error. 5 of 10 V8 failures and 32 of 111 V9 failures family-wide had no captured answer. Conservative, so every published ceiling is a lower bound. Disclosed per corpus as probes.no_answer_captured, with question IDs.

No vertical has a validated difficulty ladder. The rule that certified one ranked coverage with the calibration scaffolding in place and never compared the slope against the structural ceiling min(1, K/G). Neither correction works alone, which is why it survived three revisions. Every corpus now carries difficulty_validated: false; WorkingMemory's stamp is retired.

Arithmetic's duration never states its day-counting convention and gold silently fixes one. Four of four misses across two independent oracles are exactly gold + spells, with perfect arithmetic in every one.

Gates added

• Empty-response rate per probe arm is published as probes.empty_rate_by_arm and gated by a corpus test. The ceiling and ratchet are C# constants, never read from the record. Verified by tightening the ratchet until it failed on all seven verticals.
• A docs check refuses prose asserting a validated ladder while no corpus stamps one. Verified against the pre-fix tree: it rediscovers all three stale claims, no false positives.

Also in this release

PartnerDesk "The Trusted Supplier" sample — a third-party MCP turns a well-behaved due-diligence agent into a data-exfiltration tool, and Gatekeeper stops it at two levels. A real MCP server as a child process over stdio, four phases on one keypress, and 75 tests asserting over the recorded trajectory, the tool-effect ledger and Gatekeeper's verdicts rather than console text.

Verification

Release configuration, every target framework: 9202 net8.0, 9202 net9.0, 9420 net10.0, 985 × 3 memory, 75 PartnerDesk, 9 NuGet-consumer. Zero failures, 13/13 CI checks green.

Upgrading

Additive. Nothing removed or renamed. If you consume V3/V6 results, treat them as unestablished until the re-baseline; if you compare against published V8/V9 ceilings, exclude the no_answer_captured question IDs.

v0.26.0-beta — Bitemporal, Temporal, and a retracted conclusion

Two new verticals, and a retraction

Seven verticals now. Bitemporal and Temporal join the five, and both cleared the consuming project's independent probe before this tag.

The retraction comes first, because it matters more than the features

0.25.0-beta shipped a guide note concluding that four of five verticals "cannot measure retrieval quality", on the strength of V1 − V8 ≈ 0. That was wrong, and it is corrected in place.

V8 puts the entire haystack in context. So V1 − V8 ≈ 0 says distractors do not confuse a reader who already has everything — it says nothing about whether selecting the right sessions matters, and no real system dumps a haystack into context. It selects k, and selecting badly is far worse than either arm.

V9 — accuracy under a k-limited reference retrieval (top-K_ref from the same BM25 retriever the calibration gate uses) is the arm that was missing:

Vertical	V1 gold-only	V8 whole haystack	V9 BM25 top-K	headroom (V1 − V9)
Arithmetic	0.94	0.84	0.32	+0.62
Forgetting	1.00	1.00	0.57	+0.43
Episodic	0.96	1.00	0.60	+0.36
Prospective	0.98	0.96	0.68	+0.30
Bitemporal	1.00	0.98	0.80	+0.20
Temporal	1.00	1.00	0.82	+0.18
WorkingMemory	1.00	1.00	0.88	+0.12

Every vertical has substantial retrieval headroom. V1 − V9 is the headroom number; V1 − V8 keeps its narrow reading as an interference cost and is documented as not a headroom number.

TypedMemEval-Bitemporal (60 questions)

Valid time (when a fact was true) against transaction time (when the record learned it). They diverge only after a retroactive correction, and a single-clock store cannot represent the difference — so its ceiling here is structural, not a matter of retrieval quality.

V1 60/60 · pair-flip 30/30 · V2 60/60 · V3 60/60. Pair-flip 30/30 means every pair's two clocks give different answers, so answering the valid-time question when asked the transaction-time one is wrong, not imprecise.

A design prediction the probe refuted: we expected a large interference cost and measured 59/60. That is a better property — V1 ≈ V8 ≈ 1.0 means the corpus holds neither reasoning ambiguity nor retrieval difficulty, so a system failing the transaction arm can only be failing because its store cannot represent when it learned a thing.

TypedMemEval-Temporal (50 questions)

The order events occurred, against the order they were mentioned. Sessions narrate retrospectively and anchor each event to another by a stated relation, so the timestamps record when a thing was said rather than when it happened — a system that sorts by date gets a confident, checkable, wrong answer.

V1 50/50 · V3 30/30 · V8 50/50. Zero questions are answerable by date-sorting and no answer contains a digit, both enforced by generator checks rather than review.

Also

Episodic's interference cost is −0.04: two participant-attribution questions fail on gold alone and succeed on the whole haystack, so V1 is not a strict ceiling for attribution shapes.

Warning

Arithmetic's difficulty bands are inverted. V8 by band reads 0.33 / 0.76 / 1.00 / 1.00 / 1.00 — the band labelled easiest is where the answer model fails two questions in three. Do not cite them.

v0.25.0-beta — TypedMemEval corpus revision v5

TypedMemEval corpus revision v5

v1 through v4 were all separable. Do not cite them. Corpus ids are agenteval-typedmemeval-<vertical>-v5.

This is the first TypedMemEval release where both instruments cleared the corpora before the tag — ours and the consuming project's independent acceptance probe. The previous release did not wait, and their probe failed it within hours.

What was wrong with v4

Constructions that only gold ever receives, found by hand rather than trusted from a statistic:

Vertical	Phrase	v4	v5
prospective	while it lasts	12 gold / 0 distractors	phrase gone
forgetting	for the record	15 / 0	5 / 111
forgetting	still the same	15 / 0	3 / 53
workingmemory	since the	20 / 0	20 / 996
workingmemory	the winter	15 / 0	15 / 177
arithmetic	today	117 / 31	116 / 199

v4's central fix made filler state first-person facts in gold's construction — and reached only the statement verb. The acknowledgement, the temporal clause, the retention marker and the reminder frame all stayed gold-only.

The gate had a bypass, which is why it passed v4

role_sequence, gold_marker_ngram and boilerplate_ngram were scored for AUC outside the per-session loop, and so were never given the distribution test the other 36 features got. Fixing that catches three of the four findings on the existing rule (z = 76, 2.7, 6.1).

role_sequence is the sharpest case: it was added one revision earlier because the distribution rule is what catches role order, and it was added on the code path that skips that rule. It passed only because the position_N_is_* features go through the loop and did the work.

Phrase exclusivity is now its own test, because no AUC variant expresses it — a phrase recurring in ≥20% of questions that reaches zero distractor sessions is refused. Forgetting escapes every AUC variant and the distribution rule, because G=2 caps a within-question AUC at 0.75 when one of two gold sessions carries the marker.

Two false positives, caught before acting on them

• The screen invented phrases. N-grams were built from a flat token stream and crossed sentence and bracket boundaries. "near enough also" measured 21 Episodic gold sessions against 0 distractors — a perfect tell that does not exist; the text reads …(or near enough). (Also on my mind:. Acting on it would have meant regenerating a corpus that was already correct.
• Answer content is not a leak. Gold contains its own answer. But a plain answer exemption is self-cancelling rather than merely imprecise, because the answer paraphrases gold's construction — it dropped "since the" in exactly the 20 questions where it leaks. A gram is exempt only if some token in it is named by the question or answer and is rare corpus-wide.

The corpus fix: share the class, diverge the instance

Filler states the same kind of durable fact as gold, in the same construction, about entities no question asks about — so it cannot become alternative evidence. Parity banks are asserted disjoint from the real ones at import; the first run of that assertion caught "window cleaner" colliding with the fact noun "cleaner".

The shared cause was the echo pass: a distractor's clause echoes its own question's keywords (that is the calibration mechanism), while gold's echoed other questions' words, because echoing the query into gold busts the calibration ceiling. So foreign vocabulary appeared only in gold. The calibration gate refused three attempted fixes — a second clause (length and punctuation to 3.7–4.8 sd), merged into one clause (punctuation density 0.761), and swapped in place (Prospective saturated at 0.980 coverage). What works costs nothing on either axis: gold borrows from its own question's distractors.

Probes, re-run in full

Vertical	q	SHA-256 (first 12)	coverage	V7 worst	V1	pair	V2	V3	V6	exclusive
Prospective	50	6ddd3e9bb594	0.700	0.732	49/50	18/19	50/50	45/45	n/a	0
Episodic	50	2c6000a6912e	0.682	0.721	48/50	—	50/50	50/50	n/a	0
Arithmetic	50	efa9add22a2a	0.636	0.661	47/50	—	50/50	50/50	50/50	0
WorkingMemory	60	43df4ed5572d	0.867	0.635	60/60	—	60/60	60/60	n/a	0
Forgetting	50	ba759097b9bd	0.670	0.671	35/35	15/15	35/35	35/35	20/35	0

Prospective's V3 improved 37/37 → 45/45: eight more questions became decidable, because filler now carries the constructions the ablation probe needs to tell evidence from framing. Two V1 slips are named rather than absorbed — tme-pro-045, and arithmetic 48 → 47. V1 is oracle answerability, measured on gold sessions alone, so class parity cannot be the cause; V2 holding at 50/50 across all five is the positive evidence that non-inferability survived it.

949/949 tests on net8.0, net9.0 and net10.0.

Difficulty bands: the validator was wrong too

Regeneration made three verticals "validate", and all three were noise — the rule compared the first band with the last, which one high first band satisfies. Prospective read 1.00 / 0.67 / 0.80 / 0.70 / 0.79 (rank correlation +0.40, the opposite of a gradient) with n = 1 in the deciding band, and Forgetting's second band was 0.00 from a single question.

The rule now requires a rank-correlation trend and at least four bands at n ≥ 3, since three points falling in order happens one time in six by chance. Where rho cannot be computed the report says n/a rather than 0.00.

WorkingMemory is the only validated ladder (rho −0.78, oracle flat at 1.00 across all five bands). Arithmetic's oracle confound widened — band 1 reads 0.67 against 1.00 above, spread 0.17 → 0.33 — so it ships declared, not claimed, and fixing it needs a generation change rather than a threshold.

Important

0.22.0-beta, 0.23.0-beta and 0.24.0-beta all ship separable TypedMemEval corpora and are marked do-not-baseline. They stay listed because nothing outside the project consumed them. Pin an explicit version rather than a floating range.