v0.22.0-beta — TypedMemEval

Warning

The corpora in this release are superseded. Do not cite v1 numbers.

A follow-up adversarial separability probe (V7) found that gold sessions in all five corpora were
identifiable by cheap shape features carrying no information about the question — capitalisation
density reached AUC 0.990 in Forgetting and session length 0.992 in WorkingMemory. A classifier
counting capital letters could find the evidence without reading it, which means these corpora do
not measure retrieval as claimed.

Corpus revision v2 fixes this and supersedes v1. Because the fix rewrites every session,
retrieval difficulty moved and a v1 score is not comparable with a v2 score. The library API,
the typed outcomes, the judge and the guards in this release are unaffected — only the corpora.

TypedMemEval — a new benchmark family that measures five memory mechanisms in isolation: prospective, episodic structure, arithmetic over memory, working-memory distance, and forgetting. Five embedded corpora, 248 authored questions.

Nothing in LongMemEval changes. Every 0.19–0.21 surface and the time-grounded corpus are untouched, and no default anywhere changes what a run selects, injects, or scores.

Citation rule. Cite results as "TypedMemEval-<Vertical> v1 (AgentEval)". TypedMemEval results are not LongMemEval results and must never be presented as, summed with, or averaged with LongMemEval numbers.

Why

LongMemEval-S cannot measure these mechanisms — prospective memory has no questions there at all, episodic structure has no list-order or attribution questions, derived answers are never separable from retrieval difficulty, and working-memory distance and forgetting have no question types. It is also saturated for a competent retrieval stack (gold coverage 0.965–0.980), which makes every retrieval-side mechanism invisible. The gap is in the dataset, and a benchmark's dataset is its identity — so the answer is a separate family, not more corpora under someone else's name.

What ships

• TypedMemEvalRunner with RunAsync and RunOracleAsync; the oracle arm reuses the shipped projector so a consumer's ceiling and this one are the same number from the same knobs.
• Typed outcomes, never one percentage — correct / wrong / abstained / missed / premature per vertical and per shape, always with n. Inconclusive and Unrun are their own members so a judge outage or a skipped question can never be absorbed into Wrong.
• Evidence attribution, named for what it is: reference-level presence, necessary but not sufficient. Missing telemetry reports Unobserved and is never guessed.
• TypedMemEvalRunSet bands over repeated runs and refuses to band runs that differ in corpus, judge fingerprint, configuration, drawn questions, or answer-sampling disposition.
• agenteval bench typedmemeval --vertical X --subject Y, which prints the typed vector with every denominator, publishes no pass threshold, and records an indeterminate verdict rather than asserting a gate outcome it never measured.

Validity, measured rather than claimed

Probe records ship in every corpus (reference deployment gpt-5.5, per-question outcomes in each .meta.json):

Vertical	V1 oracle	V1 pair-flip	V2	V3	V6
Prospective	46/50	16/19	50/50	49/50	—
Episodic	50/50	—	50/50	50/50	—
Arithmetic	47/50	—	50/50	50/50	50/50
WorkingMemory	48/48	—	48/48	48/48	—
Forgetting	34/35	14/15	35/35	35/35	20/20

The judge's five-way outcome is pinned by a 120-case hand-labelled calibration set at 0.983 agreement, bound to the judge-prompt fingerprint so editing a template fails the build until agreement is re-measured.

What the checks caught before release

Eight real defects, found by the probes, by self-review, and by a 22-finding pre-merge review — most of them in this release's own work:

• A calibration clause that marked gold in every corpus: gold carried it 0 times in 501 sessions against ~99% for distractors, so a one-line string filter isolated all gold evidence. Every structural check passed and all four probes were green, because none of them ask whether evidence is separable.
• A generator that computed due dates from a timestamp it then overwrote, so 38 questions named dates their conversations could not produce.
• A pair metric that measured the opposite of what it documented, with a test that scored 19/19 on a run containing none of the thing it claimed to detect.
• A run that measured nothing projecting as a hard 0.0, indistinguishable from a system that got everything wrong.
• A published coverage mean 30% composed of vacuous values, and a documented CI check that did not exist.

Full notes in CHANGELOG.md. Design of record: ADR-026. Getting started: docs/benchmarks/typedmemeval.