v0.25.0-beta — TypedMemEval corpus revision v5

TypedMemEval corpus revision v5

v1 through v4 were all separable. Do not cite them. Corpus ids are agenteval-typedmemeval-<vertical>-v5.

This is the first TypedMemEval release where both instruments cleared the corpora before the tag — ours and the consuming project's independent acceptance probe. The previous release did not wait, and their probe failed it within hours.

What was wrong with v4

Constructions that only gold ever receives, found by hand rather than trusted from a statistic:

Vertical	Phrase	v4	v5
prospective	while it lasts	12 gold / 0 distractors	phrase gone
forgetting	for the record	15 / 0	5 / 111
forgetting	still the same	15 / 0	3 / 53
workingmemory	since the	20 / 0	20 / 996
workingmemory	the winter	15 / 0	15 / 177
arithmetic	today	117 / 31	116 / 199

v4's central fix made filler state first-person facts in gold's construction — and reached only the statement verb. The acknowledgement, the temporal clause, the retention marker and the reminder frame all stayed gold-only.

The gate had a bypass, which is why it passed v4

role_sequence, gold_marker_ngram and boilerplate_ngram were scored for AUC outside the per-session loop, and so were never given the distribution test the other 36 features got. Fixing that catches three of the four findings on the existing rule (z = 76, 2.7, 6.1).

role_sequence is the sharpest case: it was added one revision earlier because the distribution rule is what catches role order, and it was added on the code path that skips that rule. It passed only because the position_N_is_* features go through the loop and did the work.

Phrase exclusivity is now its own test, because no AUC variant expresses it — a phrase recurring in ≥20% of questions that reaches zero distractor sessions is refused. Forgetting escapes every AUC variant and the distribution rule, because G=2 caps a within-question AUC at 0.75 when one of two gold sessions carries the marker.

Two false positives, caught before acting on them

• The screen invented phrases. N-grams were built from a flat token stream and crossed sentence and bracket boundaries. "near enough also" measured 21 Episodic gold sessions against 0 distractors — a perfect tell that does not exist; the text reads …(or near enough). (Also on my mind:. Acting on it would have meant regenerating a corpus that was already correct.
• Answer content is not a leak. Gold contains its own answer. But a plain answer exemption is self-cancelling rather than merely imprecise, because the answer paraphrases gold's construction — it dropped "since the" in exactly the 20 questions where it leaks. A gram is exempt only if some token in it is named by the question or answer and is rare corpus-wide.

The corpus fix: share the class, diverge the instance

Filler states the same kind of durable fact as gold, in the same construction, about entities no question asks about — so it cannot become alternative evidence. Parity banks are asserted disjoint from the real ones at import; the first run of that assertion caught "window cleaner" colliding with the fact noun "cleaner".

The shared cause was the echo pass: a distractor's clause echoes its own question's keywords (that is the calibration mechanism), while gold's echoed other questions' words, because echoing the query into gold busts the calibration ceiling. So foreign vocabulary appeared only in gold. The calibration gate refused three attempted fixes — a second clause (length and punctuation to 3.7–4.8 sd), merged into one clause (punctuation density 0.761), and swapped in place (Prospective saturated at 0.980 coverage). What works costs nothing on either axis: gold borrows from its own question's distractors.

Probes, re-run in full

Vertical	q	SHA-256 (first 12)	coverage	V7 worst	V1	pair	V2	V3	V6	exclusive
Prospective	50	6ddd3e9bb594	0.700	0.732	49/50	18/19	50/50	45/45	n/a	0
Episodic	50	2c6000a6912e	0.682	0.721	48/50	—	50/50	50/50	n/a	0
Arithmetic	50	efa9add22a2a	0.636	0.661	47/50	—	50/50	50/50	50/50	0
WorkingMemory	60	43df4ed5572d	0.867	0.635	60/60	—	60/60	60/60	n/a	0
Forgetting	50	ba759097b9bd	0.670	0.671	35/35	15/15	35/35	35/35	20/35	0

Prospective's V3 improved 37/37 → 45/45: eight more questions became decidable, because filler now carries the constructions the ablation probe needs to tell evidence from framing. Two V1 slips are named rather than absorbed — tme-pro-045, and arithmetic 48 → 47. V1 is oracle answerability, measured on gold sessions alone, so class parity cannot be the cause; V2 holding at 50/50 across all five is the positive evidence that non-inferability survived it.

949/949 tests on net8.0, net9.0 and net10.0.

Difficulty bands: the validator was wrong too

Regeneration made three verticals "validate", and all three were noise — the rule compared the first band with the last, which one high first band satisfies. Prospective read 1.00 / 0.67 / 0.80 / 0.70 / 0.79 (rank correlation +0.40, the opposite of a gradient) with n = 1 in the deciding band, and Forgetting's second band was 0.00 from a single question.

The rule now requires a rank-correlation trend and at least four bands at n ≥ 3, since three points falling in order happens one time in six by chance. Where rho cannot be computed the report says n/a rather than 0.00.

WorkingMemory is the only validated ladder (rho −0.78, oracle flat at 1.00 across all five bands). Arithmetic's oracle confound widened — band 1 reads 0.67 against 1.00 above, spread 0.17 → 0.33 — so it ships declared, not claimed, and fixing it needs a generation change rather than a threshold.

Important

0.22.0-beta, 0.23.0-beta and 0.24.0-beta all ship separable TypedMemEval corpora and are marked do-not-baseline. They stay listed because nothing outside the project consumed them. Pin an explicit version rather than a floating range.