v0.24.0-beta — TypedMemEval corpus revision v4

TypedMemEval corpus revision v4

v1, v2 and v3 were all separable. Do not cite them. Corpus ids are agenteval-typedmemeval-<vertical>-v4.

This revision exists because an independent acceptance probe run by a consuming project found gold sessions identifiable without reading a single word — from the sequence of turn roles alone. Gold ran u|a|a|u|a in 27 of 50 Episodic questions while every distractor ran u|a|u|a|a, and on the attribution shape the sequence gave the speaker 15/15, leaking the answer rather than its location.

Why the previous fix caused it

Equalising per-role turn counts drove turn_count and both per-role counts to exactly 0.5000. That is a successful equalisation, and it is also what hid the residual: appending to a tail cannot repair a prefix, so the count tell was removed and the order tell locked in by the same pass. Three axes at exactly chance is the signature to distrust, not to trust.

Sessions are now aligned onto a shortest-common-supersequence of the role sequences in their question — inserting empty turns, never reordering. The binding condition is per question, the set of role sequences present in gold equals the set present in distractors, and it holds for all five verticals with zero gold-only sequences.

Removing the obvious conditional was not sufficient: on its own it introduced a gold-only sequence in Prospective across 12 questions, a vertical that had been clean. Treating it as a family-wide invariant rather than a one-vertical patch is what caught that.

The gate, and which half of it does the work

Role order is now measured — role_sequence and position_{0..3}_is_{user,assistant} — in Python and re-derived independently in C#. Adding the features was not enough:

pooled role_sequence AUC        0.6152   <- UNDER the 0.75 threshold. Would have PASSED.
perfectly-separated questions   27 observed vs 3.48 expected, z = 13.3   <- refuses

The distribution rule catches this, not the AUC bar. A CI self-test now rebuilds the defect and asserts refusal, so a future simplification to "the AUC is fine" fails in our CI rather than in someone's acceptance probe.

The chance model was also corrected. A question's chance of a folded AUC of exactly 1 is 2/C(n, g) — 22% at H=8, 3% at H=60 — so comparing a corpus-wide share against a corpus-wide mean chance rate penalised WorkingMemory for varying H as its independent variable. It now sums per-question chances and tests the excess as a Poisson-binomial z (≥ 2.5, with the 0.20 share floor kept).

Two further defects, both present since v1

• Padding double-counted new vocabulary, scoring a candidate's types from its own distinct words while ignoring overlap with text already there. The error grows with how much padding a session needs — exactly what padding neutralises — so gold finished closest to target. Prospective's type/token ratio separated perfectly in 24% of questions against 12% chance, on a pooled AUC of 0.602.
• Padding had no lever to add punctuation without spending a sentence. WorkingMemory read 14 separations against 6.1 expected. Fixed with punctuation-carrying tails, deliberately without em dashes — that glyph was the original v1 tell.

What the corpus now admits about itself

• Exactly one difficulty ladder validates: WorkingMemory (retriever 0.92 → 0.50, oracle flat at 1.00). Episodic came off as flat; Arithmetic came off as confounded — the steepest retriever gradient in the family, but its two easiest bands read 0.83/0.94 on the oracle, because the duration shape lives at low input counts and is where the answer model struggles. Part of that clean gradient was the oracle failing, not retrieval getting harder. Checked in CI now, both halves.
• 1387 of 30761 turns ship empty, inserted by the alignment. Declared rather than fixed: three attempted fixes each made the corpus measurably worse (3.1 sd, 2.7 sd, and a no-op), and blank-turn count separates below its chance rate in all five verticals. structure.empty_turns records the count, cause and separates_gold: false.
• V6 records say when they do not apply instead of publishing passed: 0 for a probe that never ran, and exemption reasons are per-feature — so WorkingMemory's position_in_haystack: 1.000 is explained by the exemption that waives it.

Probes, re-run in full against the shipped bytes

Vertical	q	coverage	V7 worst	V1	pair-flip	V2	V3	V6
Prospective	50	0.820	0.712	50/50	19/19	50/50	37/37	n/a
Episodic	50	0.658	0.680	48/50	—	50/50	50/50	n/a
Arithmetic	50	0.655	0.737	48/50	—	50/50	50/50	50/50
WorkingMemory	60	0.767	0.654	60/60	—	60/60	60/60	n/a
Forgetting	50	0.730	0.694	35/35	15/15	35/35	35/35	20/35

949/949 tests pass on net8.0, net9.0 and net10.0.

Provenance

AgentEvalVersion is fixed. Every release from 0.16.0-beta through 0.23.0-beta stamped result provenance with 0.16.0-beta, because packing passed only -p:PackageVersion — which moves the nupkg version and nothing else, while AssemblyVersion, FileVersion and InformationalVersion all derive from -p:Version. A gate between pack and push now refuses to publish assemblies that do not carry the tag. Found by a consuming project, not in-house.

Important

0.22.0-beta and 0.23.0-beta ship separable corpora and should not be used for TypedMemEval. Pin an explicit version rather than a floating range — "latest prerelease" can resolve to a release with no TypedMemEval at all.