v0.20.0-beta — measurable samples and run provenance

The measurable-sample release. LongMemEval learns to draw the sample you asked for, report the sample it actually drew, and prove that two runs were comparable.

No default changes what a run selects or how it is scored. The v0.19.0-beta sampling path is pinned byte-for-byte by four golden samples generated from the released loader.

The finding behind the release

A downstream consumer reported that across 52 recorded runs, not one abstention question ever ran. Measured against the real dataset rather than reasoned about:

• Both shipped datasets hold 30 abstention questions out of 500.
• Abstention is orthogonal to question_type — an abstention question carries the same type as an ordinary one, distinguished only by an _abs suffix on its id. So stratifying across types says nothing about abstention coverage.
• The shipped Subset preset (MaxQuestions = 50, RandomSeed = 42) draws zero of them.
• With a varying seed, 49 of 52 seeds would have included at least one.

The cause was not stratification. SubsetRandomSeed is a fixed constant, so all 52 runs re-drew the same 50 questions — and that one sample happens to contain none. A fixed seed makes runs reproducible, not independent.

Added

Sampling composition

• IncludeQuestionTypes — spend a budget on named types, stratified within them, reproducible under the seed. Null or empty reproduces historical selection exactly.
• AbstentionPolicy (AsSampled / Exclude / Only / TargetProportion) and AbstentionTargetProportion.

Realised reporting

• ExternalBenchmarkResult.Composition — counts by question type and abstention flag, computed from QuestionResults, the same list the accuracy denominators come from, so the two cannot disagree. A target the pool cannot fill is left short rather than topped up, and the shortfall is visible.
• QuestionResult.IsAbstention, falling back to the _abs convention so older stored results report it correctly.

Judge call accounting

• JudgePrimaryLlmCallCount, JudgeRetryLlmCallCount, JudgeAttemptsUsed, and TotalJudgeRetryLlmCalls. JudgeLlmCallCount counts retries too, so a validity gate asserting an exact provider-call count rejects runs whose only anomaly was an internal retry. Total always equals primary + retry; response-format fallback calls count as primary.

Provenance

• RunProvenanceMode (None / PromptsOnly / Full) and ExternalBenchmarkResult.Provenance — SHA-256 over the judge prompt templates, the dataset file, and the ordered selected ids.
• system_fingerprint capture on QuestionResult.JudgeSystemFingerprint / AgentSystemFingerprint and de-duplicated on the result. ChatResponse in MEAI 10.7.0 has no such property, so it is recovered from AdditionalProperties then by reflection over RawRepresentation. Absence is null, never a placeholder.

History scaffolding

• SyntheticTurnMarker prefixes every turn AgentEval synthesises during structured injection, and the default strings are now public constants on LongMemEvalHistoryFormatter.

Changed

ExternalJudgmentResult and QuestionResult gain always-present properties, so their serialized shape differs from v0.19.0-beta additively. System.Text.Json ignores unknown properties, so readers keep working; a consumer asserting an exact property set does not. Call accounting is deliberately not opt-in — a counter that appears only when requested is useless to a validity gate that runs on every result.

Fixed

• docs/memory-evaluation.md listed abstention as one of the six question types. It is not, and saying so implies a coverage guarantee stratification cannot provide.
• PreserveSessionBoundaries documented as structured-injection only: HistoryInjectionMode defaults to TextBlob, which never reads it, so setting it to false on otherwise-default options changed nothing with no way to notice. Behaviour is deliberately unchanged; the silence about it was fixed.
• Validate() now rejects AbstentionTargetProportion set under a policy that would ignore it.

Tests

AgentEval.Memory.Tests 680 → 760 on each of net8.0/net9.0/net10.0. AgentEval.Tests 9197 (net8/9) / 9415 (net10), unchanged and green.

Sampling tests run against a metadata-only fixture — the 500 real question ids and types in dataset order, with no question text or haystack content — so they work in CI where the dataset is gitignored.

Full changelog: https://github.com/AgentEvalHQ/AgentEval/blob/main/CHANGELOG.md