v0.19.0-beta — the honest-measurement release

The honest-measurement release. Three independent pieces of work, and none of them changes a default — every new option is opt-in, so sealed benchmark bases stay comparable.

Red-team reports that state their own uncertainty

A scan now reports how confident it is, and stops flattering an agent that simply refuses everything.

• Wilson score intervals on the headline score. Wilson rather than Wald, because Wald degenerates to a zero-width interval at 0 and 1 — it claims total certainty exactly where the sample is emptiest.
• A benign-control corpus — 18 probes across 7 classes that use attack vocabulary in legitimate contexts. Resisted on one of these is a false positive, not a success. A test enforces that every probe shares vocabulary with the hostile corpus, so the control set cannot quietly drift into being trivially separable and flattering.
• False-positive rate, bypass-class breakdown (which class of defence failed, not just how many got through), probe label source (canary vs oracle vs judge, so evidence tiers can't be conflated), and report redaction to keep attack payloads out of exported artefacts.

SARIF fix: inconclusive probes were exported as note results, which reads as a low-severity finding. They are now kind: "open" with level: "none". SARIF 2.1.0 defines "open" as "the specified rule was evaluated, and the tool concluded that there was insufficient information to decide whether a problem exists" — which is exactly what Inconclusive means. §3.27.10 requires level: "none" when kind is not "fail".

Microsoft Agent Framework 1.13.0 → 1.17.0

Unlike the 1.12 → 1.13 bump, this one required source changes:

1. AgentHarnessOptions.DisableFileAccess was removed — file access is now opt-in via FileAccessStore.
2. ToolApprovalAgentOptions.AutoApprovalRules now takes a ToolAutoApprovalRuleContext (a strict superset of the old argument, unwrapped non-lossily).
3. UseToolApproval / ToolApprovalAgentOptions graduated from [Experimental("MAAI001")] to stable in 1.14.0 — so AgentEvalToolApprovalExtensions is no longer [Experimental("AEGK001")]. That marker existed only because the interop rode an evaluation-only MAF API.

Also Microsoft.Extensions.AI 10.6.0 → 10.7.0 (the floor 1.17.0 requires, pinned to exactly that floor to keep the upgrade one variable), and Microsoft.Agents.AI.Harness reaches its first stable release.

A LongMemEval judge that cannot produce an unparseable verdict

Driven by a downstream consumer losing roughly one paired 50-question run per run to a verdict that could not be parsed — systematically, on the same question across separate runs.

Root cause: the free-text parser recovered the verdict from the response's leading token, then vetoed it if the word "no" appeared anywhere later. That veto is right for genuinely self-contradicting output like "yes, but no". It also fires on a judge that answered yes and then wrote ordinary reasoning prose — "there is no discrepancy". Deterministic per input, which is why an affected question failed on every run rather than intermittently.

• JudgeVerdictProtocol.StructuredJson (default remains FreeText) — asks for a JSON object with a closed verdict field and a separate reasoning field, so reasoning can never contaminate the verdict. Uses the provider's structured-output facility, degrading through JSON mode to an unconstrained call. Anything unusable becomes Invalid with a diagnostic code — never an exception, never a silent No, never a guess. Invalid stays distinct from No, so RetryThenInconclusive and RetryThenIncorrect keep meaning different things.
• RetainRawJudgeResponse (default false) — retains the bounded raw response independently of JudgeEvidenceMode, so a short explanation can still be rendered while the full text stays available. This is what makes a judge that was wrong distinguishable from a wrapper that couldn't parse.
• JudgeDecompositionMode.PerPredicate (default None) — judges each gold-answer claim separately and combines under an explicit, recorded PredicateCombinationRule. Per-predicate outcomes are reported, because knowing which claim failed is the only thing decomposition adds.
• JudgeAgreementHarness — runs a judge repeatedly over identical inputs and reports its self-disagreement rate, so judge noise is separable from a real change in the system under test. An empty run reports a null rate, not 0.

Three hazards were found by running the extractor over the real datasets rather than reasoning about it, and each is now a test carrying the verbatim gold answer: alternatives ("7 days. 8 days (including the last day) is also acceptable." offers two acceptable answers, not two required facts), enumerated lists (splitting on the ordinals leaves fragments ending in a dangling number), and abstention questions (whose judge asks whether the model recognised the question as unanswerable — a different question entirely). All three are judged whole.

Measured over both shipped 500-question datasets: 6 answers (1.20%) decompose, a 1.0140× judge-call multiplier, at most 3 predicates on any one question. Decomposition barely engages on LongMemEval because its gold answers are overwhelmingly single facts.

The judge is still not deterministic by default, and this release does not change that. JudgeTemperature remains null deliberately — an explicit temperature is rejected by reasoning-model deployments. Set JudgeTemperature = 0 for a deterministic judge.

Upgrading

The public surface moves additively only: four ExternalBenchmarkOptions properties, three on ExternalJudgmentResult and four on QuestionResult, three new enums, JudgePredicateResult, the agreement-harness types, and an IExternalBenchmarkJudge options overload supplied as a default interface method so existing implementers keep compiling unchanged.

Every new field is WhenWritingNull and every new option defaults off, so a run recorded with default options serializes to exactly the field set 0.18.0-beta produced — pinned by a byte-for-byte test.

Validation

Release configuration, all three target frameworks: AgentEval.Tests 9,197 passed (net8.0, net9.0) and 9,415 passed (net10.0); AgentEval.Memory.Tests 680 passed on each. Build clean.