RFC: distinguishing FAILED from UNVERIFIED — a Result can fail without proving it failed

[adhabnr-ux]

Where this comes from

Raised by @soul-sol in a semantic-kernel discussion about a portable EvalPort adapter for multi-agent orchestrations, responding to a point about how badly str(final_output) collapses multi-agent runs. Quoting the part that matters here directly, because it's already right:

One last distinction I'd bake into ResultSet if you can: FAILED ("provably didn't happen", safe to re-run) versus UNVERIFIED ("may have happened, couldn't judge"). Collapsing them into one failure bucket means a suite with retries re-executes side effects, and it also hides the most interesting comparison between runtimes — how often each leaves you unable to tell.

The concrete case: an orchestration leaves a worker agent in a terminal state of "alive, no output, no question" — not finished, not asking for input, just unresolved. That's not the same thing as an orchestration that ran to completion and produced a wrong answer, and it's not the same thing as a harness-level execution failure either.

Why the existing schema can't say this today

A Result has exactly two ways to be non-passing, and neither is "I don't know":

1. grader_results[] + passed: false — a grader looked at the output and judged it wrong. This asserts the run was judged and failed the judgment.
2. error (timeout | provider_error | runner_error) — the harness itself failed to produce a usable result. This asserts nothing ran to completion.

"The run executed, produced some terminal state, and no grader can honestly say whether it succeeded" is neither of those. Forcing it into passed: false claims a judgment that was never made. Forcing it into error claims an execution failure that didn't happen — the orchestration didn't crash, it just ended somewhere ambiguous. Today's producers are stuck picking the less-wrong lie.

This is the same class of gap as Discussion #47 (Result.constraint_violations, from a different maintainer's benchmark, currently in its own comment period with a reference PR at #48): a real distinction that the existing two-axis model (grader judgment vs. harness error) structurally cannot express, surfaced by someone who actually ran evals against real fixtures and hit the wall rather than by guessing at the schema in the abstract.

Two concrete consequences of collapsing them

Both from soul-sol's own writeup, and both are consumer-facing problems, not just modeling aesthetics:

• Retry safety. A suite with retries that treats UNVERIFIED as FAILED will re-execute the case — including any side effects the first, ambiguous attempt already caused. A suite that can tell the two apart can retry an UNVERIFIED case (nothing was proven, safe to redo) while refusing to blindly retry a FAILED one (side effects may have already landed).
• The actually interesting cross-runtime metric. When comparing two orchestrations (the entire motivating use case of the semantic-kernel thread), "which one more often leaves you unable to tell" is a real, measurable difference between them — and it's invisible if both outcomes get bucketed as the same kind of failure.

What this is not (yet) a proposal for

I don't have a schema shape I'm confident in — this Discussion is the "does this gap exist and is it worth solving in EvalPort specifically" step, per this project's own governance process (two weeks of comment before anything gets implemented, the same path #45 and #47 went through). Two rough directions, neither committed to:

• A closed-ish enum on Result distinguishing at minimum failed / unverified / (implicitly) passed, sitting alongside the existing passed: bool rather than replacing it — passed stays the grader-judgment axis, this would be a new, separate "was a judgment even possible" axis.
• Folding it into error as a fourth type (e.g. error.type: "unverifiable") — cheaper structurally, but risks the same denominator-exclusion problem RFC: first-class hard constraints (`Result.constraint_violations`) — distinct from graders and errors #47 already identified for error: consumers reasonably drop error rows from denominators, and an UNVERIFIED result is emphatically not something that should silently vanish from a report the way a harness crash should.

soul-sol's own three-state terminal taxonomy for multi-agent workers specifically (finished / needs-input / unknown, recorded by the launcher rather than scraped from the transcript) is a good concrete grounding case if this goes anywhere — it's a real, independently-shipped implementation (MIT, soul-sol/agent-watch), not a hypothetical.

Also worth surfacing separately from the schema question, since it's good practice regardless of what happens here: soul-sol's writeup names three ways an eval suite can pass while being wrong (positive-only fixtures with no case that must fail, a suite that only proves the mapping executes rather than that the mapping is right, a threshold that silently discards valid data). None of that needs a spec change — it's a testing-discipline note for anyone (including me) writing conformance fixtures for an EvalPort adapter, and I'm folding it into how I build the next one.

@soul-sol — tagging you since this is your framing and your worked example; corrections welcome before this goes further, especially if either rough direction above already looks wrong from where you're sitting.

[adhabnr-ux]

Reference implementation is up: #83 (draft, DO NOT MERGE)

This thread has had no comments since it opened on 2026-09-11, so I built the concrete version to give it something to react to. #83 is a draft reference implementation, not a proposal to merge. It has the same status as #46 (for #45), #48 (for #47) and #66 (for #58 / #67). It covers the JSON Schema (plus its mirror), both SDKs, both schema-consistency suites, 5 conformance fixtures, and a PROPOSED subsection in spec/SPEC.md.

The comment period restarts today, 2026-09-27, and runs two weeks, through 2026-10-11. Nothing merges before then. If there is rough consensus after that, the implementation gets re-opened fresh against main.

What #83 proposes

One optional field, Result.verdict: "passed" | "failed" | "unverified".

• "passed" / "failed": the outcome was judged (verified passing / verified failing).
• "unverified": the run reached a terminal state that could not be judged. It may or may not have happened.
• Absent: not declared. Consumers read the Result exactly as they do today.

Rules, all three enforced by the JSON Schema itself (allOf + if/then/const), not only by the SDKs:

1. "passed" requires passed: true. "failed" and "unverified" require passed: false (VERDICT_PASSED_MISMATCH).
2. If error is present, verdict must be absent or "unverified" (VERDICT_ERROR_CONFLICT).
3. No other values, and explicit null is rejected (INVALID_VALUE).

Both SDKs check the full 28-case truth table against both validation paths. The raw schema and the hand-rolled validators agree on every case.

Consumer semantics in the spec text:

• "unverified" rows MUST stay in result-level denominators, unlike error rows. Consumers SHOULD report an unverified rate next to the pass rate.
• error + "unverified" is how a producer says "the harness timed out or lost the exit code, but the worker may already have acted, so count this row."
• Consumers SHOULD NOT automatically re-run an "unverified" row without first reconciling external state.

How it relates to what the spec already has

Validation Rule 6 already has a per-grader "not verified": score: null. verdict is the Result-level counterpart, not a duplicate. Neither can be derived from the other:

• @soul-sol's case has every grader scored (the final answer string matches), and the run is still unverified because a worker was left alive with no output and no question.
• A Result can have null-scored graders and still be established as failed by something outside the graders.

So "all scores null ⇒ unverified" is a SHOULD, not a validator rule.

Composition with the other open RFCs is documented but not implemented, because those fields are not on main:

• With RFC: first-class hard constraints (`Result.constraint_violations`) — distinct from graders and errors #47 (constraint_violations), an invalidating violation already forces passed: false, so verdict: "passed" is already impossible. The recommended verdict there is "failed", and PR DO NOT MERGE — reference implementation for Discussion #47 (Result.constraint_violations) #48's "error wins" rule is unchanged.
• With RFC: rate-based GraderResult denominators (trials/successes) — promoted from #58 #67 (trials/successes), there is no schema interaction.

Alternatives rejected

• A fourth error.type ("unverifiable"): error rows get dropped from denominators, which is exactly the inversion this RFC is about.
• A tri-state passed: a breaking change.
• A hard "all-null scores ⇒ unverified" rule: the deciding graders depend on the suite's aggregation strategy, which the validator never sees.
• A verdict_reason field: the terminal-state vocabulary is not settled (see below), and metadata covers it for now.
• summary.unverified: summary is producer-computed, the same call DO NOT MERGE — reference implementation for Discussion #47 (Result.constraint_violations) #48 made.
• Allowing error + "failed": starting strict, because relaxing later is backward compatible and tightening later is not.

A correction to my opening post

The "Retry safety" bullet above has it backwards. I wrote that an UNVERIFIED case is safe to redo and a FAILED one is not. @soul-sol's framing is the opposite, and it is the correct one: FAILED is "provably didn't happen, safe to re-run", and UNVERIFIED "may have happened", so blindly retrying it can re-execute side effects. #83's spec text follows soul-sol's framing.

What I found in agent-watch

I read soul-sol/agent-watch in full at a85a186. It does not contain a finished / needs-input / unknown enum. That taxonomy is from the semantic-kernel comment.

What the repo does implement is the launcher-records-the-exit-code half:

• worker_launch.sh writes the worker's $? to <name>.exit.
• worker_watch.sh reports RUNNING / DONE / FAILED / STALL. In its words, DONE is "the process finished and said nothing went wrong" (not "the work is good"), and STALL is "reserved for the cases where the outcome cannot be established at all".

One fixture uses the verbatim STALL line as an error + "unverified" example. Another encodes the "alive, no output, no question" worker.

Questions for @soul-sol

1. needs-input. Where should a participant that surfaced a question and stopped land: "unverified", "failed", or a fourth value? It did not complete, but it did not provably fail either.
2. Does "failed" promise retry safety? DO NOT MERGE — reference implementation for Discussion #49 (Result.verdict: failed vs unverified) #83 keeps verdict purely epistemic ("was the outcome established?") and does not promise that re-running a "failed" row is side-effect-free. A constraint violation (RFC: first-class hard constraints (`Result.constraint_violations`) — distinct from graders and errors #47) is a case where something did happen and the row is still "failed". Should "failed" carry your "provably didn't happen" guarantee instead, and if so, what should a verified failure that did have side effects be called?
3. error + verdict. DO NOT MERGE — reference implementation for Discussion #49 (Result.verdict: failed vs unverified) #83 allows only "unverified" alongside error. Is that right for your STALL case? And does agent-watch's preflight (NET_DISABLED / AUTH_FAILED, where the worker provably never started) argue for also allowing error + "failed"?
4. Per-participant terminal states. Should EvalPort reserve a metadata key for them? If so, should it use the comment's three states or agent-watch's four? DO NOT MERGE — reference implementation for Discussion #49 (Result.verdict: failed vs unverified) #83 leaves them producer-chosen for now.

Pushback on the field name, the rules, or the whole shape is welcome, and so are counter-examples from other runtimes.

[adhabnr-ux]

Status, 2026-09-30: still no outside comments, and one new data point

Still no comment from anyone outside the project (the comment period runs through 2026-10-11, and nothing merges before then). Two things to put on the table so the thread has more than my own writing in it.

A data point from building the Every Eval Ever exporter (examples/every-eval-ever/). EEE's score_details.score is a required number with no null, so Rule 6's per-grader "not verified" cannot be stored at all. The exporter has to leave nulls out of every mean and rate and write the counts into string-valued details. A consumer reading only score sees a mean over fewer items than the suite has, and nothing typed says so. The Croissant Tasks export has the same shape of problem: a cr:EvaluationResult is one metric and one value. That is evidence for the gap in this RFC from the downstream side, and it is also a caution: a Result-level verdict would hit the same wall in any numeric-only format unless the format has a place for a count of unverified rows. I'm not claiming either format's maintainers have a view; neither has reviewed these exports.

I've asked one outside group for its view. With the approval of the project's maintainer, I emailed the AAIF Accuracy & Reliability working group's list today with a short introduction and one question: does reliability and accuracy reporting need a first-class "not verified" outcome, and what should it be called. I stated that the project is small and has no production adopters. If anyone from that group or elsewhere lands here, the four questions in my 2026-09-27 comment are still the useful place to start, and so is a counter-example from a runtime where unverified and failed don't separate cleanly.

No change to the proposal or to the dates.

[adhabnr-ux]

Status, 2026-10-01: first outside data point, proposed narrowing, and the bar for landing

No change to the dates (comment period still runs through 2026-10-11). Three things for the thread.

1. A data point from a second runtime. Over on AgentEval#203, one of AgentEval's maintainers listed the result states its framework distinguishes: NotApplicable, NotMeasured, error and skipped, as well as pass and fail. He asked that an exporter carry that measurement state separately so "a judge outage doesn't turn into a fail (or a pass)". He wasn't commenting on this RFC and I'm not claiming he supports it. It is a case of another harness separating "judged and failed" from "couldn't be judged", and it is the kind of counter-example this thread asked for. It also raises a question #83 doesn't answer: is NotApplicable a fourth value, or just a skipped grader? My instinct is the latter, because nothing was sought. I've asked how AgentEval uses those states rather than guessing.

2. Proposed narrowing of #83, for objection.

• verdict stays purely epistemic. It answers "was an outcome established?", and nothing else. "failed" makes no retry-safety promise (a RFC: first-class hard constraints (`Result.constraint_violations`) — distinct from graders and errors #47 constraint violation is a verified failure where something did happen). "unverified" keeps its existing SHOULD NOT about blind re-runs.
• needs-input and per-participant terminal states are out of scope. They stay producer-chosen metadata. I'd rather not freeze a vocabulary on one runtime's three-state taxonomy, particularly since the agent-watch repo itself uses four states.
• error + "failed" stays rejected. Relaxing later is backward compatible and tightening later is not.

3. The bar for landing, and the alternative it has to beat. I don't want this to merge because a calendar date passed. My proposal is that #83 only goes forward if at least one implementer other than me produces a real fixture, or says in this thread that their system needs the field. If nobody does by 2026-10-11, I'll extend the period rather than merge, and be ready to close the RFC as a documented convention: Rule 6 per grader, plus a metadata.openeval.* key for a Result-level "unverified" marker, with no new schema field. That is the real competing option, and prior art points toward it: OpenTelemetry's gen_ai.evaluation.result already treats score.value and score.label as conditionally required, so an evaluation without a score is expressible there today. What a convention can't give you is a validator rule, and that is the argument for the field.

A related point for anyone designing against this: #47 and #49 both change what a non-clean Result means. Before either lands I'll write a one-page outcome-model table (passed / failed / error / unverified / constraint-violated) so the two don't drift into overlapping fields.

The four questions in my 2026-09-27 comment are still the useful place to start. Objections to any of the above are welcome, including "the field isn't worth it".