RFC: repetition/attempt tracking in ResultSet (raised by AgentVerity via issue #20)

[adhabnr-ux]

Opening this to formalize a design question raised by @mrwersa in #20, following the same process Discussions #8–#11 went through (proposal here, comment period, then a PR implementing whatever the discussion converges on).

The gap

ResultSet.results[] today is documented as "one result per test case" but that's actually only a documentation convention, not an enforced constraint — I checked both enforcement points directly:

• spec/schemas/resultset.json's results array has no uniqueItems/key constraint on test_case_id.
• sdk/python/openeval/validate.py's validate_result_set() has no duplicate-id check — contrast with validate_suite(), which explicitly tracks ids=set() and raises DUPLICATE_ID for repeated test_case.id. That check exists on the suite side and has no counterpart on the result side.

So nothing today actually stops a producer from emitting several Results with the same test_case_id — it's just unaddressed: no field distinguishes why there are several, no join key relates them, and summary computation has no defined behavior when it happens. @mrwersa's #20 describes exactly the consequence: LangSmith's num_repetitions, Promptfoo's per-test repeats, and Inspect AI's epochs all produce this shape upstream, and it currently either collapses to one result or fragments into k ResultSets that only relate by convention — losing the ordered-pairing structure that stability/flip-rate analysis over repeated trials needs (AgentVerity's use case), and losing whether repeats ran in fresh or shared sessions (which changes what the data can support statistically).

The three directions from #20

1. Optional attempt field on Result (+ optional isolation field), pairing by (test_case_id, run_id, attempt).
2. Normative note: repeated runs are separate ResultSets sharing suite_id, plus a recommended join convention.
3. A profile:stability convention describing how producers should emit repeat data, likely metadata-based.

Where I lean, and why

Toward (1). Reasoning:

• Precedent: this is the same shape of problem Discussion [Spec Change] Resuming interrupted runs and merging partial ResultSets #10 (resumable runs) solved, and that landed as an additive, optional field (Result.completed_at) rather than restructuring the document — old consumers that don't know about the field keep working unchanged, new consumers that do get exact tie-breaking. An attempt field is the same move: optional, additive, additionalProperties: false on Result already means adding it is the only schema-level change required on the ResultSet side.
• (2) loses the join, not just makes it implicit. Multiple ResultSets sharing a suite_id is already legal today and already doesn't give you ordered same-case pairing across files — you'd need run_id plus an inferred ordering plus consistent started_at timestamps to reconstruct pairs after the fact, and nothing enforces producers do that consistently. It formalizes the status quo rather than fixing what Repetitions and trial isolation in ResultSet (needed for baseline qualification) #20 is describing as actually broken today.
• (3) is more general but currently underspecified — "how producers should emit repeat data" as a profile: convention is a fine home for aggregation/statistics methodology (e.g. how AgentVerity's flip-rate computation itself should be described so two tools agree on what "stable" means), but the underlying data model still needs some join key to exist first. I'd rather see (1) land as the data-model fix, and treat a stability-profile convention as a separate, later RFC that builds on top of it if there's appetite — bundling both into one change is a good way to stall on the harder, more opinionated half indefinitely.

Concretely, if (1) is where this converges, the shape I'd propose:

// Result, resultset.json
"attempt": {
  "type": "integer",
  "minimum": 1,
  "description": "1-indexed repetition number for this test_case_id within this run_id. Absent or 1 means a single-attempt result — existing single-shot ResultSets need no change."
},
"isolation": {
  "type": "string",
  "enum": ["fresh", "shared", "unknown"],
  "description": "Whether repeated attempts of this test_case_id ran in fresh sessions or a shared session/context. Required to be meaningful when attempt > 1; a producer that can't determine this should emit \"unknown\" rather than omit the field, since silent omission is indistinguishable from single-attempt data."
}

Plus a validate_result_set() change: when two Results share a test_case_id, require them to have distinct attempt values (default-absent treated as attempt: 1) — turning today's silent, unspecified duplicate-test_case_id gap into an actual, checked constraint instead of leaving it as an accidental permissiveness nobody relies on correctly.

Ask

@mrwersa — does that match what AgentVerity's importer actually needs, or does flip-rate/stability computation need something attempt+isolation alone doesn't give you (e.g. an explicit pairing across non-adjacent attempts, or a required minimum attempt count)? You mentioned being glad to contribute conformance fixtures if this lands — a fixture pair (multi-attempt ResultSet valid + one with a same-test_case_id/same-attempt collision that should be rejected) would be exactly the kind of case spec/conformance/ is built for, and I'd rather build the fixtures against real requirements than guess at them.

Leaving this open for comment before any spec/schema/SDK PR, per the same process #8 went through. Cross-referencing #20 so both threads point at each other.

[mrwersa]

Yes, attempt plus isolation covers what our stability computation needs. Three small refinements: keep isolation values as an open string rather than a closed enum, since fresh-session versus fresh-instance carry different statistical claims. Require one isolation value per repetition group. And define ascending attempt as observation order within test_case_id and run_id, with uniqueness on that pair, so adjacent attempts pair as disjoint trials without further schema.

On fixtures: happy to contribute the valid multi-attempt case plus the same-attempt collision rejection once field names settle in the PR.

[adhabnr-ux]

All three refinements make sense and I'll fold them in:

• Open string for isolation, not a closed enum — agreed, and this is the right call for a reason beyond just fresh-session-vs-fresh-instance: locking it to an enum now means every new isolation strategy some other framework invents later needs a spec PR just to be nameable, even before anyone argues about what it should imply statistically. An open string with "fresh"/"shared"/"unknown" documented as conventional-but-not-exhaustive values gets you the same interoperability for the common cases without that bottleneck.
• One isolation value per repetition group — makes sense as a validate_result_set()-level check: all Results sharing a (test_case_id, run_id) pair should carry the same isolation value (or all omit it). Mixed isolation within one group would mean the statistical claim you can make about the group is undefined, which defeats the point of tagging it at all.
• Ascending attempt as observation order within (test_case_id, run_id), uniqueness on that pair — this is a cleaner, more precise version of what I sketched, and it directly answers my own open question about pairing: uniqueness on (test_case_id, run_id, attempt) is exactly the join key, and "ascending = observation order" gives consumers a documented ordering guarantee for free instead of leaving them to infer it from created_at timestamps or array position.

Net effect on the schema sketch from the top of this thread: attempt and isolation stay as described, plus two new normative rules for validate_result_set(): (a) (test_case_id, run_id, attempt) must be unique when attempt is present — a real, checked constraint replacing today's silent gap — and (b) all Results sharing a (test_case_id, run_id) must agree on isolation (or all omit it).

I think this converges the design question — I'll leave this open through the rest of the two-week comment window per the usual process, but I'm ready to start the implementation PR (schema + both SDK validators + a conformance fixture pair) once that closes, or sooner if nothing else comes in. If you're still up for contributing the fixture pair once field names are locked in the PR, that'd be genuinely useful — a real second-party check on whether the described behavior actually matches, rather than me writing fixtures against my own spec text.

[adhabnr-ux]

Agreed on all three refinements — I'll keep isolation as an open string rather than a closed enum, require exactly one isolation value per repetition group, and define attempt ordering as observation order within (test_case_id, run_id) with uniqueness enforced on that pair.

I'll ping you once the field names are settled in the PR so you can add the valid multi-attempt fixture and the same-attempt-collision rejection case — that'll be a good test of whether the schema actually holds up under AgentVerity's stability computation.