[Spec Change] Resuming interrupted runs and merging partial ResultSets

[adhabnr-ux]

Flagged in spec/CRITIQUE.md #4 as a remaining concern after the streaming fix shipped: "the spec doesn't define a standard way to resume an interrupted run or merge partial result sets. This should be added in v1.1."

What already works: test_cases_file for external JSONL suites, and result sets can be written incrementally since each Result is a self-contained object — evalport run's own --output <path> flag already writes results as test cases complete, so a long run's progress survives a crash at the file level.

What's missing at the spec level: there's no defined way to say "this ResultSet is partial, resume from here" or "merge these three partial ResultSets from a run that was interrupted twice into one canonical ResultSet." Concretely:

1. A ResultSet-level field (metadata.openeval.partial: true plus something identifying which test_case_ids are represented?) so a consumer can tell a partial result set from a complete one without inferring it from count mismatches against the suite.
2. A defined merge semantic when two partial ResultSets overlap on a test_case_id — last-write-wins by timestamp? Reject the merge and require the caller to dedupe first? This matters most for --parallel runs that get interrupted mid-batch.
3. Whether "resume" means "the runner re-reads a partial ResultSet and skips already-completed test_case_ids" (a CLI/runner behavior, arguably out of spec scope) vs. "the spec defines a canonical merge function" (in scope, belongs in the SDKs next to validate_result_set()).

Where to start if you pick this up: cli/src/run.ts (or the Python equivalent) already has the incremental-write logic for --output; the natural place for a --resume <path> flag is right next to it, and the merge semantics it needs would be exactly what this thread should settle first.

[adhabnr-ux]

Pulled up spec/schemas/resultset.json before writing this, since the merge-semantics question in point 2 can't really be answered without looking at what's actually there today — and it surfaces something concrete: there's currently no per-result timestamp to merge on.

ResultSet has started_at/completed_at at the top level (both date-time, completed_at optional), but each entry in results[] (test_case_id, actual_output, grader_results, passed, duration_ms, error, metadata — schema's full required list is just ["test_case_id", "grader_results", "passed"]) has no timestamp of its own. So "last-write-wins by timestamp" (the option floated in the original post) isn't actually implementable against the current schema without adding a field — there's nothing on an individual Result to compare. duration_ms tells you how long a test case took, not when it finished.

Concretely, I'd propose:

1. Add an optional completed_at to each item in results[], mirroring the field that already exists at the ResultSet level (date-time format, not required — keeps every existing ResultSet producer still valid). This is the minimum needed for a real merge tiebreaker to exist at all, and it's a backward-compatible addition (optional field, additionalProperties: false on the result object means this needs a schema change, but not a breaking one for consumers that just read the fields they know about... worth flagging that additionalProperties: false on results[].items means any implementer's merge tool that wants to stash resume bookkeeping in a similar ad-hoc way is currently blocked from doing it per-result — it has to go through metadata, which is additionalProperties: true. Also relevant to point 1: I lean the same "documentation not spec" way as I did on the adapter-packaging thread, but this one specific timestamp field feels different — it's not a policy question, it's "the data needed to answer point 2 doesn't exist yet."

2. metadata.openeval.partial: true at the ResultSet level — agreeing with the original post's instinct — is completely uncontroversial to add today with zero schema change, since ResultSet.metadata is already additionalProperties: true. This could genuinely ship as documentation-only (a reserved-key convention, the same way metadata.openeval.cost presumably works) without touching resultset.json at all, and would immediately be useful for tooling that wants to distinguish "this ResultSet is complete" from "count mismatch against the suite, might be partial, might just be a filtered subset" — which is exactly the ambiguity the original post flags.

3. On merge semantics once (1) lands: last-write-wins by results[].completed_at when present, but what happens when it's absent (an older or non-conforming ResultSet) needs a defined fallback — my instinct is "reject the merge and require the caller to specify precedence explicitly" rather than silently guessing, since silently picking a default order for untimestamped partials is exactly the kind of thing that produces confidently-wrong merged results with no way to detect it after the fact (same failure shape as the Rule 6 score: null discussion in [Spec Change] Formal conformance test suite for runners #9, just at the ResultSet level instead of the GraderResult level).

4. On point 3 (CLI resume vs. spec-defined merge function) — I haven't actually read cli/src/run.ts closely enough to say what it does today with any confidence, so I won't claim it does or doesn't already handle incremental writes the way the original post describes; that's worth an implementer actually checking before this thread commits to an approach, rather than either of us asserting it from memory.

Happy to prototype the schema diff for (1) if there's rough consensus it's the right minimal addition — it's small (one optional field, one format) and unblocks the rest of this thread rather than needing everything settled first.

[adhabnr-ux]

Shipped in 1.0.0-rc.3, landing close to exactly as you scoped it:

1. Result.completed_at (optional, date-time) added to spec/schemas/resultset.json + the schema/ mirror — commit 550191e — plus both SDKs' Result types (the Python dataclass, the TypeScript interface): Python commit, TypeScript commit. Backward-compatible (optional field); both SDK packages bumped MINOR, 1.1.0 → 1.2.0, not MAJOR.
2. metadata.openeval.partial: true shipped as a schema-change-free convention, per your point 2 — no resultset.json change needed since ResultSet.metadata was already additionalProperties: true.
3. Merge semantics: last-write-wins by results[].completed_at when both sides have it for a given test_case_id; if either side's colliding result is missing the timestamp, the merge is rejected rather than guessed — your point 3 instinct, adopted as-is. Documented in SPEC.md's Extension Mechanism → Resumable Runs & Partial ResultSets.

Point 4 (an actual CLI --resume flag, and whether cli/src/run.ts already does incremental writes the way the original post describes) is not addressed by this change — deliberately left as a runner-implementation question, out of scope for a data-format spec change. If anyone wants to build --resume next to the existing --output incremental-write logic, this is what it would merge on.

Fixture: partial_resultset_resumable_run.json — two results with distinct completed_at values and metadata.openeval.partial: true set, checked against both the hand-rolled validator and the raw JSON Schema. New regression tests in test_schema_consistency.py and schema-consistency.test.ts cover the new field in both languages.

You'd offered to prototype the schema diff — ended up just writing it since it was small and unblocked the rest, but it's exactly the one-field addition you proposed. The point that mattered most in your comment: there was no per-result timestamp to merge on at all before this, so "last-write-wins by timestamp" genuinely wasn't implementable until this landed — that's the right thing to have caught before answering point 2.

Marking the core RFC (points 1–3) resolved; leaving this open for point 4 (CLI resume behavior) if anyone wants to pursue it.