feat(cli): coder-eval execute + detached grading via evaluate <run_dir>

[akshaylive]

Splits running from grading, so an external harness can own the verdict — and closes the loop so a run executed now can be graded later.

The motivating case is Harbor (Terminal-Bench 2.0), which builds its own container, calls coder-eval as the agent, and grades with its own tests/test.sh. Grading twice there would be worse than not grading at all: coder-eval's verdict would be reported alongside Harbor's without being the one that counts.

This is Part A, phases 1–5 of the Harbor interop plan (tmp/harborframework.md).

coder-eval execute  tasks/hello.yaml --run-dir ./r   # run, capture, score nothing
coder-eval evaluate ./r/default/hello/00             # supply the verdict later
coder-eval aggregate ./r                             # run.json now reports it

---

1. coder-eval execute

coder-eval run with the grading half removed. The agent runs and the full trajectory lands in task.json as usual, but no criterion is checked, weighted_score stays None, and the row finalizes as the new FinalStatus.NOT_GRADED.

NOT_GRADED is a fourth reporting category

category == "ungraded" — not a fold into the existing three. Folding into failed would depress every pass rate, into succeeded would invent verdicts, into error would report a healthy run as broken.

Ungraded rows leave both sides of every rate: RunSummary / VariantAggregate pass_rate and error_share now divide by tasks_graded (tasks_run - tasks_not_graded), identical to tasks_run for any graded run. tasks_not_graded is part of the sum-to-tasks_run invariant, not a tasks_failed sub-counter, and is defaulted so existing run.json / experiment.json still parse.

weighted_score is set to None explicitly rather than left to calculate_weighted_score, which writes 0.0 for an empty results list — indistinguishable from "graded and scored zero", and every downstream score or 0.0 would launder it into a real-looking failure.

Only SUCCESS/FAILURE collapse into it. ERROR, TIMEOUT, BUILD_FAILED, MAX_TURNS_EXHAUSTED and the budget stops are facts about the run, not about grading — they still apply, and execute still exits non-zero on a crash.

The switch

BatchRunConfig.grade → Orchestrator(grade=...), gating all four grading call sites. It crosses the docker boundary in context.json, defaulting to True in-container so a host predating execute keeps grading.

Deliberately not a task-config field: no 5-layer merge, no -D path. A task YAML must never declare itself ungraded; only the invoking command decides.

run and execute share one body (run_pipeline) — no third code path. Only the Typer signature is restated, and a test keeps the two option sets in step.

Refused rather than degraded

Not supported	Why
--junit-xml	A report of verdicts, and there are none. (reports_junit still emits <skipped> for an ungraded row met elsewhere.)
Simulation tasks	The dialog loop reads criteria results to decide whether to keep talking; an ungraded dialog would silently change its own stopping behavior.
stop_early:	Goes inert — it exists to cut a run once the criteria decide, and here the full trajectory is the deliverable.

--resume is supported — see part 4.

---

2. Detached grading — evaluate <run_dir>

evaluate now takes two shapes, told apart by a pure resolver (cli/evaluate_target.py) on one probe: a target holding task.json is a run directory. Passing a task file over a run directory re-grades it with different criteria, reusing the trajectory and workspace of a run you already paid for.

A re-grade must describe the run that happened

Run-dir mode rebuilds the task from the run's own task_config.resolved, not by re-reading the YAML. resolved is post-merge, so variant overrides, -D flags and dataset row expansion are already baked in — re-loading the source would silently grade a different task. Fallback to source_file happens only when resolved no longer validates, and says so loudly.

Orchestrator(prior_result=...) seeds the fresh result, carrying:

• the trajectory — every derived figure (tokens, cost, command_stats, model_used) recomputes from iterations, so seeding it reproduces them exactly;
• iteration_count, which evaluate-only used to flatten to 1;
• early_stop — load-bearing. Gate selection is FIRED-ONLY: when set, the checker gates on the weighted armed subset instead of strict-AND. Dropping it would re-grade a truncated trajectory under the full-run gate and flip the verdict;
• execution facts (max_turns_exhausted, error_message, sdk_options).

Grader-host environment_info is preserved under a graded_by sub-dict rather than overwriting the run's.

Two further parity fixes, both closing gaps the code already knew about:

• command_base_path is now persisted and restored in the evaluate-only branch. _sync_sandbox_command_path_with_agent's docstring already named "evaluate-only mode" as a known PATH gap; without this a detached grade resolves run_command binaries against ambient PATH and can disagree with the run it grades.
• _join_litellm_actual_cost skips when prior_result is set. It keys on a per-Orchestrator nonce the prior turns never carried, so it would match nothing and overwrite already-correct per-turn costs.

A re-grade refuses outright on a reference_digest mismatch — grading then would score the agent's old work against a new answer key.

The verdict is written back into the run's task.json, keeping the pre-grade record as task.execute.json. That in-place write is what makes plain coder-eval aggregate <run_dir> rebuild a graded run.json with zero new code.

---

3. Sandbox.adopt — and the pre-existing bug it fixes

adopt(workspace) reuses setup's adoption half but skips every materializing step (_setup_template, _generate_cli_recorders, venv/package installs, the destructive $HOME remediation), running only non-mutating derivation: mock-dir +x, venv discovery, plugin-tools pin. _cleanup_on_exit stays False, so an adopted tree is never moved or deleted.

In-place is more correct, not merely faster. _setup_template filters its copy through _should_ignore_template_file, whose default list drops node_modules, dist, build, .venv, .git. So evaluate today scores a file that is plainly there as missing:

copy path:  Score 0.00   "File 'node_modules/x/a.js' does not exist"
in place:   Score 1.00   "File 'node_modules/x/a.js' exists"

That is a defect independent of execute — it breaks grading for any task that builds something.

Defaults: in-place for a run directory (it is the run's own output), copy for a bare work directory (criteria can mutate it and it is the user's tree). --in-place / --copy override. adopt hard-errors on driver: docker: a container workspace is unreachable from the host, so adopting one would grade whatever happens to sit at that host path.

---

4. --resume now distinguishes "executed" from "graded"

--resume decided a task was finished by asking "does task.json carry any final_status". NOT_GRADED is a final status, so run --resume over an executed run reported the tasks complete, graded nothing, and exited 0:

after execute:            NOT_GRADED
$ coder-eval run --run-dir tmp/res --resume
↻ Resume: 1 task(s) already complete, running 0 remaining
Results: 1/1 executed, not graded
real exit code: 0
after run --resume:       NOT_GRADED

"Finished" is relative to the resuming command. partition_for_resume(tasks, *, grade) returns a four-way ResumePartition:

On disk	run --resume	execute --resume
No task.json, unreadable, or no final_status	re-run	re-run
NOT_GRADED	grade in place	already complete
Any other status, incl. FAILURE / ERROR	already complete	already complete

A NOT_GRADED row owes execute nothing but owes run a grade, so run --resume runs the criteria against the trajectory and workspace already on disk rather than paying for the agent twice — the entire reason the two commands are separate.

The carve-out is only for NOT_GRADED. Resume has never retried failures, and a parametrized test pins that so this cannot grow into a general "retry bad rows" rule. clear_rerun_artifacts skips to_grade, whose artifacts are the very thing being graded. A per-task grading failure is warned and folded back in with its original ungraded result, so one bad row neither aborts the resume nor vanishes from run.json.

grade is exempt from the config-drift warning: execute → run --resume is a supported flow, and that warning's "already-finalized tasks keep their original-config results" text is actively wrong for it. execute --resume is consequently supported and no longer refused.

orchestration/regrade.py is the single implementation shared by this path and evaluate's run-dir mode — two copies of "how to re-grade" would drift into two verdicts for the same run.

A fidelity bug the test caught

A re-graded row was reporting the grading pass's clock. A 10-minute agent run re-graded in 2 seconds would record 2 seconds — and duration_seconds feeds average_duration, the report tables and the evalboard, so harness comparisons would have been quietly wrong. A task row describes the task, so it now keeps the agent run's started_at and duration_seconds; the grading cost is preserved separately as environment_info["grading_duration_seconds"].

---

Ripple

The explicit-mapping guards did their job — every surface below failed loudly rather than silently mis-bucketing the new status: pyright on reports_junit._category_of, the _status_badge category tests, the published-action "every FinalStatus must be classified" test, and CE018's enum-parity check.

• reports_junit — ungraded → <skipped> (already counted by _set_counts).
• reports_html — neutral badge; the "no member falls through to neutral" guard now allows it for ungraded only.
• reports / reports_experiment — a Not Graded line; the pass rate reads n/a for a fully ungraded run instead of 0.0%. An ordinary empty run keeps its original 0/0 rendering — different facts.
• experiment aggregation — average_score means over graded rows only; _pick_worst_status ranks ungraded least-urgent so any real verdict wins.
• verify-published-action.yml — NOT_GRADED hard-fails. That job runs the published action, which always grades, so reaching it means the action dispatches the wrong command and every score gate in the job is measuring nothing.
• evalboard statusCategory — NOT_GRADED → "unknown", the category every consumer already treats as "no verdict here".

Also: evaluate's Typer command is now a thin wrapper over run_evaluation(...) with real Python defaults — the same split run/execute use. Calling a Typer command in-process hands unspecified options an OptionInfo sentinel, which silently made in_place=None truthy.

---

Verification

make verify green (4612 passed, 92.07%) and make evalboard-verify green (608 tests).

The headline test asserts execute + evaluate reaches the same status, score and per-criterion results as a single run — compared against a real run rather than hardcoded values, so a change breaking both paths still fails. Alongside it:

• an end-to-end execute that asserts pre_run's file is written, so a merely-skipped task cannot pass;
• a negative control proving run still scores that same task 1.0;
• aggregate rebuilds a graded run.json unaided; the trajectory survives the re-grade; the adopted workspace is not moved or deleted; task.execute.json preserves the ungraded record;
• adopt writes nothing, deletes nothing, and exposes the filtered directories;
• the target resolver, table-tested over every (one arg / two args) × (run dir / plain dir / file / missing) combination;
• the docker context.json round-trip, and run/execute signature parity.

Scoped out

Relaxing the non-empty success_criteria validator. execute on an existing task YAML needs no such change; it is only needed for a foreign task format with no criteria to declare, and belongs with that work.

🤖 Generated with Claude Code

[github-actions]

Claude finished @akshaylive's task in 1m 49s —— View job

---

Working on review...

Todo List

• Read .github/code_review.md for guidelines
• Read CLAUDE.md for project conventions
• Get full diff with git diff origin/main...HEAD
• Review all changed files with full context
• Perform cross-file consistency checks
• Analyze for missing pieces
• Format and post review