Skip to content

fix(grade): skip incomplete command checks - #312

Merged
slowdini merged 1 commit into
devfrom
issue-305-partial-dispatch-command-checks
Sep 3, 2026
Merged

fix(grade): skip incomplete command checks#312
slowdini merged 1 commit into
devfrom
issue-305-partial-dispatch-command-checks

Conversation

@slowdini

@slowdini slowdini commented Sep 3, 2026

Copy link
Copy Markdown
Owner

Summary

  • Gate held-out command checks on the corresponding runner-owned run.json before setup injection, cache handling, command execution, or result writes.
  • Bind reusable command-check results to both the authored assertion and the exact run-record bytes, automatically refreshing stale and legacy results.
  • Report skipped incomplete tasks and warn when an old cached result means the task environment may already be contaminated.
  • Document the partial-ingest and cache contracts in CLI help, the shipped judging guide, and the intermediate-result schema.

Why

Partial ingest previously iterated every task in dispatch.json, even when only one task had completed. That could copy held-out setup files into an undispatched environment and execute the held-out command there, mutating the environment and exposing evaluation material before a retry.

Closes #305.

Behavior

Before: ingesting one completed task could execute command checks in every task environment.

After: only tasks with a runner-owned run.json are eligible. When another task completes, its next ingest executes the check exactly once; subsequent ingest reuses the result.

Safety and compatibility

  • The run record is the completion boundary, and the pipeline returns warnings for the CLI to print.
  • run_record_digest is optional in the schema, so existing result artifacts remain readable. A legacy result cannot be reused and is replaced after one execution for a completed task.
  • A cached result on an incomplete task produces a fresh-iteration warning. The grader does not attempt to clean or resume an environment that may already be contaminated.
  • --overwrite continues to force execution even when both cache digests match.

Verification

  • cargo test command_check — 32 unit, 3 CLI, and 8 run tests passed
  • cargo test — 952 unit, 216 CLI, and 208 run tests passed
  • cargo fmt --check
  • cargo clippy --all-targets -- -D warnings
  • git diff --check

The pre-push hook also repeated the complete test suite successfully.

Gate held-out command checks on the runner-owned run record so partial ingest leaves undispatched environments untouched. Bind reusable results to both the assertion and run record, and warn when legacy artifacts imply possible contamination.
@slowdini
slowdini merged commit 289900b into dev Sep 3, 2026
7 checks passed
@slowdini
slowdini deleted the issue-305-partial-dispatch-command-checks branch September 3, 2026 00:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ingest executes held-out command checks for undispatched tasks

1 participant