Skip to content

feat(grading): add diff-scope assertions#164

Merged
slowdini merged 1 commit into
devfrom
feat/diff-scope
Jul 24, 2026
Merged

feat(grading): add diff-scope assertions#164
slowdini merged 1 commit into
devfrom
feat/diff-scope

Conversation

@slowdini

Copy link
Copy Markdown
Owner

Summary

  • add a deterministic diff_scope assertion with max_files_touched and max_lines_changed thresholds
  • capture post-dispatch task-environment metrics for every run in diff-scope.json and preserve raw per-run values in benchmark.json
  • give every (eval, condition, run) a private environment, exclude .eval-magic-outputs, and capture baselines after staging, guard setup, and shadow preflight
  • update stray-write boundaries, schemas, CLI help, workflow documentation, golden prompts, and integration coverage

Why

The grading pipeline could check correctness, transcripts, and held-out commands, but could not measure whether an agent made a focused change. This adds scope as a deterministic secondary signal while retaining correctness assertions as the primary measure.

Before, dispatch prompts constrained all writes to the framework output directory and benchmark artifacts contained no raw change-scope context. After this change, agents may work normally inside a private task environment while eval-magic measures files, added/removed byte-lines, and zero-context Myers hunks outside the hidden framework-output subtree.

Older iterations remain readable. Their benchmark omits diff_scope; configuring a scope assertion without a captured baseline produces rebuild guidance.

Closes #116

Verification

  • cargo fmt --check
  • cargo test (554 unit, 107 CLI, 94 run tests)
  • cargo clippy --all-targets -- -D warnings
  • git diff --check
  • pre-commit formatting and Clippy hooks
  • pre-push full test hook

@slowdini
slowdini marked this pull request as ready for review July 24, 2026 05:54
@slowdini
slowdini merged commit a6140b7 into dev Jul 24, 2026
6 checks passed
@slowdini
slowdini deleted the feat/diff-scope branch July 24, 2026 16:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: diff_scope assertion — measure fix scope (files/lines changed)

1 participant