feat(eval): first-class artifact grading — measure an ACTING persona by her HANDS (workspace file), not her mouth#1942
Merged
Conversation
…by her HANDS (workspace file), not her mouth
The eval grades `settled.spoken` — her spoken answer. But when a persona ACTS (writes code to a file, compiles,
fixes), her code is in the FILE and her spoken text becomes a summary — so the act→verify loop scored near-zero
not because she failed but because her work was invisible to a speech-grader. (Confirmed: solo WITH a verify
directive scored 25% vs 75% spoken, purely because she filed the code.)
Fix (first-class, replaces the /tmp/grade.py hack): a `solution_file` field on EvalTask. When set alongside
`test`, the grade reads HER FILE via new `gym_grader::test_grade_file` and runs the SAME harness as a spoken
answer (`grade_rust`: strip her `main`, append the task's test, compile, run). The file resolves relative to the
core cwd — the workspace root `code/write` sandboxes to — so a relative `sol_x.rs` lands exactly where the grader
reads. Fails LOUD (never a silent pass) on a missing file ("never wrote it" = acts=0), an empty file, or a path
escaping the workspace; the file is removed after grading so a stale artifact can't false-pass a later task.
Committed reproducible gym `docs/genome/hard-rs-acted.jsonl` (8 tasks: verify-prompt + solution_file + hidden
test) — no /tmp fixtures, no external grader script; the eval grades her file natively. Unit test covers
pass-correct / clean-up / fail-loud-missing / refuse-traversal.
This is the seam that finally makes the act→verify loop — and the team's reviewer catching a real compile error —
show up in the score. Runs detached via #86; survives the client via #85.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The eval grades
settled.spoken— her spoken answer. But when a persona ACTS (writes code to a file, compiles,fixes), her code is in the FILE and her spoken text becomes a summary — so the act→verify loop scored near-zero
not because she failed but because her work was invisible to a speech-grader. (Confirmed: solo WITH a verify
directive scored 25% vs 75% spoken, purely because she filed the code.)
Fix (first-class, replaces the /tmp/grade.py hack): a
solution_filefield on EvalTask. When set alongsidetest, the grade reads HER FILE via newgym_grader::test_grade_fileand runs the SAME harness as a spokenanswer (
grade_rust: strip hermain, append the task's test, compile, run). The file resolves relative to thecore cwd — the workspace root
code/writesandboxes to — so a relativesol_x.rslands exactly where the graderreads. Fails LOUD (never a silent pass) on a missing file ("never wrote it" = acts=0), an empty file, or a path
escaping the workspace; the file is removed after grading so a stale artifact can't false-pass a later task.
Committed reproducible gym
docs/genome/hard-rs-acted.jsonl(8 tasks: verify-prompt + solution_file + hiddentest) — no /tmp fixtures, no external grader script; the eval grades her file natively. Unit test covers
pass-correct / clean-up / fail-loud-missing / refuse-traversal.
This is the seam that finally makes the act→verify loop — and the team's reviewer catching a real compile error —
show up in the score. Runs detached via #86; survives the client via #85.
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo