You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
Added
The eval grades each run with pass/fail graders, after claude plugin eval and skill-creator, and score is the
share that pass: structure (deterministic, against the facts: the story lights at least 80% of the changed
code, the key dependency is one the change touches, decision evidence points into the change, flow steps exist in
the code, no field repeats another), content (each example's assertions.json), prose (a rubric of
PASS/FAIL conditions) and grounded (each decision's why against its source). The summary lists the failing
graders first, with their evidence: what to fix.
examples/real/<name>/assertions.json: what any good explanation of that PR says, written once by a model from
the PR's title, description and diff, never from the tool's digest or explanation (bun scripts/eval-assertions.ts).
Train and test splits (real.jsoneval, --split, train by default): an improvement loop reads the failures
of train only and keeps a change when test does not drop.
--calibrate, the meta-eval: each committed explanation and its mutants (a dropped story step, a key dependency
the change does not touch, evidence outside the change, generic prose, a TL;DR that lists files, a goal that
repeats it, decisions that restate the change, a made-up why) graded; it reports which defects the graders catch.
--compare results.json: a blind A/B of 2 evals' explanations, run by run, asked in both orders (a verdict that
changes with the order is a tie), with a sign test.
--max-cost USD stops starting runs past the cost; --baseline also lists the graders an iteration fixed or
broke.
The summary lists the issues of each run's first render, most often first: what makes the agent render again.
Each eval run records its tool calls in order (tools, with the size of each input and result), the prompt
tokens read from and written to the cache, and the refused words behind a first render's issues: where the turns
and the tokens go.
docs/EVALS.md: the eval's graders, splits, improvement loop, options and the judge's calibration against sonnet.
Changed
local and pr no longer write the draft to explanation.json: it ends digest.md (and is in draft.json),
and the agent writes explanation.json in 1 call. An agent no longer spends a turn reading a file it already has,
since an existing file must be read before it is overwritten. The explanation is missing until then.
The digest says how to read a source file at the head of a PR (git show <head>:<path>): the checkout can be
another commit, and a failed read cost a turn.
Counts in words (two … twelve) are no longer refused: the page writes them as digits. The refusal made most
eval runs render twice.
Each placeholder states its limit in words too (≤160 chars, ~22 words): a story body or heading over its limit
was the commonest cause of a second render.
The skill asks for the digest and the fixes in as few messages as possible.
Together they cut the agent's prompt tokens by 23% and its turns by a third, at the same score: on the train
split (haiku, 1 run) 231k → 177k tokens per run, 10.8 → 8 turns, score 0.88 → 0.88; on the test split (2 runs)
210k → 161k tokens, 11.6 → 7.6 turns, score 0.82 → 0.83, cost per run −14%.
The judge's calls are lean (no tools, MCP servers, skills or settings; a 1-line system prompt): about 600 tokens
of overhead per call instead of 20k+. Haiku stays the default judge.
The agent under test runs with no MCP server and, unless --discover, no other skill: each run is isolated from
the machine's setup. Tokens do not compare with earlier evals.
The agent under test gets only the tools it may use (--tools): the others' definitions cost ~5.5k tokens on
every model call.
sourceReads also counts read-only shell commands (git show, sed, grep), which run without a permission.
The grounded judge reads the whole hunk around a code evidence (at most 80 lines), not the line ±3.
fidelity, draft and lift stay as a regression check of the structure, not a measure of quality: they compare
with the committed explanation, which the same tool wrote.
Removed
faithful: it checked the prose against the digest, which is the tool's own selection, and sat near 0.9 in
every eval. The content assertions, written from the diff, replace it.