Skip to content

v0.10.0

Choose a tag to compare

@github-actions github-actions released this 18 Oct 07:23
· 50 commits to main since this release
a238f2e

Added

  • The eval grades each run with pass/fail graders, after claude plugin eval and skill-creator, and score is the
    share that pass: structure (deterministic, against the facts: the story lights at least 80% of the changed
    code, the key dependency is one the change touches, decision evidence points into the change, flow steps exist in
    the code, no field repeats another), content (each example's assertions.json), prose (a rubric of
    PASS/FAIL conditions) and grounded (each decision's why against its source). The summary lists the failing
    graders first, with their evidence: what to fix.
  • examples/real/<name>/assertions.json: what any good explanation of that PR says, written once by a model from
    the PR's title, description and diff, never from the tool's digest or explanation (bun scripts/eval-assertions.ts).
  • Train and test splits (real.json eval, --split, train by default): an improvement loop reads the failures
    of train only and keeps a change when test does not drop.
  • --calibrate, the meta-eval: each committed explanation and its mutants (a dropped story step, a key dependency
    the change does not touch, evidence outside the change, generic prose, a TL;DR that lists files, a goal that
    repeats it, decisions that restate the change, a made-up why) graded; it reports which defects the graders catch.
  • --compare results.json: a blind A/B of 2 evals' explanations, run by run, asked in both orders (a verdict that
    changes with the order is a tie), with a sign test.
  • --max-cost USD stops starting runs past the cost; --baseline also lists the graders an iteration fixed or
    broke.
  • The summary lists the issues of each run's first render, most often first: what makes the agent render again.
  • Each eval run records its tool calls in order (tools, with the size of each input and result), the prompt
    tokens read from and written to the cache, and the refused words behind a first render's issues: where the turns
    and the tokens go.
  • docs/EVALS.md: the eval's graders, splits, improvement loop, options and the judge's calibration against sonnet.

Changed

  • local and pr no longer write the draft to explanation.json: it ends digest.md (and is in draft.json),
    and the agent writes explanation.json in 1 call. An agent no longer spends a turn reading a file it already has,
    since an existing file must be read before it is overwritten. The explanation is missing until then.
  • The digest says how to read a source file at the head of a PR (git show <head>:<path>): the checkout can be
    another commit, and a failed read cost a turn.
  • Counts in words (two … twelve) are no longer refused: the page writes them as digits. The refusal made most
    eval runs render twice.
  • Each placeholder states its limit in words too (≤160 chars, ~22 words): a story body or heading over its limit
    was the commonest cause of a second render.
  • The skill asks for the digest and the fixes in as few messages as possible.
  • Together they cut the agent's prompt tokens by 23% and its turns by a third, at the same score: on the train
    split (haiku, 1 run) 231k → 177k tokens per run, 10.8 → 8 turns, score 0.88 → 0.88; on the test split (2 runs)
    210k → 161k tokens, 11.6 → 7.6 turns, score 0.82 → 0.83, cost per run −14%.
  • The judge's calls are lean (no tools, MCP servers, skills or settings; a 1-line system prompt): about 600 tokens
    of overhead per call instead of 20k+. Haiku stays the default judge.
  • The agent under test runs with no MCP server and, unless --discover, no other skill: each run is isolated from
    the machine's setup. Tokens do not compare with earlier evals.
  • The agent under test gets only the tools it may use (--tools): the others' definitions cost ~5.5k tokens on
    every model call.
  • sourceReads also counts read-only shell commands (git show, sed, grep), which run without a permission.
  • The grounded judge reads the whole hunk around a code evidence (at most 80 lines), not the line ±3.
  • fidelity, draft and lift stay as a regression check of the structure, not a measure of quality: they compare
    with the committed explanation, which the same tool wrote.

Removed

  • faithful: it checked the prose against the digest, which is the tool's own selection, and sat near 0.9 in
    every eval. The content assertions, written from the diff, replace it.