Skip to content

v0.2.9

Choose a tag to compare

@ymadd ymadd released this 31 Jul 11:29
· 65 commits to main since this release

Reasoning effort is now a dial you can turn

One canonical ladder — low < medium < high < xhigh < max — mapped to whatever each backend's CLI calls it, in one auditable place. A rung a backend cannot express is clamped down, never up, and the clamped value is what gets recorded, so telemetry says what actually ran rather than what was asked for.

agentpit rescue --backend claude --effort xhigh "…"
agentpit ensemble --effort high "…"
agentpit profile run --backend codex --effort xhigh

Also configurable as [backends.<id>].effort and [workflow.roles.<name>].effort, and exposed on the MCP dispatch_task / run_ensemble tools. Precedence mirrors model exactly: flag > role > backend default.

canonical claude codex agy opencode
low / medium / high same same same same
xhigh xhigh xhigh clamped to high xhigh
max max clamped to xhigh clamped to high max

agy binds effort to the model id (gemini-3.6-flash-high) and rejects a separate --effort, so a pinned model wins and says so instead of failing the dispatch.

Capability rows are keyed by what produced them

A capability score was never a property of a CLI's name — it belongs to (backend, model, effort). profiles.toml now keys on the triple, so a high-effort measurement and a low-effort one coexist instead of one silently overwriting the other. An unmeasured variant falls back to the unpinned row and never borrows a sibling rung's number.

Existing files load unchanged: a bare [profile.codex] is the unpinned row it always was.

agentpit arena — blind head-to-head, judged by a human

agentpit arena run --contenders claude,codex "<what to build>"
agentpit arena vote
agentpit arena leaderboard

Every contender builds the same thing in its own git worktree; its work is reduced to a patch; you pick the better one blind and pairwise, with the names withheld until every vote in the round is in. Standings come from a Bradley–Terry fit with a bootstrap interval — a unanimous handful of votes is marked provisional rather than presented as a finding. Votes flow into the ordinary learning fold as MemberGraded, not as a new source that outranks a measured benchmark.

Two long-standing bugs, found by running the real CLIs

Both had passing unit tests the whole time; the tests asserted argument construction, which is not the same as the flag working.

  • --model never reached antigravity. agy --print is an alias for --prompt and eats the next token as its value, so any flag after it was swallowed into the prompt.
  • codex could not write files. codex exec sandboxes to read-only by default and the adapter set none, so codex dispatches described the edit they would have made and changed nothing — while declaring full autonomy. Now --sandbox workspace-write; danger-full-access stays reserved for the workflow manager.