Skip to content

v0.2.1 — codex adapter live-validated

Choose a tag to compare

@github-actions github-actions released this 22 Aug 06:46
· 24 commits to main since this release

Live validation on a real harness 🎯

The codex adapter is now live-validated (codex-cli 0.149.0-alpha, gpt-5.6-sol): the first preregistered controlled A/B on a real agent.

What the live run produced (4 tasks × 2 variants, description-clarity A/B):

arm subject selection ops success consumption real tokens/op median steps
control (plain description) 0/4 4 100% 4/4 188,520 9.5
clear (rewritten description) 1/4 4 100% 4/4 215,021 22.5

Verdict: honest null (+25pp observed at n=4/arm, p=0.29) with next-round sizing guidance (≈31/arm) — exactly the discipline the engine exists to enforce. A weak default description losing to better-described competitors matches Hasan et al. (arXiv 2602.14878).

Fixes found by the live run (scripted tests could not catch these): mcp_tool_call item parsing; --approve-for-me for MCP calls in exec mode; stdin DEVNULL; toolserver contract fix (spec id→name, crashed tools/list) with a contract regression test; candidate-set steering in the episode prompt; token usage metered as cost units (1 unit = 1 token) so the budget breaker bounds real spend.

claude-code shares these fixes and remains scripted-transcript-tested pending its own live run. 78 tests.