Repository navigation
Releases: sblattj/agentic-coding-harness
Release list
v0.25.0
v0.24.0
Added
- Fixes #97:
?key-help overlay inach dash(keys, status glyph legend, flags), shown once on first launch, with--help-keys/--no-help
What's Changed
Full Changelog: v0.23.0...v0.24.0
v0.23.0
Added
- Fixes #67 — Per-model cost chips in run header (web + dash)
- Fixes #70 — Registry retention/compaction: TTL policy per artifact class,
ach gccommand - Fixes #94 — Global stats endpoint + page (totals across all runs: models, agents, success rates)
Changed
- Release workflow publishes to npm with trusted publishing (OIDC) instead of an NPM_TOKEN secret (#147)
What's Changed
- feat(web): GET /api/stats global totals + stats bar by @sblattj in #144
- feat(cli): ach gc retention for raw transcripts and run records by @sblattj in #146
- ci(release): publish to npm with trusted publishing (OIDC), not NPM_TOKEN by @sblattj in #147
- feat(web): per-model cost chips in the run header and ach dash by @sblattj in #145
Full Changelog: v0.22.0...v0.23.0
v0.22.0
Added
- Codex runs now show which model actually ran next to the one you asked for.
codex exec --jsonnever names the model, so afterach run --agent codexthe harness reads the thread's own rollout file and records the observed model and reasoning effort from itsturn_contextlines. Usage records whose model wasunknownget the observed model. The run verdict (usage.model) keeps requested and observed apart, flags a mismatch as a warning, and when no rollout is found it says the requested model is unverified instead of passing it off as observed. The run summary,ach dash,ach webandach reportshow it. Codex transcript records (ach stats) now carry their model and effort, so--by modelhas real codex rows. A run's own cost is still computed from the stream, before the session file is read, so a codex run without-mshows cost n/a; its pricing warning now names the observed model instead ofunknown model "unknown", andach statsprices the session by that model. (#140) - Codex token counters now say what they include and whether anything is missing. Transcript reading supports the modern
event_msg/token_countlines (with the latest call's tokens and the stated context window), still reads the legacytoken_usage_record, and never counts the same usage twice when a file has both. A child session's file no longer takes its parent's id, and history it replays from its parent is not counted again. Child sessions are linked to their parents, andach statsreports how many session trees are complete, child-inclusive, or missing children, whose usage stays unknown rather than zero. A codex run's verdict marks its stream totalsparent-onlywhen the run spawned child sessions, adds child-inclusive totals from the child rollouts it finds, and lists the missing ones. The run summary,ach dash,ach webandach reportshow a partial/complete badge, and a run that has not reported usage yet showsn/ainstead of zeros. (#142)
Fixed
- Context meter: aggregate usage (codex
turn.completed, geminiresult.stats, a claude result with no per-message usage) is no longer shown as context occupancy. A synthetic codex turn that summed 3,000 tokens against a 1,000-token window reported300%. Such aturn-totalreading now has nopercentage; it carries the count ascumulativeTokensand every surface shows occupancy as n/a beside a labelled cumulative count: the dashn/a (cum 3.0k), the HTML reportctx n/a · cumulative 3,000 tok, the run summaryctx n/a (cumulative 3,000 tok, not occupancy), theach webfeedΣ3.0k cumwith no gauge bar and the run detailctx n/a (cumulative 3,000 tok, not occupancy), andach stats --jsonwithcontextPercentage/contextTokensnull and a newcumulativeTokensfield. Records written before this release that stored a turn-total percentage render the same way. Nothing is clamped to 100, and the overflow warning still fires only on a latest-call reading.totals.contextTokensis now set only for an occupancy reading (#141). - Context meter: native latest-call usage and a reported context window are now used when an agent provides them. A usage record's
extra.lastTokenUsage({ inputTokens, cacheReadTokens?, cacheWriteTokens? }) is read as alast-calloccupancy reading, andextra.modelContextWindowsets the window with the newwindowSource: 'reported'(estimated: false). Each latest-call reading replaces the previous one, so a smaller call after compaction lowers the reading (#141). ach run --agent codexnow gets a real context-occupancy reading when the thread's rollout has one. The driver feeds the rollout's latest call of this run (last_token_usage, uncached input plus cache read) and its stated window to the context meter before the run's final reading is sealed, so the summary, registry, report,ach stats --jsonandtotals.contextTokensshow alast-callpercentage against areportedwindow instead ofn/a (cumulative). A rollout without per-call usage stays cumulative, and compaction is handled because the newest call wins. Rollout records whosetoken_countline states no window take it from the turn'stask_startedevent (contextWindowSource: 'task_started'). (#141, #142)
v0.21.0
Added
- Fixes #63 — Run lifecycle notifications: desktop notification + optional sound on finish/fail/budget events
What's Changed
Full Changelog: v0.20.3...v0.21.0
v0.20.3
Fixed
- README Quickstart (#139, found by running
ach smokeon it): it now says what you need before the first command. That is Node.js 22+ and an agent CLI that is installed and signed in, with the Claude Code and Codex installs as commands you can copy. It explains that eachach rungives Claude Code a fresh config dir, so the examples pass--claude-default-config(or you setCLAUDE_CODE_OAUTH_TOKEN/ANTHROPIC_API_KEY). It also saysach webkeeps running, so start it in its own terminal (--no-openskips the browser). ach smoke:spawn claude ENOENTis reported as a missing prerequisite that names the agent CLI, not as a bare failure. Friction evidence skips npmnotice/warnlines. A command missing because an earlier Homebrew install was skipped (host-level) now says so, and no fix is planned for it.ach smoke: opening a loopback URL (127.x,localhost,[::1],0.0.0.0) is never treated as a browser login. It is recorded aslocalUrls. A step that is still running after 8 s while listening on a TCP port (ach web, a dev server) passes as a service and is stopped at teardown, instead of failing on the step timeout.ach smoke: an agent CLI that runs but is not signed in ("Not logged in", "please run /login", "not authenticated") isauth-needs-humanfriction. A step that refuses to run outside a git repository (codex: "Not inside a trusted directory") is anunstated-assumption.ach smoke: the sandbox pre-creates.codex, because codex exits whenCODEX_HOMEpoints at a missing directory.
Fixes #139
v0.20.2
Fixed
ach smokeon macOS (#139): a step, or a program it runs, that called/usr/bin/open URL(or/usr/bin/osascript, which Python'swebbrowseruses) by absolute path bypassed the PATH shims and opened your real browser. A literal/usr/bin/openin the step text is now caught by a zsh function that runs the sandbox shim, so the URL is recorded. Shell steps also run undersandbox-execdenying exec of those two binaries, so child processes are blocked withOperation not permitted, reported as a specificdeniedfriction.sandbox-execis probed once per sandbox and skipped (plain/bin/zsh) when it is unavailable. Prompt steps are unchanged.ach smoke --fix-agent claudeand--fix-toolwith claude failed with "Not logged in": the fix agent ranach run --agent claudewith a fresh per-run config dir, which cannot see your Claude login. The fix agent works on the host on your behalf, so with claude it now passes--claude-default-configand uses your login. Found in a live run.ach smoke --copyon macOS put only RTF on the clipboard (pbcopy -Prefer rtf), sopbpasteand plain-text fields got nothing and Chrome pages such as Gmail could not paste it as rich text. It now writes HTML, RTF and the plain markdown in one pasteboard write.ach smoke --fix-agent: when the agent wrapped its replacement step in a markdown fence (```sh), the fence was kept. The revised message then had nested fences, and the rerun passed the backticks to zsh as a command substitution. A fence around the whole replacement is now dropped, and the prompt asks for plain command text. Found in a live run.ach smoke: tokens spent by the--fix-agent/--fix-toolagents were dropped, so the footer saidtotal tokens: 0after a run that called Claude. They are now in the report asfixAgentTokensand in the footer as(+ N fix-agent).
Fixes #139 (follow-up: gaps found by live-testing ach smoke)
v0.20.1
Fixed
ach smokeon macOS (#139): the sandbox keychain no longer fails to set up on GitHub's macOS 26 runners. It now gets a random password, kept in the sandbox'sstate/keychain-password(mode 0600), instead of an empty one. The separateunlock-keychaincall is gone: a freshly created keychain is already unlocked, and on those runnersunlock-keychainrejected even the correct password. The "real missing command" runner test, which needs the macOS sandbox shell, now skips on Linux.
Fixes #139 (follow-up: CI keychain setup on macOS runners)
v0.20.0
Added
ach smoke <message-file>(#139): before you send someone setup or how-to steps, the harness runs those exact steps itself as a brand-new user and reports the friction. The steps come from the message'ssh/bash/zsh/consolefences,promptfences (typed into an agent throughach run, with tokens counted) andopenfences or "Open https://..." lines. Each step runs literally in one sandboxed terminal session: a freshHOME, config dirs for the common CLIs (npm, git, gh, Claude Code, Codex, Gemini, Azure, gcloud, AWS, Docker, kube and more), a minimalPATHwith none of your installs (--allow-pathadds a dir), and on macOS its own empty-passwordlogin.keychain-db, so CLIs inside no longer raise "A keychain cannot be found" popups on the host. The host keychain search list and default are read before the run, checked after, and restored if they changed; measured byte-identical. Every process a step starts is killed on exit, including on Ctrl-C: tracked process groups, then any escapee with the sandbox'sHOMEin its environment or its cwd under the sandbox. Browser logins run in your real browser over CDP (--cdp, default$CDP_BASEorhttp://127.0.0.1:9333) in a new background tab that is always closed. Device codes are filled in, and Continue/Authorize is clicked only when the page shows the account the step needs (--accountor a fence'saccount=). If the browser is signed in to a different account, or the step names none, nothing is clicked and you get a one-line login command instead. Electron endpoints such as an IDE are refused. A device-code login blocked by policy (for exampleAADSTS53003conditional access) is rerun as the normal browser login. Outward writes go to local stubs (--stub jira=JIRA_BASE_URL, presets for Jira, GitHub, Linear and Slack,--stub host:*.atlassian.netor--block-hostrefused through a local proxy), so the run creates nothing real. Each step records pass/fail, time, tokens, prompts and denials. Fixes are made in place and the step is rerun in the same sandbox: a missing install is inserted before the first step that needs it (with its own prerequisites), a health check that says a binary is installed when it is not onPATHis flagged astool-lied, a policy-blocked device-code login becomes the browser login, and a confirm prompt gets its non-interactive flag.--fix-agent Ahands other failures to a coding agent to rewrite the step, and--fix-tool REPOlets it patch the lying tool (uncommitted, reported as a diffstat). Host-level commands (sudo, Homebrew, xcode-select, writes under/usr/local,/optand so on) are never run, because the sandbox shares the OS. The output is the friction table, the details, and<name>.smoked.mdwith every fix applied;--copycopies it as rich text, and--jsonprints the report. Exit 0 clean, 1 friction left, 2 usage. macOS only for now. See docs/SMOKE.md.
Fixes #139 — Smoke the instructions: run a how-to as a brand-new user in an isolated sandbox and fix friction before sending
v0.19.1
Fixes #137 — kiro: context tokens use an assumed 200k window and the last percent, so start size reads wrong
Fixes #126 — Release: npm stuck at 0.16.1 (0.17.0 to 0.19.0 never published), so the homebrew job fails and bun add -g installs a stale ach
Fixes #135 — Exit reason picks a routine INFO log line instead of the real stderr error
Fixes #133 — kiro (headless v3): nonzero exit hides the CLI's stderr reason, e.g. monthly usage limit
Fixes #131 — Kiro tap leaks HTTPS_PROXY into the agent's shell tools; other HTTPS clients get 502
Fixes #128 — Piped stdout is truncated at 64 KiB (binary) / 128 KiB (bun from source) since 0.17: ach dash --json --all | jq gets invalid JSON
Fixes #127 — kiro ACP transport: --kiro-engine v3 dies in initialize; kiro-cli acp rejects --agent/--model/--effort/--trust-all-tools with v3
Fixed
- kiro: the context window is now derived instead of assumed (#137). A v3 run's
context_usagebreakdown gives the window from its largest category (tokens / percent), snapped to a known size, so Opus 5.5 andautoruns read a 1M window (windowSource: 'derived') instead of the assumed 200k, which read context about 5x low. Summing the breakdown's token counts is wrong after the first update, because kiro zeroes some category tokens while keeping their percent.usage.contextalso gainsstartPercentage/startTokens(the first reading, the session's start size) and an optionalbreakdown;percentage/tokensstay the end reading, and the run summary prints both. Window precedence: session store, then derived, then assumed. - npm stuck at 0.16.1 (#126): the release workflow now publishes the tagged version to npm (
npmjob, provenance, idempotent, fails loudly without theNPM_TOKENsecret), and the homebrew job waits on it. A backfill of an older tag (gh workflow run release-binaries.yml -f tag=vX.Y.Z) publishes under thebackfilldist-tag, solatestnever moves backwards. See docs/PACKAGING.md. - The nonzero-exit reason no longer picks routine log lines (#135). A headless kiro-cli run that exited 1 read
kiro-cli exited with code 1: [INFO] [KRS] <-- GenerateAssistantResponseCommand done totalEvents=9, because the fallback took the last stderr line.pickStderrReason(src/adapters/shared.ts) now never picks INFO/DEBUG/TRACE-tagged lines ([INFO],DEBUG,level=trace). It prefers the last line that looks like a failure (error,failed,not offered,denied,exceeded,unauthorized,forbidden), then the last WARN line, then the last other non-empty line. When only routine lines remain, the message stays bare. The stderr tail grew from 5 to 20 lines (still 4 KB per line), so trailing INFO lines no longer push the real error out. - A CLI child that exits nonzero now reports why (#133). The exit error used to say only
kiro-cli exited with code 1, so a kiro-cli headless run that stopped onThe monthly usage limit has been reachedlooked like a crash. The shared run loop (src/adapters/shared.ts) keeps the last few stderr lines and appends the most relevant one, ANSI codes stripped: the last line matchingerror, else the last non-empty line, as inkiro-cli exited with code 1: Error: Internal error (code -32603): ... The monthly usage limit has been reached. The message prefix is unchanged, so existing matchers still work. When an adapter's own stderr hook already raised a specific error (cursor), the exit message stays bare, so the reason is not reported twice. Exit 0 is unchanged. - kiro: the MITM credit tap no longer breaks HTTPS from the agent's shell tools (#131). kiro-cli hands the tap env (
HTTPS_PROXY,SSL_CERT_FILE,NODE_EXTRA_CA_CERTS) to every command the agent runs, and mitmdump intercepted every host. A host whose certificate chain mitmdump could not verify (for example behind a TLS-inspecting network proxy) came back as HTTP 502, so another vendor's CLI run from the agent's shell failed on every call. mitmdump now gets--allow-hostsfor the metered hosts only (codewhisperer.*.amazonaws.com,runtime.*.kiro.dev, extendable withACH_KIRO_TAP_HOSTS), and other hosts are tunneled raw to their origin.SSL_CERT_FILEnow points at the system roots plus the tap CA instead of the bare tap CA, so OpenSSL clients can still verify tunneled hosts. See docs/KIRO.md. - Piped stdout is no longer truncated under Bun (#128).
ach dash --json --all | wc -cstopped at 64 KiB in the compiled binaries and 128 KiB from source, because the CLI calledprocess.exitbefore the pipe drained: under Bun the emptywrite("", cb)guard fires its callback while earlier output is still queued. The CLI now records stdout/stderr backpressure (awrite()that returns false) and waits fordrainbefore every forced exit (src/cli/stdio-flush.ts), includingwatch,serveand the livedashshutdown paths. - kiro:
--kiro-transport acp --kiro-engine v3no longer dies ininitialize(#127). kiro-cli rejects--agent,--model,--effort,--trust-all-toolsand--trust-toolswith--agent-engine v3onacp, so for v3 only--agent-engine v3is passed. Aftersession/new(orsession/load) the agent (a mode), model and effort are set withsession/set_config_option, because v3 does not implementsession/set_model. Each value is checked against the options v3 sends back, since v3 accepts unknown values without an error. An agent the session does not offer fails before any prompt, and the error suggests--kiro-transport headless. A fresh workspace can load its model catalog aftersession/new, so the handshake waits up to 15 s for it.kiro.toolsis applied by answeringsession/request_permission(allallows, a list allows the listed tool ids or ACP kinds,noneor unset denies). The run evidence recordsengine,effortAck,currentEffortandtoolTrust. Verified live on kiro-cli 2.29.0.