Skip to content

Releases: sblattj/agentic-coding-harness

v0.25.0

Choose a tag to compare

@github-actions github-actions released this 16 Oct 00:33

What's Changed

  • ci(release): push the Homebrew formula with a deploy key by @sblattj in #149
  • feat(dash): --set-terminal-title keeps run state in the tab title (#92) by @sblattj in #150
  • chore(release): 0.25.0 (#92) by @sblattj in #151

Full Changelog: v0.24.0...v0.25.0

v0.24.0

Choose a tag to compare

@github-actions github-actions released this 14 Oct 04:08

Added

  • Fixes #97: ? key-help overlay in ach dash (keys, status glyph legend, flags), shown once on first launch, with --help-keys / --no-help

What's Changed

Full Changelog: v0.23.0...v0.24.0

v0.23.0

Choose a tag to compare

@github-actions github-actions released this 13 Oct 05:25

Added

  • Fixes #67 — Per-model cost chips in run header (web + dash)
  • Fixes #70 — Registry retention/compaction: TTL policy per artifact class, ach gc command
  • Fixes #94 — Global stats endpoint + page (totals across all runs: models, agents, success rates)

Changed

  • Release workflow publishes to npm with trusted publishing (OIDC) instead of an NPM_TOKEN secret (#147)

What's Changed

  • feat(web): GET /api/stats global totals + stats bar by @sblattj in #144
  • feat(cli): ach gc retention for raw transcripts and run records by @sblattj in #146
  • ci(release): publish to npm with trusted publishing (OIDC), not NPM_TOKEN by @sblattj in #147
  • feat(web): per-model cost chips in the run header and ach dash by @sblattj in #145

Full Changelog: v0.22.0...v0.23.0

v0.22.0

Choose a tag to compare

@sblattj sblattj released this 13 Oct 04:37

Added

  • Codex runs now show which model actually ran next to the one you asked for. codex exec --json never names the model, so after ach run --agent codex the harness reads the thread's own rollout file and records the observed model and reasoning effort from its turn_context lines. Usage records whose model was unknown get the observed model. The run verdict (usage.model) keeps requested and observed apart, flags a mismatch as a warning, and when no rollout is found it says the requested model is unverified instead of passing it off as observed. The run summary, ach dash, ach web and ach report show it. Codex transcript records (ach stats) now carry their model and effort, so --by model has real codex rows. A run's own cost is still computed from the stream, before the session file is read, so a codex run without -m shows cost n/a; its pricing warning now names the observed model instead of unknown model "unknown", and ach stats prices the session by that model. (#140)
  • Codex token counters now say what they include and whether anything is missing. Transcript reading supports the modern event_msg/token_count lines (with the latest call's tokens and the stated context window), still reads the legacy token_usage_record, and never counts the same usage twice when a file has both. A child session's file no longer takes its parent's id, and history it replays from its parent is not counted again. Child sessions are linked to their parents, and ach stats reports how many session trees are complete, child-inclusive, or missing children, whose usage stays unknown rather than zero. A codex run's verdict marks its stream totals parent-only when the run spawned child sessions, adds child-inclusive totals from the child rollouts it finds, and lists the missing ones. The run summary, ach dash, ach web and ach report show a partial/complete badge, and a run that has not reported usage yet shows n/a instead of zeros. (#142)

Fixed

  • Context meter: aggregate usage (codex turn.completed, gemini result.stats, a claude result with no per-message usage) is no longer shown as context occupancy. A synthetic codex turn that summed 3,000 tokens against a 1,000-token window reported 300%. Such a turn-total reading now has no percentage; it carries the count as cumulativeTokens and every surface shows occupancy as n/a beside a labelled cumulative count: the dash n/a (cum 3.0k), the HTML report ctx n/a · cumulative 3,000 tok, the run summary ctx n/a (cumulative 3,000 tok, not occupancy), the ach web feed Σ3.0k cum with no gauge bar and the run detail ctx n/a (cumulative 3,000 tok, not occupancy), and ach stats --json with contextPercentage/contextTokens null and a new cumulativeTokens field. Records written before this release that stored a turn-total percentage render the same way. Nothing is clamped to 100, and the overflow warning still fires only on a latest-call reading. totals.contextTokens is now set only for an occupancy reading (#141).
  • Context meter: native latest-call usage and a reported context window are now used when an agent provides them. A usage record's extra.lastTokenUsage ({ inputTokens, cacheReadTokens?, cacheWriteTokens? }) is read as a last-call occupancy reading, and extra.modelContextWindow sets the window with the new windowSource: 'reported' (estimated: false). Each latest-call reading replaces the previous one, so a smaller call after compaction lowers the reading (#141).
  • ach run --agent codex now gets a real context-occupancy reading when the thread's rollout has one. The driver feeds the rollout's latest call of this run (last_token_usage, uncached input plus cache read) and its stated window to the context meter before the run's final reading is sealed, so the summary, registry, report, ach stats --json and totals.contextTokens show a last-call percentage against a reported window instead of n/a (cumulative). A rollout without per-call usage stays cumulative, and compaction is handled because the newest call wins. Rollout records whose token_count line states no window take it from the turn's task_started event (contextWindowSource: 'task_started'). (#141, #142)

Fixes #140
Fixes #141
Fixes #142

v0.21.0

Choose a tag to compare

@github-actions github-actions released this 13 Oct 04:08

Added

  • Fixes #63 — Run lifecycle notifications: desktop notification + optional sound on finish/fail/budget events

What's Changed

  • feat(notify): opt-in desktop notifications on run finish/fail/budget by @sblattj in #143

Full Changelog: v0.20.3...v0.21.0

v0.20.3

Choose a tag to compare

@sblattj sblattj released this 13 Oct 00:44

Fixed

  • README Quickstart (#139, found by running ach smoke on it): it now says what you need before the first command. That is Node.js 22+ and an agent CLI that is installed and signed in, with the Claude Code and Codex installs as commands you can copy. It explains that each ach run gives Claude Code a fresh config dir, so the examples pass --claude-default-config (or you set CLAUDE_CODE_OAUTH_TOKEN / ANTHROPIC_API_KEY). It also says ach web keeps running, so start it in its own terminal (--no-open skips the browser).
  • ach smoke: spawn claude ENOENT is reported as a missing prerequisite that names the agent CLI, not as a bare failure. Friction evidence skips npm notice/warn lines. A command missing because an earlier Homebrew install was skipped (host-level) now says so, and no fix is planned for it.
  • ach smoke: opening a loopback URL (127.x, localhost, [::1], 0.0.0.0) is never treated as a browser login. It is recorded as localUrls. A step that is still running after 8 s while listening on a TCP port (ach web, a dev server) passes as a service and is stopped at teardown, instead of failing on the step timeout.
  • ach smoke: an agent CLI that runs but is not signed in ("Not logged in", "please run /login", "not authenticated") is auth-needs-human friction. A step that refuses to run outside a git repository (codex: "Not inside a trusted directory") is an unstated-assumption.
  • ach smoke: the sandbox pre-creates .codex, because codex exits when CODEX_HOME points at a missing directory.

Fixes #139

v0.20.2

Choose a tag to compare

@sblattj sblattj released this 12 Oct 22:57

Fixed

  • ach smoke on macOS (#139): a step, or a program it runs, that called /usr/bin/open URL (or /usr/bin/osascript, which Python's webbrowser uses) by absolute path bypassed the PATH shims and opened your real browser. A literal /usr/bin/open in the step text is now caught by a zsh function that runs the sandbox shim, so the URL is recorded. Shell steps also run under sandbox-exec denying exec of those two binaries, so child processes are blocked with Operation not permitted, reported as a specific denied friction. sandbox-exec is probed once per sandbox and skipped (plain /bin/zsh) when it is unavailable. Prompt steps are unchanged.
  • ach smoke --fix-agent claude and --fix-tool with claude failed with "Not logged in": the fix agent ran ach run --agent claude with a fresh per-run config dir, which cannot see your Claude login. The fix agent works on the host on your behalf, so with claude it now passes --claude-default-config and uses your login. Found in a live run.
  • ach smoke --copy on macOS put only RTF on the clipboard (pbcopy -Prefer rtf), so pbpaste and plain-text fields got nothing and Chrome pages such as Gmail could not paste it as rich text. It now writes HTML, RTF and the plain markdown in one pasteboard write.
  • ach smoke --fix-agent: when the agent wrapped its replacement step in a markdown fence (```sh), the fence was kept. The revised message then had nested fences, and the rerun passed the backticks to zsh as a command substitution. A fence around the whole replacement is now dropped, and the prompt asks for plain command text. Found in a live run.
  • ach smoke: tokens spent by the --fix-agent / --fix-tool agents were dropped, so the footer said total tokens: 0 after a run that called Claude. They are now in the report as fixAgentTokens and in the footer as (+ N fix-agent).

Fixes #139 (follow-up: gaps found by live-testing ach smoke)

v0.20.1

Choose a tag to compare

@sblattj sblattj released this 12 Oct 21:18

Fixed

  • ach smoke on macOS (#139): the sandbox keychain no longer fails to set up on GitHub's macOS 26 runners. It now gets a random password, kept in the sandbox's state/keychain-password (mode 0600), instead of an empty one. The separate unlock-keychain call is gone: a freshly created keychain is already unlocked, and on those runners unlock-keychain rejected even the correct password. The "real missing command" runner test, which needs the macOS sandbox shell, now skips on Linux.

Fixes #139 (follow-up: CI keychain setup on macOS runners)

v0.20.0

Choose a tag to compare

@sblattj sblattj released this 12 Oct 20:42

Added

  • ach smoke <message-file> (#139): before you send someone setup or how-to steps, the harness runs those exact steps itself as a brand-new user and reports the friction. The steps come from the message's sh/bash/zsh/console fences, prompt fences (typed into an agent through ach run, with tokens counted) and open fences or "Open https://..." lines. Each step runs literally in one sandboxed terminal session: a fresh HOME, config dirs for the common CLIs (npm, git, gh, Claude Code, Codex, Gemini, Azure, gcloud, AWS, Docker, kube and more), a minimal PATH with none of your installs (--allow-path adds a dir), and on macOS its own empty-password login.keychain-db, so CLIs inside no longer raise "A keychain cannot be found" popups on the host. The host keychain search list and default are read before the run, checked after, and restored if they changed; measured byte-identical. Every process a step starts is killed on exit, including on Ctrl-C: tracked process groups, then any escapee with the sandbox's HOME in its environment or its cwd under the sandbox. Browser logins run in your real browser over CDP (--cdp, default $CDP_BASE or http://127.0.0.1:9333) in a new background tab that is always closed. Device codes are filled in, and Continue/Authorize is clicked only when the page shows the account the step needs (--account or a fence's account=). If the browser is signed in to a different account, or the step names none, nothing is clicked and you get a one-line login command instead. Electron endpoints such as an IDE are refused. A device-code login blocked by policy (for example AADSTS53003 conditional access) is rerun as the normal browser login. Outward writes go to local stubs (--stub jira=JIRA_BASE_URL, presets for Jira, GitHub, Linear and Slack, --stub host:*.atlassian.net or --block-host refused through a local proxy), so the run creates nothing real. Each step records pass/fail, time, tokens, prompts and denials. Fixes are made in place and the step is rerun in the same sandbox: a missing install is inserted before the first step that needs it (with its own prerequisites), a health check that says a binary is installed when it is not on PATH is flagged as tool-lied, a policy-blocked device-code login becomes the browser login, and a confirm prompt gets its non-interactive flag. --fix-agent A hands other failures to a coding agent to rewrite the step, and --fix-tool REPO lets it patch the lying tool (uncommitted, reported as a diffstat). Host-level commands (sudo, Homebrew, xcode-select, writes under /usr/local, /opt and so on) are never run, because the sandbox shares the OS. The output is the friction table, the details, and <name>.smoked.md with every fix applied; --copy copies it as rich text, and --json prints the report. Exit 0 clean, 1 friction left, 2 usage. macOS only for now. See docs/SMOKE.md.

Fixes #139 — Smoke the instructions: run a how-to as a brand-new user in an isolated sandbox and fix friction before sending

v0.19.1

Choose a tag to compare

@sblattj sblattj released this 10 Oct 22:03

Fixes #137 — kiro: context tokens use an assumed 200k window and the last percent, so start size reads wrong
Fixes #126 — Release: npm stuck at 0.16.1 (0.17.0 to 0.19.0 never published), so the homebrew job fails and bun add -g installs a stale ach
Fixes #135 — Exit reason picks a routine INFO log line instead of the real stderr error
Fixes #133 — kiro (headless v3): nonzero exit hides the CLI's stderr reason, e.g. monthly usage limit
Fixes #131 — Kiro tap leaks HTTPS_PROXY into the agent's shell tools; other HTTPS clients get 502
Fixes #128 — Piped stdout is truncated at 64 KiB (binary) / 128 KiB (bun from source) since 0.17: ach dash --json --all | jq gets invalid JSON
Fixes #127 — kiro ACP transport: --kiro-engine v3 dies in initialize; kiro-cli acp rejects --agent/--model/--effort/--trust-all-tools with v3

Fixed

  • kiro: the context window is now derived instead of assumed (#137). A v3 run's context_usage breakdown gives the window from its largest category (tokens / percent), snapped to a known size, so Opus 5.5 and auto runs read a 1M window (windowSource: 'derived') instead of the assumed 200k, which read context about 5x low. Summing the breakdown's token counts is wrong after the first update, because kiro zeroes some category tokens while keeping their percent. usage.context also gains startPercentage/startTokens (the first reading, the session's start size) and an optional breakdown; percentage/tokens stay the end reading, and the run summary prints both. Window precedence: session store, then derived, then assumed.
  • npm stuck at 0.16.1 (#126): the release workflow now publishes the tagged version to npm (npm job, provenance, idempotent, fails loudly without the NPM_TOKEN secret), and the homebrew job waits on it. A backfill of an older tag (gh workflow run release-binaries.yml -f tag=vX.Y.Z) publishes under the backfill dist-tag, so latest never moves backwards. See docs/PACKAGING.md.
  • The nonzero-exit reason no longer picks routine log lines (#135). A headless kiro-cli run that exited 1 read kiro-cli exited with code 1: [INFO] [KRS] <-- GenerateAssistantResponseCommand done totalEvents=9, because the fallback took the last stderr line. pickStderrReason (src/adapters/shared.ts) now never picks INFO/DEBUG/TRACE-tagged lines ([INFO], DEBUG, level=trace). It prefers the last line that looks like a failure (error, failed, not offered, denied, exceeded, unauthorized, forbidden), then the last WARN line, then the last other non-empty line. When only routine lines remain, the message stays bare. The stderr tail grew from 5 to 20 lines (still 4 KB per line), so trailing INFO lines no longer push the real error out.
  • A CLI child that exits nonzero now reports why (#133). The exit error used to say only kiro-cli exited with code 1, so a kiro-cli headless run that stopped on The monthly usage limit has been reached looked like a crash. The shared run loop (src/adapters/shared.ts) keeps the last few stderr lines and appends the most relevant one, ANSI codes stripped: the last line matching error, else the last non-empty line, as in kiro-cli exited with code 1: Error: Internal error (code -32603): ... The monthly usage limit has been reached. The message prefix is unchanged, so existing matchers still work. When an adapter's own stderr hook already raised a specific error (cursor), the exit message stays bare, so the reason is not reported twice. Exit 0 is unchanged.
  • kiro: the MITM credit tap no longer breaks HTTPS from the agent's shell tools (#131). kiro-cli hands the tap env (HTTPS_PROXY, SSL_CERT_FILE, NODE_EXTRA_CA_CERTS) to every command the agent runs, and mitmdump intercepted every host. A host whose certificate chain mitmdump could not verify (for example behind a TLS-inspecting network proxy) came back as HTTP 502, so another vendor's CLI run from the agent's shell failed on every call. mitmdump now gets --allow-hosts for the metered hosts only (codewhisperer.*.amazonaws.com, runtime.*.kiro.dev, extendable with ACH_KIRO_TAP_HOSTS), and other hosts are tunneled raw to their origin. SSL_CERT_FILE now points at the system roots plus the tap CA instead of the bare tap CA, so OpenSSL clients can still verify tunneled hosts. See docs/KIRO.md.
  • Piped stdout is no longer truncated under Bun (#128). ach dash --json --all | wc -c stopped at 64 KiB in the compiled binaries and 128 KiB from source, because the CLI called process.exit before the pipe drained: under Bun the empty write("", cb) guard fires its callback while earlier output is still queued. The CLI now records stdout/stderr backpressure (a write() that returns false) and waits for drain before every forced exit (src/cli/stdio-flush.ts), including watch, serve and the live dash shutdown paths.
  • kiro: --kiro-transport acp --kiro-engine v3 no longer dies in initialize (#127). kiro-cli rejects --agent, --model, --effort, --trust-all-tools and --trust-tools with --agent-engine v3 on acp, so for v3 only --agent-engine v3 is passed. After session/new (or session/load) the agent (a mode), model and effort are set with session/set_config_option, because v3 does not implement session/set_model. Each value is checked against the options v3 sends back, since v3 accepts unknown values without an error. An agent the session does not offer fails before any prompt, and the error suggests --kiro-transport headless. A fresh workspace can load its model catalog after session/new, so the handshake waits up to 15 s for it. kiro.tools is applied by answering session/request_permission (all allows, a list allows the listed tool ids or ACP kinds, none or unset denies). The run evidence records engine, effortAck, currentEffort and toolTrust. Verified live on kiro-cli 2.29.0.