coding.sh 0.1.6

Added

• Privacy. CODING_SH_PURGE_WAIT_MS sets how long purge waits for another process's read of the index or the blob pack before it says which file still holds the old bytes. The defaults stay 10 s before the VACUUM and 5 s after it. Configuration.
• Trace format. A JSON Schema (draft 2020-12) for AgentEvent and Session, generated from the TypeScript types, ships as coding.sh/schema.json and is served at https://coding.sh/schema.json. Every adapter and OTLP fixture validates against it. Trace format
• Serve. CODING_SH_FORWARD_URL (or [serve.forward]) makes coding.sh serve forward every trace it commits to an OTLP/HTTP collector, with the same trace and span ids plus coding_sh.project. Native batches go out as invoke_agent, execute_tool and chat spans. Metadata mode sends no messages, tool arguments or results; a down collector never slows ingest, spans wait in a bounded buffer, and overflow is counted in coding_sh_forward_spans_total. Forward and sinks
• Sinks. [[sinks]] now mirror events from the OTLP receiver (coding.sh collector) and from coding.sh serve, so a team can set one sink on serve instead of on every laptop. Config
• Serve. createFetchHandler() from coding.sh/server runs serve's ingest routes (OTLP, /v1/ingest, /readyz) as a Request → Response function for Vercel Functions, Deno, Bun or Workers. With coding.sh-store-postgres, several instances can write one session at once: each session rebuild takes pg_advisory_xact_lock on the session id. Serverless
• Evals. Cases and tool-eval tasks take an optional kind: bad (fails before the change, passes after), guard or control (pass on both). exp run, exp compare, toolevals run and toolevals compare split results by kind and name broken controls as overcorrection, and toolevals validate --against <variant> reports a bad task that passes on the old variant. See Case kinds and Task kinds.
• Tool evals. coding.sh toolevals traces <server> --sessions 'since:30d' (or --from <experiment>) prints each MCP tool's calls, errors, repeats and retries after an error, and --out <dir> writes the failing traces first (at most 3 passing), an index.jsonl, tool-health.json and tools.json for an improver to read. Held-out trials are never exported. Tool evals
• Tool evals. toolevals run --shell-model <provider/model> has a model play the shell for a model driver on a CLI target, with a fixed date; the shell model is part of the arm name and compare refuses arms with different shells. Tool evals
• Tool evals. The model driver's MCP client reads resources and prompts, counts progress notifications, runs task-augmented tool calls, answers sampling with the driver's model, elicitation with a task's scripted [mcp] answers and roots with the sandbox, and runs OAuth (discovery, client registration, PKCE or client credentials) on a 401. calls checks can match resources/read by uri and prompts/get by name. MCP client
• Tool evals. The model driver caches the prompt on Anthropic (top-level cache_control; promptCache = false turns it off) and reads the final cache counts from the stream; the report shows cost with cache prices beside flat prices, the cache hit share, and says when the cheaper arm flips. Tool evals
• Cache. diff-calls gives the line where a system prompt edit starts and the share of the prompt after it that the cache could no longer read (system.firstChangeLine, system.wastedPrefixFrac in --json and the viewer API). When over 20% is lost, Why says to move dynamic content such as timestamps to the end of the system prompt. Capture with the proxy
• Experiments. compare --signals checks whether each session flag and tool errors predict failure beyond task difficulty: pooled, within-task and within-task-and-bundle AUROC with a task-cluster CI, beside difficulty, bundle and length references, and a verdict (adds-run-signal, tracks-difficulty, no-signal, too-few-pairs). Evals guide
• Experiments. exp sweep and a matrix spec in exp run, with three or more configs or cells, name the plausibly best set: the configs not significantly worse than the best, by a paired-bootstrap multiple comparison with the best, which holds the truly best config 95% of the time. Evals guide
• Experiments. compare --pick N --pick-by irt fits a Rasch (or 2PL) model on stored runs and runs first the tasks with the most information at the baseline's ability: mid difficulty, sharp discrimination. Evals guide
• Experiments. exp run --record saves each case-trial's Anthropic and OpenAI calls, and exp run --replay <EXP> serves them again up to the first changed call (--replay-until N stops after call N), so a late change re-runs only the calls it touches. System-prompt and cache_control edits still replay; API keys are never stored; with one task, the recorded config runs as a control arm and compare prints its noise floor. See docs/how-to/replay-llm-calls.md.
• Experiments. exp run --seq K --feedback bit|reasons|critic=<grader> tries a failing case-trial again with feedback on its earlier attempts, up to K attempts, and reports Seq@1..K with intervals, EGS/LGS, unchanged revisions, cost to solve and, with 2+ trials, pass@k against seq@k by cost. The pass rate stays the first attempt's; feedback never shows the expected answer; exp compare adds a seq@K row and --verdict seq (arXiv 2610.37859). Evals guide.
• Experiments. exp run --failing <EXP>, --errors-from, --case <PATTERN>, --sample N --seed S and --first-n run part of a dataset. When B ran only on A's failures, exp compare warns of regression to the mean, says inconclusive and --gate holds until A's config runs on the same cases. Evals guide.
• Experiments. exp run stores the experiment so far while it runs; after a crash exp list marks it (partial), and exp run … --resume <experiment> runs only what is left (--retry-errors also re-runs its task errors). A resume with other graders, trials, task, dataset version or case selection is refused and names what changed. Evals guide.
• Experiments. exp sweep --guard <dataset>|prior re-runs the start and winning configs on earlier suites and keeps the start config, exit 1, when the winner breaks more than --guard-tau of what the start solved there, even when the target suite ships. Shipped sweeps are remembered in adoptions.jsonl for --guard prior. Evals guide.
• Experiments. exp compare takes --pairwise more than once: it reports each judge, Cohen's κ between judges, and replicated-better, judges-disagree or unresolved, screens out a judge whose calibration failed, and with --gate holds a ship when any judge prefers A or two judges do not both find B better (arXiv 2611.04090). Evals guide.
• Experiments. exp compare --labels-a --labels-b corrects each LLM judge grader's B − A with human labels of each arm's outputs (PPI++), and warns when the judge errs differently on the two arms. Evals guide.
• Refine. coding.sh refine mine ranks failure patterns across sessions with no model call: session flags, normalized error lines (20 TS2307 errors in 5 sessions are one pattern), instructions you typed again and again, and failed eval runs, each with its count, sessions and example events, redacted. refine propose --mined FILE shows the top patterns to the proposer. Mining failure patterns.
• Refine. coding.sh refine diagnose asks a model what went wrong in recent sessions and groups the answers into Signals with stable ids; refine signals lists them. Every problem needs a citation that code checks against the redacted trace, severity is computed by code, replies are cached so a rerun makes no call, and with no API key it prints a $0 plan of the sessions and tokens it would send. [Diagnosing sessions into Signals](https://github.com/jackulau/coding.sh/...

coding.sh 0.1.5

Added

• Sessions. A new flag, unapproved-action, marks a session where the agent ran a destructive or paid command after a user turn that did not approve it: rm -rf outside temp and build dirs, git reset --hard or git push --force, dropping a database, a tool's own --yes on a repair, reinit, migrate, delete or purge subcommand, deleting a data dir, or a known paid cloud or batch job. Asked only to "fix it", agents run the --yes their tool's error names. stats.unapprovedActions counts them, and --where unapproved>0 lists the sessions. Flags.

Changed

• Refine. refine test|run --suite keeps a candidate whose pass rate is significantly better even when a listed objective such as cost got significantly worse, as long as the rise is within --max-cost-rise (default 50%, or refine.maxCostRise in config). It used to reject +27 to +67 pt gains because each run cost a few cents more. The verdict prints the trade, such as +33.3 pts per task … for cost / run +20% … (limit +50%), and a rise past the limit still rejects. The gain must still pass the exact permutation test. Climbing on an eval suite.
• CLI. An error whose next step deletes or overwrites data, cannot be undone, or spends money now says the user must approve it first, and its --json line has askUser: true. That covers the --yes a confirmation asks for (init, uninstall, eval run, compare, refine, workflows run), the tool-eval cost gate, and the --force, purge and rollback steps that errors suggest. The text no longer says "Did you mean" before a --yes command. Asked only to "fix it", agents ran the --yes an error named without asking. Output and exit codes.
• Store. A blob that compresses to under 4 KiB, such as a hook line's raw payload, is a row of one SQLite file, blobs/pack.db, instead of a file of its own; larger blobs stay files. On the 100k-event ingest benchmark the store takes 1.72 MB per 1,000 events instead of 5.51, and ingest runs at about 6,200 events/s instead of 1,700. Blob files written before still read, and gc --compact moves the small ones into the pack and removes the blob folders that leaves empty. A coding.sh older than 0.1.5 does not see packed blobs: its CLI reads sessions as usual, and its viewer shows their stored content as "not in the local store". Run purge with 0.1.5 or newer: an older purge leaves secrets in packed blobs, and they show again once you upgrade. Running the same purge again with 0.1.5 removes them, since it reads every packed blob. Storage, Store ingest.

Fixed

• Store. A coding.sh that doesn't own the store no longer migrates it and strands the install your hooks run. The owner is the CLI ~/.coding.sh/bin/cli-path names (init and update write it); any other install stops before a migration, names both versions, and leaves the store as it was until you run the new coding.sh upgrade-store, which also points the hooks at it. Upgrades in place and coding.sh update migrate as before. coding.sh run from a source checkout also no longer opens ~/.coding.sh unless CODING_SH_HOME, --store or CODING_SH_DEV_REAL_STORE=1 says to. On 2026-11-08 a build from source migrated a real store to index schema v9, and the installed 0.1.2 could not open it for 10 hours. Troubleshooting.
• Sessions. A session coding.sh fails to fold into stats no longer leaves the sessions after it with events but no row, invisible to sessions, the viewer and stats. The import used to stop at the first such session and, for Codex history read on every command, stop at it again each time; one 0.1.4 store had 906 Codex sessions like that. The session now gets a row tagged fold:error, with the error in stats.foldError, and the import goes on. coding.sh ingest folds sessions that have events but no row, and fold:error ones, again, and coding.sh doctor counts both. Flags, Session stats.
• Sessions. A shell command named like a built-in object key (constructor, toString) no longer stops a session from being folded with PAID[tool]?.test is not a function. Flags.
• Sessions. sessions, show and the viewer no longer crash with Cannot read properties of undefined (reading 'briefs') after another build of coding.sh worked out the same store. Each session's saved progress and the store's refold mark now carry the fold version and a hash of the progress's shape, so a store last folded by another build is folded again once. Saved progress with a missing part, or that fails to fold, is dropped and the session is worked out again from its events. Troubleshooting.
• Evals. An agent that is not logged in, or a setup step that fails, no longer counts as a task failure. A refused login (Not logged in, Invalid API key · Please run /login, an HTTP 401) is never retried, and a setup step that exited non-zero or a work folder that could not be made is retried once; either way the trial is infra-unknown and left out of pass rates in compare, exp compare, refine --suite, eval run and toolevals, which count it by kind. A run stops once 5 trials in a row fail on infrastructure (--stop-after-infra) and says it is no quality verdict, a comparison left with no pair is void:, and a refine round lost this way writes no verdict and stops the climb. An expired login had read as "quality held". Long runs.
• Evals. A few cases, tasks or prompts that all move the same way no longer pass a gate on their own. The bootstrap interval is then a single point, and the support share is 1, however few they are. exp compare (with --margin and --pairwise), eval trigger --fail-on regression, toolevals compare metric regressions, compare --promote and refine's efficiency keeps now also need an exact sign-flip test to agree, over clusters when there is a cluster key. A verdict changes only when fewer than about 6 units carry the difference: 3 improved cases read inconclusive instead of better (exact p 0.25), and one improved task no longer promotes. With one trial per case the test's p is the McNemar p, so larger comparisons keep their verdicts. JSON output gains signFlip, p or exact fields. Audit, Paired comparison.
• Privacy. purge no longer reports success while another process (the viewer, serve, a sqlite3 shell, a backup) holds a read on the index or the blob pack. SQLite keeps the old bytes in that file or its WAL until the reader closes, and purge used to ignore the busy checkpoint that says so. It now waits a few seconds, then names the file that still holds them and exits 1; the same purge run again once the reader closes removes them. Privacy.
• Sessions. sessions --where id:<id> and export --where id:<id> find a subagent's session, such as a Claude Code workflow agent's, as show <id> does. Both commands leave out subagents unless you pass --subagents, and that default hid every session the query named. A query with id:, parent:, is:root or is:child now decides that itself, so --where is:child lists subagents too. Query and export.
• Workflows. A Claude Code workflow run that is still going is named by its script's meta.name, read from the copy of the script Claude Code keeps in the session's workflows/scripts/ folder, instead of by its wf_… id. Claude Code writes the record, which held the only name coding.sh read, when the run ends. workflows import reads runs it stored before this fix again once. Workflows.
• Capture. The hook no longer reads .git on session-start and prompt events, which made each of those hook calls about 3.5 ms slower than in 0.1.2 in a repo with a 1 MB packed-refs. Ingest takes a hook session's start commit from the reflog again, and still uses the HEAD a 0.1.3 or 0.1.4 hook recorded. CODING_SH_NO_GIT_HEAD has nothing left to turn off. Architecture.
• Cost. A Claude Code fork subagent (subagent_type: "fork") no longer counts the parent's model calls again. Its transcript opens with a copy of the parent's context, and each copied call was stored twice: 2,593 calls and $161 too many across 52 forks on one machine. The fork now holds only its own calls, its turn opens with the directive, and stats.inheritedTokens says how much context it started with. Stored sessions are parsed again. Subagents.
• Harness A/B. compare counts cache reads and writes in tokens / run. With prompt caching the row tra...

coding.sh 0.1.4

The 0.1.3 changes, published to npm.

Fixed

• Release. 0.1.3 never reached npm: the publish job passed npm a tarball path that npm read as a GitHub repository. 0.1.4 carries all of 0.1.3's changes, and the release checks now dry-run the npm publish before it is approved. Releasing.

coding.sh 0.1.3

Sessions read back from Claude Code and Codex transcripts keep more of what the agent recorded.

Added

• Docs. Two guides for one agent each. coding.sh with T3 Code covers what is read from T3 Code's local store for each agent it runs, and how to turn a thread into an eval run by Codex or Claude Code. coding.sh with Pi covers the Pi extension, Pi's own cost per model call, and evals with --agent pi. T3 Code, Pi.
• Workflows. workflows stats --json gives each per-agent row the agent id, the attempt (a label run again after a resume is attempt 2) and how it ended (endReason: ok, failed, cancelled, superseded, null, schema-retry, …), so two agents with the same label can be told apart. The text table adds #2 to a re-run label. Workflows.
• Improve. improve and the MCP improve tool list what lost prompt cache cost, from the cache breaks ingest already stored: the extra dollars, their share of the spend, which sets the severity so the item ranks against the others by cost, and a fix per cause, costliest first. Improve.
• Sessions. A new coding.sh tree <session|run> command shows a whole multi-agent run as one tree: the top-level agent, every subagent it spawned, and their subagents, with per-node model, cost, tokens, duration and tool calls, the run's rolled-up totals, the worst flag across the loop, and which subagent to fix first. A fan-out of separate agent processes groups into one run when they share CODING_SH_RUN_ID. The viewer serves it at /api/sessions/{id}/tree and the MCP session_tree tool returns it for your agent. It is built from the parent links capture already records, so no session re-folds. Explore.
• Capture. A tool output the agent cut before the model saw it is marked on its tool.end as payload.cut: Codex's head-and-tail cut of a long command or code-mode result (with the original token count, and jsonBroken when cut JSON no longer parses), and Claude Code's <persisted-output> preview of a result saved to a file (with its size and path). Backfilled sessions only for now; a hook-captured call keeps the hook's copy. Adapters.
• Capture. API errors Claude Code retried keep what the transcript says about them: the HTTP status, the error type and class (rate_limit, overloaded, …), the rate limit that was hit (seven_day, five_hour), when it resets, how long Claude Code waited and which attempt it was. Before, every one was a bare "api error", so a three-day weekly-limit stall looked like one transient error. Read API.
• Cost. A Claude Code backfill keeps Claude Code's own cost tally (the transcript's cost-state line: total cost, API time and cost per model) on the session's session.end as payload.reportedUsage, so you can check the computed cost against it. It is never added to the session's cost. Where cost comes from.
• Capture. The hook records the repo root, HEAD commit and branch on session-start and prompt events, read from .git files without running git (worktrees, submodules, packed refs, detached and unborn heads). A hook session's start commit, which eval new starts from, is now the HEAD when the session started. Before, ingest looked it up in the reflog and, when the reflog had nothing, took the HEAD at ingest time. CODING_SH_NO_GIT_HEAD=1 turns it off. Configuration.
• Capture. Before coding.sh init, a read now works with no setup: usage, sessions, show and the viewer import the Claude Code transcripts already on disk, so coding.sh usage shows your history straight away. Once the hooks are installed, live capture takes over and the transcripts are not re-read on every read. Adapters.
• CI. A GitHub Action in this repo (uses: jackulau/coding.sh@<ref>) installs coding.sh, runs one command such as eval run triage --yes or exp compare base head --margin 0.05, writes the report to the step summary and to one pull request comment that later runs edit, and fails the job when the command fails. eval run --summary <file> appends the report as Markdown, and the workflow eval init prints uses the action. Catch a regression in CI.
• MCP. A harness_audit tool and a trim prompt let your agent audit your setup and trim what goes unused: each memory file, CLAUDE.md, AGENTS.md, rule and skill with the sessions that loaded and read it and its load tokens per session, unused ones first. With check: true it runs refine check on your uncommitted edits, and it returns the refine run command instead of running it. Shorter shared argument descriptions keep the tool list within its byte budget. Memory and instruction files.
• Harness. harness audit warns about Claude Code subagent files whose tools: frontmatter is a map such as edit: false, which leaves the subagent with no tools, and gives the disallowedTools: line to use instead. --json lists them under agentFiles. Explore guide.
• Capture. coding.sh statusline prints a Claude Code status line — model, this session's cost, today's cost and the worst failure flag — from your stored sessions. statusline install wires it into the Claude Code settings.json and statusline uninstall restores the previous one byte-for-byte. Status line how-to.
• Refine. refine test --suite and refine run --suite take --reuse baseline (or both), copying an earlier round's finished runs of the unchanged base arm instead of running the agent again, so a long climb spends roughly half the agent sessions. Reuse is keyed by content like compare --reuse, copies count $0 toward --max-cost, and the round warns that the reused arm did not run interleaved; the verdict is unchanged because the copied runs are identical. Climbing on an eval suite.
• Cache. coding.sh usage --cache turns the cache rate into advice: per agent, model, day or repo it shows the hit rate, cache-read vs cache-write tokens, the read-to-write ratio (below 1:1 you wrote more cache than you read back), the prompt-cache breaks and what they cost, and names the break causes to change — compaction, a mid-session model switch, or an idle gap past the cache lifetime. Every number is a captured token count or the per-call-priced extra the breaks cost; there is no blended-rate guess. A session that wrote a cache of at least one block over three or more turns but read back fewer tokens than it wrote is tagged cache:low, and --where cache.breaks>0, cache.lost and cache.cost query and sort on the breaks (coding.sh sessions --where 'cache.breaks>0 sort:-cache.cost'). exp compare gains cache breaks / run and cache waste / run metrics so an A/B shows whether a setup change reused the cache better, and the viewer shows the read-to-write ratio next to a session's cache rate. Explore.
• Capture. Claude Code's model_fallback and model_refusal_fallback transcript records (the main model was overloaded and Claude Code moved to --fallback-model, or a refusal went to another model) become a model.fallback notification with both models and the trigger, and tag the session model:fallback. Adapters.
• Cache. Idle cache breaks say where the wait was, in stats.cacheBreaks.byIdle and as one usage --cache line each: a foreground tool call that ran past the cache lifetime (wait: keep foreground waits under the 5-minute subagent cache lifetime and poll in chunks), a subagent idle between turns (subagent), or the main session idle between turns (away: unavoidable while you are away; the 1-hour cache helps if you return often). Read API.

Changed

• Site. The top bar links only Docs and Changelog. The comparison pages stay in the docs sidebar and the blog stays in the footer. Site.
• Setup. coding.sh init ends by saying that recording starts with the next agent session, since an agent loads hooks when a session starts. Once capture is set up, an empty store no longer sends you back to init: show latest, sessions, status, improve, the MCP status and session_current tools, and the viewer ("Waiting for your first session") say to start a new agent session and to run coding.sh doctor if none shows up. Install
• Setup. The installer run without a terminal (an agent's shell, CI) and without CODING_SH_YES=1 says no agent was set up and name...

coding.sh 0.1.2

coding.sh runs on Windows, and CI tests it there.

Added

• Sessions. A new text-tool-call flag marks a session whose model wrote a tool call out as text, so the call never ran. In Claude Code this is usually a subagent whose tools: frontmatter is a map (edit: false), which leaves it with no tools; improve names the fix. Concepts.
• Sessions. A session whose main thread moved to another model mid-session, such as Claude Code's silent fallback to an older model after a refusal, gets the model:switch tag and stats.modelSwitch with both models, when, and the trigger (refusal, error or /model). stats.models counts calls per model (both show in a Models row in show and the MCP digest), a refused call adds the refusal flag, and a session's model is now the one most of its main-thread calls ran on instead of the last one seen. Helper models (Haiku titles) and subagents are not a switch. Read API.
• Evals. eval validate takes several tasks, or --tag <tag> for every task with that tag (suite:<name> for a suite), prints one line per task and exits 1 if any did not validate. Evals.
• Evals. eval init --sessions --repo <DIR|all> drafts tasks from sessions in another checkout or worktree of the project, not only the current one. Evals.
• Docs. Use-case pages for tracking Claude Code and Codex cost, replaying a session, testing a CLAUDE.md, AGENTS.md or skill change, building evals from failures, self-hosting, and logging model calls through the proxy. Comparison pages for Langfuse, LangSmith, Braintrust, AgentOps, Laminar and ccusage, each with a feature table, where the other tool is the better choice, and its sources. Compare.
• Site. Every docs page has its own link preview image, the home page and FAQ carry FAQ markup, and the site build fails on a missing or too-long title or description, a wrong canonical URL, or a link to a page that doesn't exist.
• Evals. eval validate checks verifiers that grade the agent's answer or trace: a task dir's reference/ (output.md, stdout.jsonl) is a correct run that must pass, and near-misses/<label>/ hold wrong runs that must fail. The verifier reads them through CODING_SH_AGENT_OUTPUT, CODING_SH_AGENT_STDOUT and CODING_SH_AGENT_DIFF, as in a real run. Evals.
• Evals. A hand-written task can start from a commit with "start": {"commit": "<sha or ref>", "repo": "<path>"} in task.json, so a suite can live in the repo it tests. Evals.
• Capture. coding.sh init imports the sessions each agent already keeps on this machine, so the viewer the installer opens shows your own history. It shows progress, stops after 8 seconds and imports the rest in the background; the viewer says "Importing past sessions…" until it is done. --no-import skips it. An empty viewer shows the import command, coding.sh ingest --backfill, and what each imported session shows. What init changes.
• Examples. examples/plugin-ab A/B tests a Claude Code plugin against plain Claude Code on ten small tasks in a container, with coding.sh compare: pass rate, tokens, cost split by token type, wall time, reply length and a blind clarity judge, paired by task with bootstrap intervals. It takes any plugin directory and ships a small demo plugin. The example.
• Improve. coding.sh improve lists what goes wrong most in this repo's sessions (failure flags, and any command failing the same way in two or more sessions), each with an example session, the CLAUDE.md or AGENTS.md line that fixes it and the next commands. improve fix N adds the line, uncommitted. improve check compares the sessions before and after each committed CLAUDE.md, AGENTS.md, rule or skill change, with how often the agent follows each new line, says Helped, Regressed, No clear change or Too early, and exits 1 on a regression; --watch re-checks as sessions arrive. No API key needed. Improve your agent every week.
• MCP. improve and harness_changes tools, and improve and did-it-help prompts, run the same loop from inside your agent: what to fix first, and whether a CLAUDE.md change helped, without a shell. Improve your agent every week.
• Harness. coding.sh harness audit lists each memory and instruction file (Claude auto-memory, CLAUDE.md, AGENTS.md, rules, skills) with the sessions that loaded it, read it and wrote it, its size and its load cost in tokens, and flags files never read, written but never read back, and skills listed but never used. --agent, --since (default 30d) and --json. Memory and instruction files.
• Sessions. Text that Claude Code hooks add to the context (a plugin's SessionStart notes, a UserPromptSubmit reminder, PostToolUse additionalContext) is stored as hook.context events with its hook event, source and size. stats.hookContext counts the tokens by source, each source tags the session (hook:plugin:caveman, hook:settings), sessions --where hook:plugin:<name> or hook.tokens>500 filters on it, --facets hook counts sessions per source, and show lists it. Concepts.
• Sessions. Headless sessions (claude -p, an Agent SDK, codex exec) are marked stats.launch: "headless", and terminal or IDE sessions "interactive", from Claude Code's entrypoint and Codex's originator. sessions --where launch:headless filters on it and --facets launch counts it. Concepts.
• Improve. improve check lists, under each change, the sessions still flagged after it with their flags and start time (five, or all with --sessions; every flagged session before and after in --json), and the MCP harness_changes tool returns them as after.flaggedSessions, so your agent can answer "which sessions still failed after it?". Did it help?
• Sessions. show has a Flags row, and sessions ends with how many listed sessions are flagged and a pointer to coding.sh improve.
• Evals. coding.sh eval init writes an eval suite to evals/<name>/ from failed or flagged sessions in the repo (--sessions, --session latest), a skill's should and should-not prompts (--skill), or cases for an app (--cases, --describe with --command or --url). --draft has a model write the prompts or cases, marked draft. It picks a code grader when it can and a judge to calibrate when it can't, marks the suite capability or regression, and ends with the run command and, for cases and skills, a GitHub Actions workflow. On a terminal with no flags it asks. Your first eval in 5 minutes.
• Evals. coding.sh eval run [suite…] runs suites k times per task and reports pass@k, pass^k, flaky tasks, the session of each failed trial, small suites and saturation. A regression suite exits 1 when a task fails most of its trials or nothing could be graded; a capability suite never fails on its score. The report.
• MCP. An eval_init tool lets your agent build an eval suite for you, with the same sources as eval init; it writes files only with confirm: true. MCP server.
• Evals. eval import warns when a verifier names a tests/ path but the hidden files land at the work-tree root (tests/check.js is copied to check.js). Such a verifier finds no file, or runs zero tests and passes. Evals.
• Judges. LLM graders take "provider": "claude-code", which grades through claude -p on a Claude subscription (CLAUDE_CODE_OAUTH_TOKEN from claude setup-token) instead of an API key. The judge runs with no tools, setting sources, plugins, MCP servers or saved session, so the hooks of the arm under test never reach it. Evaluate an Agent Skill.
• Stores. The Postgres and ClickHouse store plugins answer --where in the database, with every value bound: session and event keys, has:, is:, bare words, sort: and --facets, for sessions, usage, export, the viewer API and MCP. sessions --where and facets no longer stop at the newest 20,000 sessions there, and event keys and bare words no longer fail with unsupported_store. Query language.
• Harness A/B. Each run keeps the end of its verifier's output (the last 4096 characters of stdout and stderr, colour codes removed, redacted with capture.redact), each repair round too. compare and eval run print the last lines of failed runs under the report, and --json lists them in failedRuns. [Verifier out...

coding.sh 0.1.1

Daily benchmarks for any AI agent: coding.sh bench runs every contender on fresh cases each day,
and experiments add cluster bootstrap, an A/A noise floor, interleaved runs, early stopping and compare --without.
Seven more agents are verified on a real install, the viewer and the site work on phones, and the installer uses npm once the package is there.

Added

• Benchmarks. coding.sh bench samples fresh cases each day from a command, failed sessions, GitHub issues, a sitemap or a dataset, freezes them as a dataset version and runs every contender on that set in random order. Benchmarks.
• Benchmarks. Shared timeout and retries, a budget cap, quota and rate-limit cases dropped for every contender, and bench run --resume to finish a stopped run with the same seed and cases.
• Benchmarks. bench report writes a static HTML page and JSON: trend, gaps to a reference pooled over a fixed window, head-to-head stats, difficulty and failures. bench regrade re-scores stored outputs and bench schedule prints a cron line and a GitHub Actions workflow.
• Benchmarks. An [anchor] set of fixed cases runs every day next to the fresh ones and is scored apart, so its trend has no day-to-day difficulty in it.
• Benchmarks. seed = "secret" publishes the hash of each next day's salt and bench show checks the chain. Content hashes keep the same case under a new key from counting as fresh.
• Benchmarks. Input and output tokens are kept apart, and each cost says where it came from: reported, costPerRun, the shared price table, or unknown.
• MCP. A benchmarks tool lists benchmarks and reads results and history.
• Experiments. exp compare --cluster <key> resamples by repo or session and warns under 10 clusters. Evals.
• Experiments. exp aa measures the run-to-run noise floor, and power planning uses it.
• Experiments. Compare flags uneven attrition between arms (Fisher exact) and splits infra errors from agent errors.
• Experiments. Under 100 paired cases, compare also reports P(B>A) from a paired posterior.
• Experiments. exp run --arm a=<task> --arm b=<task> interleaves both arms case by case in a seeded random order.
• Experiments. --stop-early checks at planned looks with O'Brien-Fleming alpha spending and reports a repeated confidence interval.
• Experiments. Pairwise judges see both outputs in both orders. exp compare --pairwise reports win rate, ties and position bias.
• Experiments. exp sweep races task configs on a train split and reports the winner's optimism gap on held-out cases.
• Compare. compare <bundle> --without <path> measures what one skill, AGENTS.md, CLAUDE.md or MCP config is worth. Compare agents.
• Compare. With 2+ attempts per task, failed runs split into bad runs, arm limits and tasks no arm solves, with variance shares.
• Compare. Cost rows show how many runs had a known cost, and "cheaper" needs a known cost for 80% of each arm.
• Agents. Verified on a real install against a local stub model: opencode, copilot-cli, droid, goose, pi, t3code and aider. Agent matrix.
• Agents. Antigravity backfill reads agy transcripts and conversation DBs for model, tokens, cache reads and latency per call. Tool steps missing from the transcript count as failures, and hooks reply {}.
• Viewer. Phone layouts: 44px targets, filters in a bottom sheet, a compact session header and safe-area insets.
• Viewer. The Benchmarks tab shows anchors, pooled gaps, difficulty, cost sources and tokens, and ranks follow the pooled gaps.
• Site. Mobile pass: touch targets, safe areas, stacked agent cards and 16px body text on phones.
• Site. A new landing page, a changelog page and docs pages.
• Install. install.sh installs from npm when the registry has the version, else from the GitHub release. CODING_SH_SOURCE=npm|github picks one.
• Release. A tag creates its GitHub release as soon as the checks pass, so the installer gets it right away.
• Release. The release workflow can publish all five packages to npm and PyPI with trusted publishing. Nothing is on npm or PyPI yet.
• Docs. A new README, reference pages, how-to guides and a code of conduct.

Fixed

• Fixes. Safer installs: symlinked and JSONC agent configs survive, init --dry-run prints a short plan, and blockers show before the Y/n.
• Fixes. Each SQLite migration runs once when the hook and the CLI open a new index at the same time.
• Fixes. Experiments stop cleanly on Ctrl-C, kill their child processes and keep every finished run.
• Fixes. Capture, ingest and storage state bugs from an audit of each state machine. Sessions are priced locally at ingest.
• Fixes. Adapter fixes for opencode, goose, copilot-cli, droid, aider and t3code, found running each on a real install.
• Fixes. coding.sh run --agent <name> records sessions for names with no adapter.
• Fixes. exp aa refuses to compare a run with itself.
• Fixes. coding.sh ui prints the viewer URL once in a terminal.
• Fixes. Muted text in the viewer passes WCAG AA in both themes.
• Fixes. Timing tests compare against a baseline, so a loaded machine no longer fails them.

coding.sh 0.1.0

First public release, pre-1.0. Install with curl -fsSL https://coding.sh/install | sh. It installs the release tarball attached below and runs coding.sh init. The npm package comes with a later release.

Added

• Vercel AI Gateway. Sessions routed through the gateway are tagged gateway:vercel. coding.sh gateway status|enrich adds billed cost, serving provider, client session join and the resolved vmc/ model per call. The receiver maps gateway trace drains and AI SDK gateway spans. compare --gateway and the gateway grader provider run arms across models with one key.
• Install. curl -fsSL https://coding.sh/install | sh checks for Node 22.15+, installs the latest release with npm, runs coding.sh init (one plan, one Y/n) and opens the viewer. After that, coding.sh with no command opens the viewer; on first run it sets up capture.
• MCP server. coding.sh mcp lets agents query captured sessions, failures, latency, usage, subagents and experiments over stdio. coding.sh init registers it with Claude Code, Codex, Cursor, OpenCode and Gemini CLI (--no-mcp to skip). coding.sh uninstall restores each config file byte-for-byte if nothing else edited it since.
• Update alerts. Commands print one dim line when a newer release is out. The check runs in the background at most once a day; coding.sh update installs the release. Off in CI and with NO_UPDATE_NOTIFIER=1, CODING_SH_NO_UPDATE_CHECK=1 or [update] check = false.
• No ingest step. Read commands, the viewer and the MCP server ingest new hook events first, and the hook starts a background ingest (lock-guarded) when a session ends.
• Capture. Hook-based capture for coding agents: prompts, tool calls, file changes, turns and errors, written to a local spool and ingested into sessions. Secrets are redacted before anything is stored.
• Capture. coding.sh run -- <cmd> wraps any agent and takes shadow-git snapshots of the working tree before and after, without writing to your repository.
• Adapters. claude-code, codex, opencode, cursor, gemini-cli, antigravity, copilot-cli, amp, pi, droid, goose, t3code, aider, and the generic wrapper. Each one has fixture tests and an ingest --backfill path where the agent keeps history. Claude Code, Codex and Antigravity are verified on real sessions; the rest are fixture-tested so far.
• T3 Code. The t3code adapter imports T3 Code threads from its local store. Codex and Claude Code sessions that T3 runs become one t3code session, with the underlying agent in provider, instead of being counted twice. coding.sh usage --by agent-model compares tokens per turn and cache-read share across harnesses.
• Subagents. Subagent starts and ends are captured and linked to their child sessions.
• OTLP receiver. coding.sh collector accepts OTLP/HTTP traces and logs (protobuf or JSON) on 127.0.0.1:4318. It understands OTel GenAI semconv, OpenInference, the Vercel AI SDK and Claude Code / Codex / Gemini CLI log events.
• Latency. Every model call becomes an llm.call event with model, token usage, duration and time to first token.
• Storage. A single SQLite index (node:sqlite, no native deps) plus a content-addressed, zstd-compressed blob store. OTLP and HTTP sinks are metadata-only by default.
• Viewer. coding.sh ui serves a local viewer: sessions, search, a timeline of turns, tools and subagents, and outcome labels.
• Evals. Repo-replay eval tasks that start from the snapshot taken before the agent ran. eval validate checks the verifier: a do-nothing attempt fails, the reference fix passes k of k, near-misses fail, and nothing leaks.
• Evals. Datasets with deterministic graders, command graders and LLM judges. Judges are calibrated against human labels with Cohen's kappa on a held-out split; uncalibrated judges are flagged in every report.
• Experiments. Run two arms on the same cases and compare them with a paired bootstrap CI of the difference and an exact McNemar test on flipped cases. Harness compare adds per-arm Wilson intervals and the first step where traces diverge.
• Tool evals. Evals for CLI and MCP tool surfaces: description size, schema checks, and per-task tool-call metrics against a sandbox.
• Integrations. eve: span enrichment, a read-only importer for eve's local traces, and an experiment task that drives a local eve agent.
• CLI. init, doctor, status, ingest, run, collector, uninstall, sessions, show, search, usage, ui, mcp, eval, dataset, exp, judge, toolevals, workflows, harness, compare, gateway, gc, config, update, version, with --json output.
• Docs. README with a quickstart, the agent matrix, OTLP setup, storage, privacy and uninstall; CONTRIBUTING, SECURITY and a release checklist (docs/RELEASE.md).