coding.sh 0.1.2

coding.sh runs on Windows, and CI tests it there.

Added

• Sessions. A new text-tool-call flag marks a session whose model wrote a tool call out as text, so the call never ran. In Claude Code this is usually a subagent whose tools: frontmatter is a map (edit: false), which leaves it with no tools; improve names the fix. Concepts.
• Sessions. A session whose main thread moved to another model mid-session, such as Claude Code's silent fallback to an older model after a refusal, gets the model:switch tag and stats.modelSwitch with both models, when, and the trigger (refusal, error or /model). stats.models counts calls per model (both show in a Models row in show and the MCP digest), a refused call adds the refusal flag, and a session's model is now the one most of its main-thread calls ran on instead of the last one seen. Helper models (Haiku titles) and subagents are not a switch. Read API.
• Evals. eval validate takes several tasks, or --tag <tag> for every task with that tag (suite:<name> for a suite), prints one line per task and exits 1 if any did not validate. Evals.
• Evals. eval init --sessions --repo <DIR|all> drafts tasks from sessions in another checkout or worktree of the project, not only the current one. Evals.
• Docs. Use-case pages for tracking Claude Code and Codex cost, replaying a session, testing a CLAUDE.md, AGENTS.md or skill change, building evals from failures, self-hosting, and logging model calls through the proxy. Comparison pages for Langfuse, LangSmith, Braintrust, AgentOps, Laminar and ccusage, each with a feature table, where the other tool is the better choice, and its sources. Compare.
• Site. Every docs page has its own link preview image, the home page and FAQ carry FAQ markup, and the site build fails on a missing or too-long title or description, a wrong canonical URL, or a link to a page that doesn't exist.
• Evals. eval validate checks verifiers that grade the agent's answer or trace: a task dir's reference/ (output.md, stdout.jsonl) is a correct run that must pass, and near-misses/<label>/ hold wrong runs that must fail. The verifier reads them through CODING_SH_AGENT_OUTPUT, CODING_SH_AGENT_STDOUT and CODING_SH_AGENT_DIFF, as in a real run. Evals.
• Evals. A hand-written task can start from a commit with "start": {"commit": "<sha or ref>", "repo": "<path>"} in task.json, so a suite can live in the repo it tests. Evals.
• Capture. coding.sh init imports the sessions each agent already keeps on this machine, so the viewer the installer opens shows your own history. It shows progress, stops after 8 seconds and imports the rest in the background; the viewer says "Importing past sessions…" until it is done. --no-import skips it. An empty viewer shows the import command, coding.sh ingest --backfill, and what each imported session shows. What init changes.
• Examples. examples/plugin-ab A/B tests a Claude Code plugin against plain Claude Code on ten small tasks in a container, with coding.sh compare: pass rate, tokens, cost split by token type, wall time, reply length and a blind clarity judge, paired by task with bootstrap intervals. It takes any plugin directory and ships a small demo plugin. The example.
• Improve. coding.sh improve lists what goes wrong most in this repo's sessions (failure flags, and any command failing the same way in two or more sessions), each with an example session, the CLAUDE.md or AGENTS.md line that fixes it and the next commands. improve fix N adds the line, uncommitted. improve check compares the sessions before and after each committed CLAUDE.md, AGENTS.md, rule or skill change, with how often the agent follows each new line, says Helped, Regressed, No clear change or Too early, and exits 1 on a regression; --watch re-checks as sessions arrive. No API key needed. Improve your agent every week.
• MCP. improve and harness_changes tools, and improve and did-it-help prompts, run the same loop from inside your agent: what to fix first, and whether a CLAUDE.md change helped, without a shell. Improve your agent every week.
• Harness. coding.sh harness audit lists each memory and instruction file (Claude auto-memory, CLAUDE.md, AGENTS.md, rules, skills) with the sessions that loaded it, read it and wrote it, its size and its load cost in tokens, and flags files never read, written but never read back, and skills listed but never used. --agent, --since (default 30d) and --json. Memory and instruction files.
• Sessions. Text that Claude Code hooks add to the context (a plugin's SessionStart notes, a UserPromptSubmit reminder, PostToolUse additionalContext) is stored as hook.context events with its hook event, source and size. stats.hookContext counts the tokens by source, each source tags the session (hook:plugin:caveman, hook:settings), sessions --where hook:plugin:<name> or hook.tokens>500 filters on it, --facets hook counts sessions per source, and show lists it. Concepts.
• Sessions. Headless sessions (claude -p, an Agent SDK, codex exec) are marked stats.launch: "headless", and terminal or IDE sessions "interactive", from Claude Code's entrypoint and Codex's originator. sessions --where launch:headless filters on it and --facets launch counts it. Concepts.
• Improve. improve check lists, under each change, the sessions still flagged after it with their flags and start time (five, or all with --sessions; every flagged session before and after in --json), and the MCP harness_changes tool returns them as after.flaggedSessions, so your agent can answer "which sessions still failed after it?". Did it help?
• Sessions. show has a Flags row, and sessions ends with how many listed sessions are flagged and a pointer to coding.sh improve.
• Evals. coding.sh eval init writes an eval suite to evals/<name>/ from failed or flagged sessions in the repo (--sessions, --session latest), a skill's should and should-not prompts (--skill), or cases for an app (--cases, --describe with --command or --url). --draft has a model write the prompts or cases, marked draft. It picks a code grader when it can and a judge to calibrate when it can't, marks the suite capability or regression, and ends with the run command and, for cases and skills, a GitHub Actions workflow. On a terminal with no flags it asks. Your first eval in 5 minutes.
• Evals. coding.sh eval run [suite…] runs suites k times per task and reports pass@k, pass^k, flaky tasks, the session of each failed trial, small suites and saturation. A regression suite exits 1 when a task fails most of its trials or nothing could be graded; a capability suite never fails on its score. The report.
• MCP. An eval_init tool lets your agent build an eval suite for you, with the same sources as eval init; it writes files only with confirm: true. MCP server.
• Evals. eval import warns when a verifier names a tests/ path but the hidden files land at the work-tree root (tests/check.js is copied to check.js). Such a verifier finds no file, or runs zero tests and passes. Evals.
• Judges. LLM graders take "provider": "claude-code", which grades through claude -p on a Claude subscription (CLAUDE_CODE_OAUTH_TOKEN from claude setup-token) instead of an API key. The judge runs with no tools, setting sources, plugins, MCP servers or saved session, so the hooks of the arm under test never reach it. Evaluate an Agent Skill.
• Stores. The Postgres and ClickHouse store plugins answer --where in the database, with every value bound: session and event keys, has:, is:, bare words, sort: and --facets, for sessions, usage, export, the viewer API and MCP. sessions --where and facets no longer stop at the newest 20,000 sessions there, and event keys and bare words no longer fail with unsupported_store. Query language.
• Harness A/B. Each run keeps the end of its verifier's output (the last 4096 characters of stdout and stderr, colour codes removed, redacted with capture.redact), each repair round too. compare and eval run print the last lines of failed runs under the report, and --json lists them in failedRuns. Verifier output.
• Harness A/B. Harness bundles carry Claude Code plugins (hooks, skills, agents, commands, MCP servers: the whole plugin directory) with harness snapshot --plugin <name|dir> or plugins = [...] in a harness TOML, and compare A --with plugin:<name> runs "with the plugin vs without" in one command (--without plugin:<name> drops one). Each run loads the plugin with --plugin-dir from its own home; your Claude config is never changed. Evals, Does this plugin help me?.
• Harness A/B. compare --repair N reports Seq@1..N+1 per arm: the share of tasks solved by the k-th attempt with verifier feedback, with intervals, where the gain lands (EGS, LGS), rounds that changed no file, and the agent's spend per solved run. With --k 2 or more it adds a cost front of k attempts with feedback against k independent attempts (unbiased pass@k), and a note when the two arms rank one way on the first attempt and the other way after the rounds (Seq@K, arXiv 2610.37859). Evals.
• Harness A/B. compare --repair N tells the agent "Verification failed." in its own session (Claude Code --resume, codex exec resume) after a failed verifier, up to N rounds, with the hidden tests out of its tree. The report adds the repair rate, mean rounds and the rounds' cost per arm. --repair-signal names adds the failing test names from the verifier output; a round that edits, deletes or skips tests is not a repair; --false-alarm P tells a seeded share of passing runs they failed; --repair-verdict decides on the outcome after repair. Repair settings are part of the --reuse key (BOBA, arXiv 2611.02207). Evals.
• Skills. eval trigger <skill-dir> checks that the agent loads a skill when it should and not when it shouldn't. It runs the should and should_not prompts in triggers.toml k times each with the skill and its siblings installed, reads each load from the captured session, and reports precision, recall and F1 with Wilson intervals, the missed and flaky prompts, and near misses that went to the expected sibling. --description or --candidate A/Bs a new description with a paired bootstrap CI and the prompts that flipped. --min-f1 and --fail-on regression set the exit code. CI checks it with real Claude Code and Codex against a stub model. Evaluate an Agent Skill.
• Examples. The skill eval tests an edit to the skill: run.sh <out-dir> --candidate <skill-dir> runs the pinned skill as arm A and the edited copy as arm B, graded the same way, with the --promote line in the report. TASKS="01 06" runs some of the tasks. Skill eval example.
• Self-host. docker run -p 4318:4318 -v coding-sh:/data ghcr.io/jackulau/coding.sh starts a team server with no configuration: with no keys set, the first start makes a read key and an ingest key, keeps them in the volume and prints them with the connect command. Each tagged release publishes the image for amd64 and arm64, starting with this one. deploy/compose/ has SQLite, Postgres and ClickHouse stacks that need no edits. The viewer's sign-in page takes the read key in a form. serve waits up to 30 s for its database to accept connections. Self-host for a team.
• Self-host. coding.sh init --remote <url> --key csh_ik_… connects a machine to a team server in one command: it checks the key, installs capture hooks and adds the team sink. --remote-mode full also sends prompts and output. Self-host for a team.
• Replay. coding.sh replay <session> and a Replay mode on the viewer's session page step through a session with the files the agent touched rebuilt as of each step: each step's diff, a file's content at that step, what a range of steps changed, and shell commands with exit code and output. Where content can't be rebuilt, it says which step and why. The viewer plays at 1×, 4× or 16× with a scrubber and ?step=N links; the MCP session_replay tool returns the same. Explore.
• Capture. coding.sh proxy is a local pass-through for the Anthropic and OpenAI APIs. Each model call is logged with its redacted request (system prompt, tool definitions, messages) and response, usage with cache reads and writes, status, time to first byte and token, and rate limits, inside the Claude Code or Codex session its hooks capture. Auth headers are forwarded and never stored. It adds 0.34 ms per streamed chunk and 1.96 ms to first byte (CI fails over 2 ms and 5 ms). Capture with the proxy.
• Install. init --proxy points Claude Code (ANTHROPIC_BASE_URL) and Codex (a coding-sh provider) at the proxy and starts it; the hook restarts it when a session starts. uninstall --proxy undoes only that, and doctor shows whether it runs. CI checks it with real Claude Code and Codex. Capture with the proxy.
• Install. init --proxy keeps a base URL the agent already used (a gateway or CLIProxyAPI in Claude Code's ANTHROPIC_BASE_URL, your own Codex provider, openai_base_url or OPENAI_BASE_URL) as the proxy's upstream and chains through it, instead of asking for --force. uninstall --proxy puts the old value back. Already using a gateway or CLIProxyAPI?
• Capture. Proxy calls record reasoning effort (including Anthropic's output_config.effort), service tier and speed from the request when the response has none, and compaction: true on calls that compacted the conversation, including Codex's /responses/compact, which is now logged. What is stored
• Capture. The proxy stores a call's messages in content-addressed blocks shared with the session's earlier calls, instead of the whole conversation again on every call, and reads the body back exactly. A synthetic 500-call session took 5× fewer bytes (828 KiB instead of 4,116 KiB); reading a call back takes about 10 ms. Works on SQLite, Postgres and ClickHouse stores; calls stored whole still read, and gc and purge follow the blocks. What is stored
• Capture. Proxy calls from a Claude Code subagent land in the subagent's own session instead of the parent's. Calls record the parent session, Claude Code's agent id and Codex's parent thread, fork and subagent kind when the client sends them, the account as a 16-character hash (the account id is not stored, in the event or the request body), and behind CLIProxyAPI the credential index it used. A session seen only through the proxy links to its parent. What is stored
• Serve. coding.sh serve mounts the proxy at /proxy/: send an ingest key in x-coding-sh-key and the agent's own API key goes upstream unchanged. REST API.
• Sessions. show <session>#n (or a call id) prints one model call's system prompt, tools, settings, cache use and cost, and diff-calls compares two calls in prompt-cache order and names what broke the cache. The viewer shows both on a proxy call. Capture with the proxy.
• Sessions. replay <session> --step N --fork sends the model call behind step N again with a changed CLAUDE.md, AGENTS.md, rule, skill, system prompt or model (--with, --harness, --model, --system, -k) and prints the recorded next action beside the forked one. --run carries on with Claude Code in a copy of the repo, the stored history spliced into its requests, and --verify runs a check there. It needs a session captured through the proxy. The viewer's Replay pane has a "Fork from here" tab, and the MCP session_replay tool takes fork. Explore sessions.
• Evals. replay --fork --save-case <suite> keeps the step as a case in evals/<suite>: a request task that sends the stored request with the project's current context files, graded by a next-action grader that fails when the model repeats the recorded action. eval run runs it like any suite. Evals.
• Replay. exp replay and replay --fork use CLAUDE_CODE_OAUTH_TOKEN (from claude setup-token) for Anthropic calls when ANTHROPIC_API_KEY is not set. Log model calls.
• Experiments. exp replay <call> --vary model=a,b --vary system=@file.md --n 5 resends a logged model call with your own API key, grades whether each reply takes the agent's first action (or with a judge), and reports the paired comparison of exp compare. --dry-run prints the cost estimate. Experiments.
• Install. Windows: run the installer from Git Bash. Hook commands use C:/… paths that cmd.exe, PowerShell and Git Bash all run. CI runs the tests and the installer on Windows. Install guide.
• Capture. CI runs real OpenCode, Pi, Goose, Factory Droid, GitHub Copilot CLI and Aider sessions on Linux and Windows against a stub model, checks the recorded prompt, tool call, file change, model and reply, and checks that ingest --backfill adds no duplicates. Install guide.
• Gateways. OpenRouter, LiteLLM, Cloudflare AI Gateway, Portkey, Helicone and Ramp Router (router.com) join Vercel: sessions get a gateway:<name> tag, gateway enrich fetches billed cost where the gateway has a lookup API, gateway status shows one row per gateway, and compare --gateway <name> routes runs through it. AI gateways.
• SDKs. The TypeScript and Python SDKs detect all twelve gateways, not only Vercel: by base URL, a self-hosted gateway's URL env var, a response header (cf-aig-log-id, helicone-id, x-litellm-call-id, x-portkey-trace-id, …) or the response id. Each model call records the gateway and its request id, so SDK sessions get the gateway:<name> tag and gateway enrich can fetch their billed cost. The SDK tables are generated from the server's gateway list. AI gateways.
• Experiments. compare records the agent release each run started on (agentVersion, checked at most once a minute, so an auto-update mid-run shows). When the arms ran on different releases, the report adds agent claude-code 2.1.316 ×12 2.1.319 ×12 release differs and a note that the delta mixes the change with the release, --gate repeats it, and --promote holds with release_confound. --json carries report.agentVersions (arXiv 2610.37332). Evals guide.
• Experiments. exp compare and compare give the exact McNemar power next to the normal MDE: power MDE 15.6pp · no 80% power at this n (exact McNemar ceiling 62%) or · exact MDE 10.6pp. The exact test needs 6 flipped cases to reject at α 0.05, so a small suite can have no 80% power for any gap; a no-power warning says so, and --json carries power.exact (arXiv 2610.04433). Experiments.
• Experiments. exp run --series NAME tags each round of an iteration loop and prints whether to stop. exp plateau runs a restart e-detector over the series and names the version to ship. It exits 3 to stop, 0 to continue and 2 with too few experiments (Yin et al., arXiv 2610.04756). Experiments.
• Experiments. exp metrics A/Bs two commands that print numbers (ms, bytes) in interleaved pairs, with a 95% paired bootstrap CI and a sign test per metric. --fail-on-worse exits 1 for CI. Experiments.
• Experiments. compare notes the runs whose model calls another model answered: after a refusal Claude Code retries on an older model, so a run can measure a model its arm did not ask for. The note counts those runs per arm and names the model; --json carries report.modelMix. Evals guide.
• Capture. Sessions stored before an adapter's parsing changed are re-parsed from their transcripts and spool in the background, so this release's dedupe fixes reach data already captured. coding.sh ingest --rebuild does it now. Troubleshooting.
• Harness A/B. compare --gate answers "ship B or not": Ship it: B wins 9 tasks, loses 0, Hold: not enough evidence, or Don't ship: B breaks 3 tasks A solved with the broken tasks listed. It exits 0 only for "Ship it" (SAGE, arXiv 2609.36043). --by suite: gates each tagged suite too, and one broken suite blocks a pooled "Ship it". Each arm's failed runs are split by the agent's own last test run: A fails 12: own tests green 7 (58%), red 3, stale 1, none 1, where green fails are silent misses (BOBA, arXiv 2611.02207). compare --promote also keeps a B that is cheaper at the same quality: pareto: support 0.93 ≥ 0.70 → promote (cost-cut: …) or → hold, exit 0 only on promote (ParetoSaddler, arXiv 2611.02079). When B adds or removes one rule, skill or context file, the report adds a one-item gate scored only on tasks where the item reached the agent's context: item gate k=3 τ=.34 m=2: KEEP · …, or RECHECK, KILL, RERUN, and ACCEPT, REJECT, UNMEASURED for removals; --item-gate sets the exit code (Cortex, arXiv 2610.14216). CI checks the gate and the own-tests split on real Claude Code and Codex sessions against a stub model. The MCP harness_compare tool lists these A/B results and returns one with its gates and own-tests split. Compare agents.
• Experiments. exp compare --gate gives the same ship-or-hold answer per case and exits 1 unless it is "Ship it". exp sweep ends with adopt <winner> or keep <start> and the reason. The MCP experiment_compare tool returns the gate too. With 2+ trials, exp compare also lists the cases B gained and lost (passed in any trial), counts cases that got more or less reliable, and warns when the two halves of the trials disagree; --matrix prints the success counts (EvoAudit, arXiv 2611.07624). Experiments.
• Durable workflows. A Vercel Workflow, Temporal, Inngest, Restate, DBOS, LangGraph or Cloudflare Workflows run (Temporal, Restate and DBOS in both TypeScript and Python) is one session across processes, traces, crashes and deploys (keyed by run id). show, workflows show and the viewer list each step with its attempts, failed and lost ones included, show sleeps and LangGraph interrupts as waiting rather than work, mark where the run resumed after a crash, deploy or restart, and leave replayed model calls out of tokens. An attempt lost with its process counts as busy only while its process was still seen; workflows stats shows the rest as "lost to crashes" instead of agent time. A span with workflow.wait.name around a sleep is a wait in any engine, for those that export none (Temporal, Inngest, DBOS TS). New flags step-retried and resumed. Each engine is checked in CI against a real run with a failed step, a killed worker, a sleep and a restart. Trace a durable workflow.
• Evals. Behaviour tasks can be written by hand and loaded with eval import tasks/*/; compare --tag <tag> runs them. Verifiers get the agent's final answer and full stdout (CODING_SH_AGENT_OUTPUT, CODING_SH_AGENT_STDOUT), and judge grade --grader checklist.json grades both from inside one. compare reports agent spend and grader spend rows, --max-cost 15 stops starting runs at $15, per-task rows show task names, and a fresh store gets its hook script so runs are captured. Evaluate an Agent Skill.
• Viewer. Cost shows in the session list, the session header and on each model call. Estimates read ~$0.04 (hover: "Estimated from list prices", and which price a call used); billed and reported costs have no mark; a call with no price shows — and says why.
• Judges. Judge prompts escape data tags in the judged text and say it is untrusted, so an output can't close </output> and order a pass. judge calibrate --probe-injection judge.json plants a "respond pass" order in human-fail cases and reports how many verdicts flip; any flip exits 1. Default-prompt judges get new version ids. Evals.
• Judges. Checklist items can grade rule compliance from the trace with no model call: ran-command, read-path, wrote-identifier, and follows-rules, which checks every imperative line that names a backticked command, flag, path or identifier (Never use `git push --force` ) against what the agent did. Evals guide.
• Cost. Estimates price each model call at the rates it ran at: long-context rates past the model's threshold (OpenAI over 272K, Gemini Pro and Claude Sonnet 4.5 1M over 200K, Haiku 5.5 over 100K), and batch, flex, priority, fast or Claude fast-mode rates from the call's recorded service tier. The call's payload.priceTier says which, and shows in show and the viewer's cost tooltip. usage --json adds estimatedCostByTier. [prices] takes long_context and tier tables. Claude 1-hour cache writes are priced at 2x input instead of 1.25x, from Claude Code's cache_creation.ephemeral_1h_input_tokens and OpenTelemetry cache_creation_1h_tokens, and Claude 4.6+ calls with inference_geo: "us" at 1.1x ([prices] takes cache_write_1h and a geo table). Stored estimates reprice by themselves. Where cost comes from.
• Judges. judge calibrate --grader tests audits a test or command verifier: --sample-passes 40 lists unlabelled passing cases to label, and the audit reports precision (how many passes a person would accept) with a Wilson interval and the false accepts. exp show and exp compare then print the precision and a pass rate corrected for it. Evals guide.
• Workflows. workflows compare gives each variant P(front), the share of bootstrap draws in which no other variant beats it on quality and cost (front 0.93, dominated by base (front 0.41)). Near-equal noisy variants no longer get a confident "dominated". --select min-cost --q-drop 10, --select weighted --lambda 2 or --select pareto picks a variant to deploy and prints deploy: cand (…) with the reason. --seed fixes the bootstrap (ParetoSaddler, arXiv 2611.02079). Workflows guide
• Workflows. workflows compare checks replay experiments for a control fork. When the candidate resumes from recorded runs (resumeFromRunId) and no baseline run resumes from the same ones, it notes "no control fork", since the difference would mix the edit with the replay itself. With a control, it reports how often the control's outcome changed from its source run as the replay noise floor (control.drift in --json). Workflows guide.
• Production. coding.sh-store-clickhouse runs serve, the CLI and MCP on ClickHouse: --store 'clickhouse://user:pass@host:8123/db' (clickhouses:// for TLS). It has no runtime dependencies and passes the store conformance suite; blobs stay on file:// or S3. The same page shows how to send sessions to an OpenTelemetry Collector that writes to ClickHouse, with SQL for cost, slow tools and failed sessions. Self-host for a team.
• Evals. eval validate flags hidden tests that name a helper or file only the reference adds and the instruction never mentions, which fail valid fixes written differently (--strict-coupling makes it blocking). eval crosscheck <task> has a reviewer model judge stored runs' diffs against the instruction and lists the runs where it disagrees with the verifier, as false rejects or false accepts. --solver <bundle> solves the task once first. The counts are stored on the task in the same shape as a human verifier audit. Evals guide
• Experiments. Long runs survive interruptions. exp run, exp aa, exp sweep, compare, bench run and workflows run save each trial as it finishes, and --resume [<id>] re-runs only the unfinished ones, with the same seed, order and --stop-early decision as an uninterrupted run. A changed config refuses to resume and names what changed. The first Ctrl-C lets trials in flight finish. Rate limits, 5xx and network errors are retried with backoff (--retries, default 3), task failures never are. --trial-timeout caps a trial, and exp status shows progress, ETA and cost so far. Evals guide.
• Judges. follows-rules reads rule-file frontmatter: a Claude Code rule with paths:, a Cursor rule with globs: (or neither glob nor alwaysApply), or a Copilot rule with applyTo: is graded only after the agent touches a matching file or opens the rule. Before that the agent never saw it, so it no longer counts as skipped. Evals guide
• Experiments. coding.sh exp ab tests a CLAUDE.md, AGENTS.md, rule or skill edit on your own recent sessions, with no eval tasks to write. It finds the moments your agent went wrong (said done without running anything, stopped with tests failing, retried a failed command unchanged), replays the next turn there with your committed files and with your edit, and has a judge score both. It prints the fixed and broken cases, the same ship gate as compare --gate, and the replay noise floor from the control arm. --dry-run shows the cost first, --max-cost caps it, reruns are free from the cache, and --watch reruns on every save. CI runs it on real Claude Code sessions against a stub model. Test a CLAUDE.md change in five minutes.
• Experiments. The pieces under exp ab work on their own: a continuation task replays a frozen session's next turn with given context files, and a decision-point miner turns sessions into a dataset of those moments, each with its own rubric. A continuation experiment needs a control arm: run it as an A/B, or pass --no-control and get a warning that there is no noise floor to read the difference against. Evals guide.
• Refine. coding.sh refine propose|test|apply|rollback|log|run improves CLAUDE.md, AGENTS.md, rules and skills in tested rounds. propose asks a model for lines that address your sessions' most severe flags, with protected lines marked READ-ONLY. test checks a candidate before any model call (protected lines, harmful or duplicate added lines), A/Bs it against your current files on the decision points exp ab mines, and rejects it when it forgets what an earlier kept edit won (guard: forgets kept edit r_…). apply writes a kept edit with a backup, rollback restores it byte for byte, and log lists every round with its verdict, effect CI, guard result and reason. run loops until --rounds, --max-cost or the plateau detector says stop. Evals guide.
• Refine. coding.sh refine score labels.jsonl --pred FILE scores diagnoses against hand labels by code: occurrence, Signal and session precision, recall and F1, Signal coverage, recall per taxonomy family and, with 2+ --pred files, repeat agreement, each with its counts. Matching is maximum one-to-one on session, component and top-3 type ids. Without --pred it measures the precision of each session flag (Signals, arXiv 2611.04507). Evals guide.
• Refine. refine test|run --suite <name> judges a harness edit on an eval suite of repo tasks, k trials per arm, and can make the harness cheaper and faster, not only better: --objective cost,wall keeps an edit whose pass rate holds within --margin (2pp by default) and whose cost, tokens, wall time, LLM time, turns or size drop significantly, and never keeps one that loses quality beyond the margin. With a cost or time objective the proposer cuts, shown what made the runs slow or expensive; --proposer-command 'claude -p' writes the edit with a command. --until-plateau climbs until the e-detector says it stopped gaining, --resume continues an interrupted round, a skill directory can be the --target, and refine log shows each climb's rounds and Pareto front. Evals guide.
• Harness A/B. coding.sh refine check exits 1 on a CLAUDE.md, AGENTS.md, rule or skill edit that deletes or rewords a protected safety line, or adds an exception that relaxes one (protected-line: AGENTS.md:3 "Never run `git push --force`." (auto)), since a pass-rate A/B can't see that harm (Masked Erosion, arXiv 2610.38392). A line is protected by a <!-- coding.sh:protect --> block, a [refine] protect regex, or automatically when one sentence pairs a gate word with a safety topic; refine protected lists them. exp ab runs the same check before any model call and fails --gate on it. --allow-protected-edit AGENTS.md:3 allows one on purpose. Evals guide.
• Docs. A Workflow experiments page: capture and score multi-agent runs, workflows compare, replays, the control fork and the replay noise floor. Workflow experiments
• Long sessions. A session that runs for hours or days reads as one timeline split into phases: the start, each claude --resume, each compaction and each long idle gap. show lists the phases with day lines, show --all lists every event, and show --follow prints new events while a session runs. Derails mark where a run went off the rails: looping, failure-streak, churn, no-progress, stalled and context-thrash, each with the event and time it started. They show in show, on the viewer timeline (with ‹ › to step through them), in sessions --flag <name>, and in the MCP sessions_list and session_get tools. compare reports derails and time to first derail per run, and a progress grader scores time on track for exp compare. The viewer loads a 40,000-event session in the background and scrolls it smoothly, and MCP session_get with tail reads a long session's latest events. Live hook ingest updates a long session from its new events only: 0.46 s per update on a 30,000-event session, down from 1.16 s. Long sessions.
• Experiments. exp compare --margin 0.03 checks that B is at most 3pp worse than A (paired non-inferiority on the 95% interval's lower bound) and exits 1 otherwise: non-inf B − A ≥ -3.0pp: yes (95% CI lower bound -1.2pp > -3.0pp). It lists cases A passed on every trial that B now mostly fails. Experiments.
• Tool evals. A calls check: tools the trial must call (must), must not call (mustNot), at most max calls, and an order, with partial argument matches (~text substrings, <=N bounds, dotted paths). It reads the trial's tap; the MCP tap now keeps each call's arguments, redacted. exp run takes the same fields as a calls grader over the task's trace. Tool evals.
• Experiments. exp run and toolevals run take --samples adaptive: 2 trials per case, a 3rd when they disagree, and a case fails at half or more failing; the report gives case verdicts and no CI. --stop-on-regression --baseline <experiment> stops the run and exits 1 once a case that passed every baseline trial fails. Gating runs.
• Tool evals. toolevals compare prints p (perm) beside McNemar: a task-paired permutation test on the per-task differences, exact over all sign flips up to 20 tasks and seeded Monte Carlo above. It holds its level at the 5-11 tasks a suite often has, where bootstrap intervals are too narrow (AT-AT, arXiv 2610.34679). --json carries report.permutation; verdicts are unchanged. Tool evals.
• Judges. judge calibrate prints the judge's discrimination gap and label efficiency η, and warns when auditing outputs directly would need fewer human labels: label efficiency η 0.21 · auditing outputs directly needs ~4.8x fewer human labels for the same precision. --judge-cost 0.001 adds the human label cost above which the judge pays off. The calibration state is unchanged; --json carries calibration.efficiency (Choi et al., arXiv 2610.35698). Evals.
• Judges. exp compare --pairwise and judge graders in exp compare print how often a resample of the cases picks the same winner, and n*, the smallest size where that holds 95% of the time: stable B better in 97% of resamples (n*=50) or unstable B better in 61% of resamples (n* censored). An uncalibrated judge that never settles warns "pick or calibrate the judge before collecting more verdicts" (arXiv 2611.04090). --json carries stability. Evals.
• Judges. judge sample <exp> --budget 60 --out plan.json says which cases to label next: a seeded random sample stratified by the judge's verdict and confidence, with labels in proportion to stratum size, so the labels don't favour the doubtful cases. judge calibrate --plan plan.json then prints the human-audited pass rate: human-audited pass rate 0.71 [0.63, 0.79] (42/50 of plan labelled; worst case ±0.09) (Choi et al., arXiv 2610.35698). Evals guide.
• Site. Pages are ready about 0.5 s sooner on a slow phone, with half the font bytes and less layout shift. A Lighthouse A/B with budgets runs on every site change. Site speed.
• Site. A blog at /blog: posts are Markdown files in apps/site/src/content/blog, each with its own page, title card and sitemap entry, and an Atom feed at /blog.xml. The first post explains why coding.sh keeps your sessions on your machine. Blog.
• Capture. Claude Code Bash commands that edit files (sed -i, a heredoc write, tee) record a file.change per file with its diff, taken from Claude Code's own edit diff, even when the command then failed. A nested CLAUDE.md or skills folder loaded mid-session, a Skill call's skill folder and the skills restored after /compact become harness.loaded events and show in show and the viewer timeline. Stored Claude Code sessions are re-parsed to pick them up. CI checks both on a real Claude Code 2.1.319 session. Adapters.
• Sessions. Each skill a session used, through the Skill tool or by reading or running a file in its folder, adds a skill:<name> tag, so sessions --flag skill:pdf finds those sessions. show lists them as Skills pdf (project, invoked, first, before any edit) · pdf-tools (personal, read), and show --json and MCP session_get return skills. A new skill-copy-conflict flag marks a session that used two copies of one skill (arXiv 2610.11647). Concepts.
• Cost. Two kinds of waste per session. Re-fetches are read-only calls (Read, Grep, cat, git diff, …) that returned the same result as an earlier call in the same context; three or more add the duplicate-fetch flag (arXiv 2611.07238). Cache breaks are model calls that found the prompt cache gone and paid for the prompt again, by cause (compaction, model switch, idle, other), priced as cache write minus cache read (arXiv 2610.35496). show prints Cache breaks 2: 31k prompt tokens billed again, $0.1086 extra (other 1, compaction 1), usage totals both per group, and the viewer header, --json and MCP carry stats.waste and stats.cacheBreaks. API reference.
• Sessions. New test-tamper flag: the agent added a skip marker to a test, deleted a test file while tests failed, or changed an assertion and that edit alone turned a failing run green. It reads Edit/Write/apply_patch changes and shell writes (sed -i, >, tee, rm, mv, git checkout), and covers test-runner configs and CI files. New tests and files the agent created itself are not flagged. On 3,957 local sessions it flags 13 (EPISTEME, arXiv 2611.06563). Concepts.
• Harness A/B. compare links each tamper flag to the tool call that made the change and splits tampered passes by whether the agent said it was changing the test: tampered passes B 3: disclosed 2, stealth 1, then one line per pass with the run, file, tool, event id and the agent's words. --json adds tamperAttribution per run, and the MCP harness_compare tool a tamper field. Evals guide.
• Judges. New no-test-tamper checklist check, with no model call: it answers no when a test change hid a failure (a skip marker, a test deleted while red, an assertion edit that alone turned the tests green), cites the change and the test runs around it, and quotes what the agent said: weaken test/sum.test.js by Edit in e42 (turned failing tests green), disclosed: "…". Saying why never makes it pass. Evals guide.
• Evals. eval audit <task> --experiment <id> --witness <file> --witness-cmd <cmd> replays a witness test, one that checks a requirement the verifier skips, on every passing run: claude-tdd: passes 6/10, refuted 3, audited pass rate 30% (3/10). The witness must pass the reference 3/3 first. A replay whose verifier no longer passes is replay-mismatch, never refuted, and a witness refuting over half the passes is ambiguous and not counted. No model calls (TestJack, arXiv 2610.10619). --promote adds it to the verifier, and eval export/import carry witnesses. Evals guide.
• Harness A/B. After eval audit, compare shows each audited arm's raw and audited pass rate, audited pass B 60% → 30% (3 refuted, coverage 100%). --audited counts refuted passes as fails and labels the verdict verdict (audited), with a note when the audit changes it. It needs every pass in both arms replayed, or the raw verdict stands with a coverage note. --json and the MCP harness_compare tool add the per-arm counts. Evals guide.
• Harness A/B. compare adds a surface row: the bytes of instruction text each bundle puts in front of the agent (CLAUDE.md, AGENTS.md, GEMINI.md, skills, rules, system-prompt files, ~/ ones too). When B's is more than 20% bigger, a note asks for a success CI above 0 rather than a tie, since those bytes cost tokens on every turn. --json has the sizes per file in report.surface. Compare agents.
• Evals. Experiment sessions: exp session new --name <topic> starts one in a folder, and each exp run there prints its delta against the previous run on the same dataset as it finishes (pass 0.0% → 100.0% Δ +100.0pp […] mcnemar p=0.125, the grader deltas, the cases fixed and broken, the files changed), with no exp compare. exp session note records hypotheses and readings, exp session summary [--json] prints the timeline. exp run --watch reruns on every save of the spec, the agent script, grader files or watch globs, aborts a run made stale mid-way without storing it, and regrades the stored outputs with 0 agent calls when only the graders changed. Evals.
• MCP. exp_session_note adds a redacted note to the project's experiment session (the hypothesis before a run, the reading after it) and exp_session_summary returns its timeline, paged, with the same blocks as exp session summary --json, so an agent can pick an investigation back up after a context reset. Without a session both return a not_found error naming coding.sh exp session new --name <topic>. Explore.
• Experiments. exp regrade <experiment> spec.json grades a stored experiment's outputs again with new graders, without calling the task, and prints the pass rate before and after, each grader's version change and every run that flipped (flips 1 of 4 runs: 1 pass → fail). exp compare pins grader versions: a grader that changed version or graded one side only gets a warning above the verdict, and cases left ungraded on one side are listed. Evals.
• Experiments. exp run matrix.json runs a spec with "matrix" (JSON Pointer axes, every combination) or "variants" (named cells plus "vary") against a baseline cell: the cells run interleaved, each is stored as its own experiment with metadata.matrix, each is compared with the baseline, and the McNemar p-values are Holm-adjusted over the cells. A cell that differs from the baseline outside vary, or touches graders or the dataset, is refused before any call; --plan prints the cells and runs nothing, --cell runs a subset, --resume continues an interrupted matrix. Evals.
• Graders. The agent-slop grader flags placeholder code, as any and @ts-ignore, empty catches, leftover debugging, focused or skipped tests and narrating comments in the lines an agent's diff adds, and coding.sh judge slop <session|run|experiment> scores captured work with it, per arm for a compare experiment. Verifiers get the agent's diff in CODING_SH_AGENT_DIFF. Evals guide.
• Query. coding.sh sql "select …" runs one read-only SQL query against the SQLite index: one SELECT or WITH statement on a read-only connection with query_only on, with a timeout (--timeout, 10 s) and a row cap (--limit, 1000). Writes, ATTACH, PRAGMA and a second statement exit 2; Postgres and ClickHouse stores get an error that points to export. coding.sh export sessions|events|labels streams rows as JSONL or CSV with the same filters as sessions, on every store. Query and export.
• Privacy. Sessions count what redaction replaced, by kind, in stats.redactions (counts only, never values; blobs and metadata mode included), and a session in which a vendor token was redacted gets the secret:exposed tag, since the agent and its model provider still saw it. coding.sh sessions --flag secret:exposed lists the credentials to rotate, and show prints a Redacted row. Sessions stored earlier are counted from their redaction markers. Storage, sinks and privacy.
• Privacy. coding.sh purge --literal <file> removes a leaked secret from the SQLite store after the fact: event payloads and the search index, session titles and stats, labels, eval docs, settings, blobs (rewritten under a new hash, with references repointed) and the spool, then VACUUMs the index. Values come from a file or stdin and are never printed; --dry-run counts the copies. Storage, sinks and privacy.
• Cost. coding.sh usage marks a cost that counts only some of a row's sessions as a lower bound (≥$1.50, replacing the + suffix), prints how many sessions have tokens but no cost, and names up to five unpriced models with the [prices."<model>"] fix. --json is unchanged. Find expensive sessions.
• Experiments. compare A B --reuse baseline copies A's finished runs from earlier comparisons instead of running them again (--reuse both copies B's too), so several candidates against one baseline pay for it once. A run is reused only when its task, verifier and hidden files, bundle, agent version, attempt, timeout and route hash the same, it finished without an error, and it is newer than --reuse-max-age (default 7d). --dry-run prints how many runs each arm would reuse and run, and the report notes that reused runs did not run interleaved. Evals.
• Evals. An executability grader scores a run by how far it got before its first invalid tool call, one the harness refused to run (a bad argument, an unknown tool, an edit of a file not read yet, a string to replace that is not there, a missing file or command, a denied call). A test that exits 1 ran and is not invalid. The score is the share of steps before that call (its mean is AUC-PEC), and it passes with at most maxInvalid invalid calls (its pass rate is TER); kinds counts only some kinds (Prefix Executability, arXiv 2610.40444). Explore guide.
• Query. sessions, search, usage and export take --where, one filter language over any session or event field: agent:codex cost>0.5 tool:Bash ok:false -flag:compaction, with OR, ( ), wildcards, ranges, sort: and limit:. sessions --facets model,tool counts sessions per value. The viewer API (?where=, /api/facets) and the MCP tools sessions_list, search and usage (where, facets) take the same queries. On SQLite a query is one parameterized SQL statement; a typo is a usage error that points at the character. Query and export
• Query. --where on tools, MCP servers, event kinds and ok reads indexes instead of scanning events, and their facet counts come from a small per-session table. On a 3.3 GB, 1.4M-event store, tool:Bash ok:false takes 5 ms instead of 26 s and the tool, MCP and kind facets 31 ms instead of 8.5 s. The store migrates once on open (about a minute at that size, +4% on disk). GET /api/events returns matching events across sessions, newest first and keyset-paged, and /api/facets takes prefix for autocomplete. Query and export
• Viewer. The session list has one filter bar for the --where language: it completes keys and values (with session counts), runs as you type, and on Enter says what is wrong with a bad query and where. Clauses show as removable chips. The sidebar counts sessions per agent, model, repository, branch, flag, tool, MCP server and event kind; a click adds a value, Alt-click excludes it, and groups fold shut. The query is in the URL, and old ?agent= and ?q= links still open. Full-text hits moved out of the list; the list matches sessions by their text. Explore
• Viewer. An Events tab next to Sessions lists the events that match the query, newest first, with the session each belongs to and the matched words marked. Enter opens the event's step in its session, and n and N step through that session's other matches. GET /api/events takes scope=top. Explore
• Privacy. coding.sh purge --rule <regex> removes every match of a secret's shape (sk-ant-api03-[A-Za-z0-9_-]{80,}) the way --literal removes a value, without printing the rule or a match. Purge also scrubs eval run dirs (~/.coding.sh/runs), where a kept run holds the agent's transcripts. Privacy
• Experiments. exp run matrix.json --reuse baseline copies the baseline cell's finished case-trials from earlier experiments instead of running them again, so several matrices against one baseline pay for it once (--reuse all copies any cell's). A case-trial is copied only when its case, task spec and files, graders, trial number and timeout all match, within --reuse-max-age (default 7d). --plan prints how many each cell would copy, and the comparisons warn that the copies did not run interleaved. Evals guide.
• Compare. compare A B --experiment <id> --retry-errors runs again, in place, only the runs of a stored comparison that errored or failed on infrastructure, and each task, arm and attempt counts once. --failing <id>[@bundle] runs only the tasks A failed in that comparison, with a note that the pass rates are conditioned on that failure. --sample N --seed S runs a seeded random sample of the tasks. --json adds planned, skipped, retried and selection, and the consent prompt counts only the runs still to go. Evals guide.
• Docs. A grader field reference: every grader kind, its fields and defaults, and when it passes, with the provider fields judges share. Grader fields
• Experiments. exp ab and refine run on a Claude or ChatGPT subscription with --provider claude-code (claude -p) or --provider codex (codex exec), and pick one on their own when no API key is set and the CLI is logged in, saying so in one line. Each call runs without your hooks, plugins, MCP servers or settings, in a throwaway config dir; with claude -p the model gets the session's own Claude Code tools and a hook of ours refuses every call, so none runs. Calls run 2 at a time, rate limits back off and retry, and a spent usage limit stops the run with the finished trials kept. Costs show as API-equivalent, with no cap unless you pass --max-cost; improve and improve fix suggest the test on a subscription too. Replaying on a subscription.
• Judges. LLM graders take "provider": "codex", which grades through codex exec on a ChatGPT subscription. Evals.
• Experiments. exp ab with no uncommitted harness edits tests your last commit that changed CLAUDE.md, AGENTS.md, a rule or a skill against the version before it, --dry-run included, instead of stopping with "No uncommitted changes". Evals.

Changed

• Harness A/B. The harness changes block of a compare report collapses a file list longer than 20 lines into directories with file counts (+ .claude/skills/ (412 files)). Before, comparing a full ~/.claude against plain Claude Code printed one line per file, 1,500 of them. --json still lists every path in report.bundleDiff. Harness A/B.
• Docs. Long docs are split into pages that load faster: the CLI reference has a page per command, a long changelog release continues over numbered pages, Evals continues on Experiments and Harness evals, and the Read API has its own two pages. Old links to a section that moved still land on it. CLI reference
• Docs. The vercel-security-audit case study has the live result: six read-only tasks against a real Vercel team, three runs per arm. Counting the 27 sessions that stayed on the requested model, graded with the fixed site check, 0/14 runs passed without the skill and 9/13 with it (+69.2 points, 95% CI +23.1 to +100, McNemar p = 0.25, gate Hold), at 6 times the cost per run. A Corrections section says what changed from the first 4/18 and 14/18: the $LIVE_SITE check, task 06's wording, and 9 sessions that Claude Code moved to an older model after a refusal. Aggregate numbers only. Case study.
• SDK. The TypeScript SDK is published as @codingdotsh/sdk (npm org names can't contain a dot). Install it with npm i @codingdotsh/sdk. See the SDK reference.
• Experiments. The item gate no longer accepts removing a skill or rule when the agent fails the exposed tasks without it; such items read UNCOVERED (neither arm solves them) or UNMEASURED (pass rate without the item under 0.8). The gate line shows U= and A= and a count of internalized, works, uncovered, hurts and between tasks (Wang et al., arXiv 2610.32322). Evals guide.
• Experiments. exp run, exp aa and exp regrade add each --grader to the spec file's graders instead of replacing them, so exp run spec.json -g judge.json grades with both; --replace-graders keeps the old behavior. Evals

Fixed

• Sessions. Replay knows that a file a step created did not exist before that step, instead of calling it unknown, so replay --fork --run removes it from the fork's copy. Replay a session.
• Evals. eval validate applies a reference or near-miss patch after setup when it only applies to what setup creates, as an agent's run sees it. Before, a task whose setup made the mess the agent cleans up could not validate: its reference "did not apply". Evals.
• Improve. improve check --repo <DIR|all> counts the sessions that ran in other checkouts and worktrees of the repo. Before, it ignored --repo and counted only sessions whose repo root was the current checkout, so a repo whose agents work in worktrees showed every change as "No sessions yet". Improve.
• Harness. harness audit skips a file or directory it can't stat (a symlink loop, or a Windows junction Node can't follow) instead of failing with "unknown error". Memory and instruction files.
• Harness A/B. harness snapshot follows linked skill, command and agent directories (npx skills, a dotfiles repo) and stores the files they point to. Before, a ~/.claude/skills made of links snapshotted no skills, so the arm ran without them. Harness A/B.
• Harness A/B. compare and eval run say why a failed run's agent failed: its exit code or timeout and the last lines of its stderr, redacted, under the report and in failedRuns[].agent. Before, an agent that couldn't start (Claude Code refusing bypass permissions as root, a bad flag) failed every run in seconds with only the verifier's output shown. Verifier output.
• Capture. Backfill follows symlinked and junctioned transcript directories and files, and skips dangling links. Before, a ~/.claude/projects made of links ingested nothing. Adapters.
• Harness A/B. Two tasks from the same repo no longer error when they start at once: the first run of a repo set up its snapshot store while the second was reading it. Evals.
• Harness A/B. The tamper check no longer flags a run that only added new test cases to an existing test file, so a regression test written next to the fix keeps the task in the gate's evidence. Changed, removed or skipped tests, early returns, hooks, mocks and shadowing definitions still flag. The run lists those files in tamper.addedTests. Evals.
• Sessions. premature-completion no longer flags an agent that stopped on a failing test run and said so ("2 tests still fail", "I couldn't get test_x to pass", or a question about the failing test). It still flags one that says it is done, fixed or passing, or says nothing about the failure. Stored sessions get their flags recomputed once. Concepts.
• Site. A page that shows "±" no longer downloads a second font file for it. Site speed
• Harness A/B. compare --gate no longer says "Don't ship: B breaks a task A solved" when both arms passed it by editing tests. Tamper-flagged passes now count as fails for the gate in both arms, not only in B. Evals.
• Experiments. exp ab with no ANTHROPIC_API_KEY or OPENAI_API_KEY stops before the first trial with missing_key and points at coding.sh improve check, instead of running every case into an error and reporting "Hold: not enough evidence". refine propose without a key names the proposer, not the judge. Improve your agent every week.
• Rules. A CLAUDE.md line like "Run npm test before you say you are done, and never claim…" is read as a requirement to run npm test: the clause that holds the command decides, not a "never" later in the line.
• Judges. The Anthropic and OpenAI judge providers read ANTHROPIC_BASE_URL and OPENAI_BASE_URL, like the vendors' SDKs, so judges and drafts reach a proxy, gateway or local stub without --base-url. Judges.
• Experiments. When no attempt in either arm solves a task, compare suggests coding.sh eval validate <task> instead of an eval curate command that does not exist.
• Refine. refine check, refine test and exp ab judge only the new sentences of an edited line when looking for an edit that relaxes a protected line. Editing a one-line paragraph that already held a conditional ("use the CLI only if …") no longer rejects the edit. Protected safety lines.
• Examples. The skill eval's live site check counts a network command that uses $LIVE_SITE as touching the site; it missed those, and passed runs that had probed the site. Scenario 06 no longer tells the agent to record open questions instead of waiting, since its pass is to stop and ask. Skill eval example.
• Privacy. coding.sh purge never repeats what was typed on its command line: a secret pasted as an argument, a flag value, an unknown option or a --literal path that does not exist is refused with exit 2 and (value not shown), instead of being printed to the terminal and logs. Other commands still name a stray argument. Purge.
• Compare. The agent spend and grader spend rows no longer count runs copied by --reuse; a reused row shows what those runs cost when they first ran, marked not spent. Evals.
• Sessions. Claude Code subagents get a title and a turn: the title is the Agent tool's description, else the first line of the prompt, and the prompt is recorded as the subagent's first message. Claude Code's internal forks (the compaction summarizer, prompt suggestions) no longer leave empty subagent sessions with no model, tools or title. Stored sessions are parsed again on the next ingest. Concepts.
• Capture. Claude Code subagents that fire no SubagentStart or SubagentStop (web-fetch) are linked to their parent from the parent's Agent tool result, with their prompt, result, model and end, instead of showing as a top-level session with no prompt. Adapters.
• Sessions. skill:<name> tags and skill-copy-conflict count only installed skills: a skills/ folder under an agent's dot folder or in a plugin. A skills/ folder in source code (skills/mod.rs), a reference checkout, text inside a heredoc and a write to a skill file no longer tag a skill. Codex's .system/ and synced/<id>/ folders are skipped, so skill:.system and skill:synced become the skill's own name, and a plugin named for its one skill is tagged once (skill:frontend-design). Concepts.
• Sessions. test-tamper no longer flags adding tests: a skip guard inside a new test, fixing a test the agent added earlier, updating an expectation its own source edit broke, or a green run of fewer tests than the failing one. A file a shell command's diff picked up without the command naming it (a test run, a git rebase, another agent in the checkout) is never flagged, and BSD sed -i '' no longer counts '' as a file. A skip marker inside a string literal (a diff or sed command a test holds as data) is not a skip. Stored sessions are re-checked in the background after the upgrade. Concepts.
• Sessions. coding.sh sessions leaves out sessions with no turn, tool call or model call, such as a process that started and ended a session and did nothing; --empty lists them too, and --limit still fills with sessions that have work. CLI reference.
• Cost. usage TOK/TURN (tokensPerTurn) is input, output and cache-write tokens per user turn. Cache reads are left out, because each model call re-reads the whole context, and TURNS counts only top-level sessions' turns, because a subagent's prompt comes from its parent's turn. Expensive sessions.
• Capture. The background re-parse after an upgrade no longer blocks reads: it holds the ingest lock at most a second at a time and ingests new hook lines between holds, so sessions and the viewer answer in about a second instead of waiting 8 to 24 seconds. It writes blobs in batches, resumes where a cut-short run stopped, and works out stats and flags again for every stored session when an upgrade changes them. Listing a large spool is faster too. Troubleshooting.
• Doctor. Hooks in a project's config (.claude/settings.local.json, .codex/hooks.json, .gemini/settings.json, .cursor/hooks.json, …) show as hooked in <file> (project), found in the current directory and in the directories the agent's recent sessions ran in. An agent with events in the last 5 minutes and no hook found reads events arriving, hook not in user config, never not hooked. A stale hook bin now says run coding.sh update --refresh-bin instead of init, and coding.sh ingest (which the hook starts at every session end) refreshes the bin itself after npm i -g of a newer version. Troubleshooting.
• Viewer. API sessions with a failed request are red in the session list again, with the reason: 1 failed request (429), the first error on hover. Sessions carry stats.llmErrors, counted at ingest (existing indexes are filled in on upgrade), and a list page costs the same store queries at any length.
• Viewer. The top bar fits from 320px up, in any font (on Linux without Geist it ran 2px past a 320px screen; a tab label now ends in an ellipsis before the page scrolls sideways): between 641 and 820px search folds to its icon instead of squeezing the theme button, and every top bar control has a 44px hit area.
• Viewer. The session title in the list keeps at least 12rem on any desktop width (at 1024px, next to the filter sidebar, it was 20px): tokens, then duration, then repo, then activity give way first.
• Windows. coding.sh update and the MCP launcher run npm and coding.sh without a shell, so paths with spaces or & arrive intact.
• Windows. Goose's config is found under %APPDATA%\Block\goose\config, where Goose keeps it on Windows. Eval runs find agent binaries on PATH (Windows spells it Path), and Amp and Pi hooks use the node on PATH, as elsewhere, instead of the one that ran init.
• Aider. Eval runs pass --no-show-release-notes: on a new version's first run, --yes-always made aider open its release notes in a browser.
• Codex. Code mode: a command run inside an exec cell now points at the cell, the cell fails when a command in it fails, and session stats count each command once instead of twice (hooks and backfill agree).
• Viewer. A big ingest no longer blanks reads: coding.sh ui opens at once (0.3 s, was 41 s on a 25k-line spool) and indexes in a background worker with an "Indexing N files…" pill, and status, sessions and search answer within 1.5 s, noting what is still pending (--fresh waits). CLI reference.
• Capture. A full ingest of a 25k-line spool takes 4 s instead of 294 s on macOS, and 39 s instead of 153 s on Linux: blobs are made durable once per checkpoint instead of with two fsyncs per line, and each checkpoint commits its events, session rows and spool cursors in one transaction, so a crash mid-drain neither loses nor duplicates events.
• Cost. coding.sh collector prices model calls with tokens and no cost, as serve and ingest do, using your [prices].
• Viewer. The top bar fits a 360px phone.
• MCP. Tool calls no longer wait for a big ingest: they answer within 1.5 s from what is indexed, with pending: {files, indexing} in the result. CLI reference.
• Harness A/B. compare and eval crosscheck runs keep the agent's text from mid-turn: the run dir, which holds the agent's own transcript, was removed before it was ingested, so the session kept only the final reply.
• Examples. The skill eval's live mode grades fairly: --help and help calls never count as writes, a refused env ls reads "refused secret read", live task copies name the team, project and site only through $VERCEL_SCOPE, $VERCEL_PROJECT and $LIVE_SITE, the site checks match the team's real domains (kept from the agent), and out/checks records each run's task and arm. run.sh <out-dir> --regrade grades stored runs again without calling the agent. Evaluate an Agent Skill.
• CLI. status, sessions and search with --json carry "pending": {"files", "indexing"} while captures are still waiting, instead of dropping the note. CLI reference.
• Evals. A command task, grader or source that crashes names the error and its cause, such as exited 1: TypeError: fetch failed (cause: connect ECONNREFUSED 127.0.0.1:8803), instead of the last three lines of stderr (for Node, the closing braces of the error object). A refused connection now reads as an infrastructure error, so the trial is retried and counted as infra, not as a failure. Evals
• Evals. A task drafted from a session names the files its setup was guessed from, such as setup guessed from package.json (no lockfile): npm install, instead of saying "from lockfiles" when there was none.
• Experiments. A grader error leaves a run ungraded only when no other grader failed it. With exact and a judge that is down, the runs exact failed count as failures and only the runs it passed are left out, instead of every run reading ungraded and the pass rate going blank. exp regrade follows the same rule. Evals
• Evals. eval run reports on a cases suite say "case" and on a triggers suite "prompt" (the header, the table, never-passed, saturation and the regression lines), and --json carries the report's unit.
• Evals. eval validate shows the last 12 lines of the verifier's output when a check goes the wrong way (the reference fails, the no-op or a near-miss passes, or setup fails), once per kind of check, and --json carries every step with its output in steps. Evals
• Evals. eval init --force no longer deletes your work: it stops when the suite holds a file eval init did not write or that you changed since (a reference.patch you added, a prompt you fixed) and lists them. --force-all deletes those too. evals/<name>/.eval-init.json records what eval init wrote. Evals
• Evals. eval init --skill <dir> uses the skill's own triggers.toml when it has one and you give no prompts and no --draft: suite.json points at it instead of a TODO template. Evals
• Evals. eval init --sessions and --session latest include a session that just ended: they wait up to 30 seconds for the ingest its hook started, then ingest the rest, instead of skipping the ingest when its lock is busy after 1.5 seconds. Evals
• Graders. When a command grader, command task or bench source crashes on Node 24.22, its reason names the error again (TypeError: fetch failed (cause: connect ECONNREFUSED …)) instead of the crash's closing braces. Node 24.22 prints an uncaught fetch failure as [TypeError: fetch failed] {. Evals
• Store. A SQLite index that fails to open (a newer schema, a failed migration) no longer stays open in the process that tried, so on Windows the file can be replaced or removed right away. Concepts
• Evals. On Windows, eval init and eval run show suite paths with forward slashes, like the file list under them: Wrote … to evals/triage, evals/pdf/triggers.toml still say TODO and the --skill path in the --draft --force hint read the same on every OS. Evals