coding.sh 0.1.1

Daily benchmarks for any AI agent: coding.sh bench runs every contender on fresh cases each day,
and experiments add cluster bootstrap, an A/A noise floor, interleaved runs, early stopping and compare --without.
Seven more agents are verified on a real install, the viewer and the site work on phones, and the installer uses npm once the package is there.

Added

• Benchmarks. coding.sh bench samples fresh cases each day from a command, failed sessions, GitHub issues, a sitemap or a dataset, freezes them as a dataset version and runs every contender on that set in random order. Benchmarks.
• Benchmarks. Shared timeout and retries, a budget cap, quota and rate-limit cases dropped for every contender, and bench run --resume to finish a stopped run with the same seed and cases.
• Benchmarks. bench report writes a static HTML page and JSON: trend, gaps to a reference pooled over a fixed window, head-to-head stats, difficulty and failures. bench regrade re-scores stored outputs and bench schedule prints a cron line and a GitHub Actions workflow.
• Benchmarks. An [anchor] set of fixed cases runs every day next to the fresh ones and is scored apart, so its trend has no day-to-day difficulty in it.
• Benchmarks. seed = "secret" publishes the hash of each next day's salt and bench show checks the chain. Content hashes keep the same case under a new key from counting as fresh.
• Benchmarks. Input and output tokens are kept apart, and each cost says where it came from: reported, costPerRun, the shared price table, or unknown.
• MCP. A benchmarks tool lists benchmarks and reads results and history.
• Experiments. exp compare --cluster <key> resamples by repo or session and warns under 10 clusters. Evals.
• Experiments. exp aa measures the run-to-run noise floor, and power planning uses it.
• Experiments. Compare flags uneven attrition between arms (Fisher exact) and splits infra errors from agent errors.
• Experiments. Under 100 paired cases, compare also reports P(B>A) from a paired posterior.
• Experiments. exp run --arm a=<task> --arm b=<task> interleaves both arms case by case in a seeded random order.
• Experiments. --stop-early checks at planned looks with O'Brien-Fleming alpha spending and reports a repeated confidence interval.
• Experiments. Pairwise judges see both outputs in both orders. exp compare --pairwise reports win rate, ties and position bias.
• Experiments. exp sweep races task configs on a train split and reports the winner's optimism gap on held-out cases.
• Compare. compare <bundle> --without <path> measures what one skill, AGENTS.md, CLAUDE.md or MCP config is worth. Compare agents.
• Compare. With 2+ attempts per task, failed runs split into bad runs, arm limits and tasks no arm solves, with variance shares.
• Compare. Cost rows show how many runs had a known cost, and "cheaper" needs a known cost for 80% of each arm.
• Agents. Verified on a real install against a local stub model: opencode, copilot-cli, droid, goose, pi, t3code and aider. Agent matrix.
• Agents. Antigravity backfill reads agy transcripts and conversation DBs for model, tokens, cache reads and latency per call. Tool steps missing from the transcript count as failures, and hooks reply {}.
• Viewer. Phone layouts: 44px targets, filters in a bottom sheet, a compact session header and safe-area insets.
• Viewer. The Benchmarks tab shows anchors, pooled gaps, difficulty, cost sources and tokens, and ranks follow the pooled gaps.
• Site. Mobile pass: touch targets, safe areas, stacked agent cards and 16px body text on phones.
• Site. A new landing page, a changelog page and docs pages.
• Install. install.sh installs from npm when the registry has the version, else from the GitHub release. CODING_SH_SOURCE=npm|github picks one.
• Release. A tag creates its GitHub release as soon as the checks pass, so the installer gets it right away.
• Release. The release workflow can publish all five packages to npm and PyPI with trusted publishing. Nothing is on npm or PyPI yet.
• Docs. A new README, reference pages, how-to guides and a code of conduct.

Fixed

• Fixes. Safer installs: symlinked and JSONC agent configs survive, init --dry-run prints a short plan, and blockers show before the Y/n.
• Fixes. Each SQLite migration runs once when the hook and the CLI open a new index at the same time.
• Fixes. Experiments stop cleanly on Ctrl-C, kill their child processes and keep every finished run.
• Fixes. Capture, ingest and storage state bugs from an audit of each state machine. Sessions are priced locally at ingest.
• Fixes. Adapter fixes for opencode, goose, copilot-cli, droid, aider and t3code, found running each on a real install.
• Fixes. coding.sh run --agent <name> records sessions for names with no adapter.
• Fixes. exp aa refuses to compare a run with itself.
• Fixes. coding.sh ui prints the viewer URL once in a terminal.
• Fixes. Muted text in the viewer passes WCAG AA in both themes.
• Fixes. Timing tests compare against a baseline, so a loaded machine no longer fails them.