v0.2.0

A release with breaking changes that closes the loop in CI: critical: true tasks, argument matchers and per-task given, run --baseline with what changed per task, passed, failed, flaky and skipped verdicts with errored trials skipped, exit 1 for a failed gate and 2 for an error, init drafts that keep their sources' facts, show their plan and cost with --dry-run and print what they cost, a simulated caller that says it doesn't know instead of guessing, and a Claude Code plugin installed from the release; read the seven Breaking: entries under Changed before upgrading.

Added

• Domain spec: a task's called and not_called entries can say which call, as { tool, args } with a where clause on the call's arguments, such as called: [{ tool: transfer_to_human, args: { mc_number: "100101" } }] or not_called: [{ tool: book_load, args: { load_id: L-1004 } }]. A failed check says how the closest call differed, and validate rejects an argument the tool doesn't have or an unknown operator. Plain tool names work as before, and the library exports the entry's schema as CallExpectSpec (#25).
• Critical tasks: critical: true on a task marks a release blocker. sandtable run exits 1 when any trial of a critical task fails, with or without --fail-under, and names those tasks. The terminal summary and the HTML report list critical tasks first and mark them, results.json carries "critical": true on them, and the library exports criticalFailures. The bundled freight and clinic examples mark three refusals each as critical, so their naive agents' runs now exit 1 (#28).
• Domain spec: a task can change the world it starts in with given, instead of a seed row only it uses or a copy of the domain. given.rows merges rows into the seed by each table's key for that task only: a seeded key patches the fields listed (the same carrier with insurance_status: expired), a new key adds a row. given.context, such as a caller ID or a trigger payload, is given to the agent and not to the simulated user: it follows the policy in the default system prompt, and an agent over HTTP or a JS/TS module agent receives it as context. lint and regrade use the patched world, and a value in the context counts as shown for never_disclose (#37).
• sandtable run --baseline <results.json> compares the run with an earlier run of the same domain, such as the last run on main. The summary lists each task that broke, got worse, got fixed or got better, new failures first, counts added, removed and unchanged tasks, and shows pass^1 and pass^k before and after over the tasks both runs ran. A task whose test changed since the baseline is listed as changed instead of fixed or broken, and left out of the scores: each task in results.json carries two short hashes, inputs (what the agent and caller are given and the world they act in) and grading (its expectations and the invariants), and a task whose hash differs from the baseline's changed. The summary also names what else differs: the agent, the simulated user, k, and how many tasks' tests changed; a baseline written by 0.1.x has no such hashes, and the summary says it can't tell. The baseline is read before any trial runs, so a wrong file or another domain's run costs no model calls, and the exit code is unchanged. results.json keeps the baseline's per-task results as baseline, the HTML report opens with the changes and marks regressed tasks, and report and regrade show them again without the baseline's file. The library exports compareRuns and baselineSnapshot (#12, #309).
• sandtable run: each task gets a verdict over its graded trials, passed, failed, flaky or skipped, and a 95% Wilson interval for its pass rate, and runs report pass@k next to pass^k, the flip rate (the share of tasks that are flaky) and the flaky tasks. The terminal summary and the HTML report show them, and results.json adds passAt, flipRate, flaky and skipped, and per task graded, skipped, verdict and interval. The library exports passAtK, wilsonInterval, trialStatus and the Verdict type (#27).
• Domain spec: a reference call can be one the API rejects, marked expect_error: true or with text its error contains, such as expect_error: not found. lint then requires that call to fail, so a refusal task whose correct agent looks up an unknown ID is proven solvable instead of solvable ?, and a domain edit that drops the rejection is caught. A call with arguments the tool rejects is still a lint error. The freight example's unknown-mc-number uses it, and the library exports ReferenceCallSpec (#274).
• sandtable init: after --sop and after --transcripts, it prints the step's token usage per model and an estimated cost at list prices, as run does: one Claude call for the SOP and one per transcript, so an engineer can see what a draft cost before drafting from many more transcripts (#297).
• sandtable init --dry-run: with --sop or --transcripts, prints a plan line (the model calls, 1 for the SOP and 1 per transcript, the model, the provider and region, and a cost bound at list prices) and stops: no model call, nothing written, and no API key, AWS credentials or Bedrock package needed. With --openapi, it lists the files it would write. A real run prints the same plan line before its first model call (#296).
• init --transcripts: each drafted task gets reference calls, the tool calls a correct agent makes in order, so lint can prove it solvable instead of reporting solvable ?. init lints each drafted task, replaying its reference, and keeps one that fails for review, with a warning and a # TODO(review): lint: ... comment above it. References list only calls that succeed, so a refusal whose correct path includes a call the API rejects may have lookups only. A drafted range, such as a negotiated rate between two bounds, now keeps both bounds; before, only the last condition on a field was kept (#278).

Changed

• Breaking: sandtable run: a trial that ends in an error, such as a rate limit or timeout that outlasted the retries or an agent endpoint that went down, is now skipped instead of failed, unless the agent had already broken a hard invariant or called a not_called tool. A skipped trial is left out of pass^k, pass@k and the per-tag scores, carries skipped: true in results.json and is listed under Skipped trials, and run exits 2 when any trial was skipped, even without --fail-under, because the run is incomplete. A CI job that passed with a few errored trials now fails: fix the cause, such as credentials or a lower --concurrency, and rerun (#27).
• Breaking: library: in the exported Task type, expect.called and expect.not_called entries are string | { tool, args } (the new CallExpect) instead of string, since a task can match a call's arguments. TypeScript code that reads them as tool names must handle the object form, e.g. typeof e === "string" ? e : e.tool. Domains that list tool names are unchanged (#25).
• Breaking: examples/freight and examples/clinic: three refusal tasks in each are now critical: true, so a run of their naive scripted agents (--agent scripted:naive or scripted:clinic-naive) exits 1 instead of 0. A script that renders their reports should accept a failed gate; demo and the careful agents are unaffected (#28).
• Breaking: sandtable validate: a domain with an unknown operator, $in or $nin without a list, or a top-level $ key in any where clause now fails validation, so no command loads it. On 0.1.x it loaded, then crashed mid-run or silently matched nothing. Fix the clause at the path validate names (#279).
• Breaking: Simulated caller: its prompt changed, and the prompt is part of a run's inputs hash. So sandtable regrade refuses a run stored by 0.1.x, and sandtable leaderboard stops without running anything when a model's stored results come from 0.1.x. Regrade such a run with the version that made it (npx sandtable-ai@0.1.2 regrade …), and for a board, move those models' folders away or pass --no-resume, which reruns every model and costs model calls. Scores of new runs can differ from 0.1.x runs of the same agent, since the caller now says it doesn't know a fact its instructions leave out; run --baseline against a 0.1.x run says the prompts changed. Review tasks whose caller instructions leave out a fact the agent will ask about, such as an item's condition, a receipt or a date, and add it to user.instructions (#287).
• Breaking: CLI exit codes: a failed gate exits 1 and an error exits 2, as grep and diff do. 1 means a gate failed on a complete result: run below --fail-under or with a failed critical task, lint errors, validate --strict with pending items, and validate or lint on a domain that exists but is invalid. 2 means the command couldn't do its job: a usage error (an unknown command or option, a missing argument, an invalid value, or no command at all), a missing or unreadable file, no domain at the path, an invalid domain for any other command, an adapter that can't be built, a run or demo --live in which every trial ended in an error, a run with skipped trials (even when a gate failed too), and a leaderboard on which every model failed. Before, all of these exited 1. Help, --version and 141 for a closed pipe are unchanged. A CI step that treats any non-zero status as a failure needs no change; one that checks for 1 to mean an error must accept 2, and one that lets a failing agent through, to render its report, can accept only 1 with || [ $? -eq 1 ] (#282).
• Status: the README, the docs and the PyPI classifier describe sandtable as beta instead of early alpha. It works and is tested, and the spec and outputs may still change before 1.0 (#267).
• Leaderboard docs: the freight board's history includes its re-render at v0.1.2, which ran no model and changed no verdict, and leaderboard.json is described as recording 0.1.2 at eb4cfd2 (#260).
• Roadmap: the v0.3 section of ROADMAP.md and sandtable.dev/roadmap separates what is done on main and not yet released (run --baseline, reliability verdicts, critical tasks, argument matchers and given) from what is left, including the follow-ups filed since: a regression gate, exit codes, per-tag floors, the leaderboard and demo --live, and sandtable compare (#288).
• Claude Code plugin: after a fix, running-and-reading-results and the failure-triager subagent confirm a change to grading only (an invariant, an expectation, a never_disclose rule) with sandtable regrade, which calls no model, and rerun with --tasks only the tasks whose caller instructions, tools, policy or agent changed. They fix grading and regrade before changing a caller's instructions, since regrade refuses a run once any caller prompt changed (#300).
• Claude Code plugin: attribution no longer blames an agent that stops because the records contradict the request: running-and-reading-results checks the seed rows and given.rows against the caller's instructions, and a disagreement is the task's fault. A fact the caller states that its instructions lack is fixed in the task's instructions. Before recommending a model from a leaderboard, the skill has failure-triager attribute every failure of the top candidates, since a cheaper model's only failure may be the task's (#299).
• Breaking: sandtable report -o: a path that doesn't end in .html is now a folder, and the report goes in it as report.html, as run -o and regrade -o take a folder. Before, report results.json -o runs/rerender wrote an HTML file named rerender with no extension. A path ending in .html still names the file itself, so a script that passes a file name without .html should add it (#317).
• sandtable demo --live: the simulated caller now defaults to Claude Haiku 5.5 instead of the agent's Claude Opus 5.5, about half the cost of a first live run, as the docs already recommended with --user-model claude-haiku-5-5. --user-model still picks another caller, the agent keeps its default, and run --user claude is unchanged (#329).
• Claude Code plugin: the drafting-a-domain skill says how to install the CLI with its optional packages. Prefer a project install (npm install --save-dev sandtable-ai @anthropic-ai/bedrock-sdk, then npx sandtable-ai <command>). It lists which package each path needs: @anthropic-ai/bedrock-sdk for --provider bedrock, @aws-sdk/client-bedrock-runtime for a bedrock: agent and @aws-sdk/credential-providers for --aws-profile. Without an install, the form is npx -p sandtable-ai -p <package> sandtable <command>, where the command is the bin, sandtable. It also says to keep runs inside the project. An agent no longer has to read sandtable's compiled code to run init on Bedrock (#295).
• Claude Code plugin: before init --sop or init --transcripts, the drafting-a-domain skill has the agent run the same command with --dry-run and show the engineer its plan line (model calls, model, provider and a cost bound), even when model steps are already authorized. When they are, it goes on to the paid run after the plan, instead of reporting the cost afterwards; otherwise it asks. It suggests -m claude-sonnet-5-5 for a first draft of many transcripts. init then prints what the step actually cost (#296).
• Claude Code plugin: the review checklists of the drafting-a-domain skill, and the review workflow docs, gain the checks a run on real support chats needed. A drafted task keeps its transcript's facts and outcome, every row it touches agrees with the transcript, and its caller's instructions state every fact the agent will ask about. An ambiguous case in the SOP goes to the customer, and until then the task grades only what every reading agrees on. A drafted invariant that requires a step the agent can skip while still deciding correctly becomes soft: true, or checks the decision itself (#298).

Fixed

• Docs: sandtable.dev documents the latest release instead of main, so a user of the published CLI no longer finds flags and spec keys it doesn't have yet, such as --baseline or critical: true while npm served 0.1.2. The site deploys from a docs-live branch that each stable release moves to its tag (#313).
• sandtable validate: an unknown operator such as $like, $in or $nin without a list, or a top-level key such as $or, now fails validation in every where clause, with the clause's path: a tool's list op and checks, invariant require.where and never.when, and a task's expect.state where and fields. Before, only a called/not_called entry's args were checked: elsewhere, lint and run stopped on a bare "Unknown operator" without writing results, or a clause matched nothing and a valid call was rejected (#279).
• CLI: a path that names nothing now says which argument or option it came from, instead of Node's ENOENT: no such file or directory, open '…': report, regrade, run --prices and --baseline, leaderboard -m and --prices, and init --openapi, --sop and --transcripts, e.g. --prices: no file at prices.json. A directory given where a file is expected says so, and points at the results.json inside it when there is one. init reads every input before drafting, so a wrong --transcripts path no longer fails after --sop has called a model, and a --transcripts directory with no .txt or .md file is named. report -o creates the directory it writes into, as run -o does (#280).
• init --transcripts: a drafted task keeps the facts its transcript states. An existing row is reused only when every fact the transcript states about it matches, so a second caller with the same name but another membership level or order gets a new row instead of a task rewritten to fit the first one. When a draft still reuses a row that contradicts its transcript, init keeps the task for review, prints a warning naming the row and each differing field, and writes it as a # TODO(review): comment above the task (#276).
• init --sop: every rule the SOP states now ends up as an invariant or a Soft rule comment. Before, a rule it couldn't express, such as a cancellation window with an exception, was in neither list. Claude lists the SOP's rules first and maps each one. Each soft rule comment says why no invariant checks it, such as needs today's date, depends on a fact no record holds or needs either of several tools first. A rule whose drafted invariant was dropped, or that the draft mapped nowhere, is listed too, with a warning (#277).
• Simulated caller: a fact its instructions don't state about the caller, its account or what it is calling about is now unknown to it, so it answers "I'm not sure" instead of guessing, also to yes-or-no questions such as an item's packaging or a receipt. It also no longer describes the agent or explains how it is answering. Reports flag the turns where it still steps out of its role, talking about its instructions, saying its goal is met or talking about the agent in the third person: the terminal summary and the HTML report list them as Caller notes and mark them in the transcript, and results.json has callerNotes per trial and a count per run. They never change a verdict. The library exports callerNotes. Runs stored by 0.1.x can't be regraded or resumed by this version: see the Breaking: entry under Changed (#287).
• Claude Code plugin: the marketplace installs the plugin from plugin/ at the latest release's tag instead of from main, so its skills describe the CLI that npx sandtable-ai runs and its version is that release's. Before, they could document features only main had, such as given and critical: true under 0.1.2, and an agent that found them missing guessed which features existed. Existing installs get the pinned plugin with /plugin marketplace update sandtable. The skills now say to learn what the installed CLI supports from npx sandtable-ai --version, --help and validate, never from sandtable's own code, in npx's cache or in the project's node_modules/sandtable-ai, and that a Claude model on Bedrock takes its plain name (-m claude-sonnet-5-5), which sandtable maps to the region's inference profile. The writing-good-tasks and running-and-reading-results skills also load for requests about release-blocker (critical) tasks and CI gates, so their guidance reaches the agent when it's asked to make a task a release blocker. To try the skills from a clone, start Claude Code with claude --plugin-dir ./plugin (#301).
• sandtable run --agent <module>: a .ts or .js agent that Node loads as CommonJS, as in a project made with npm init ("type": "commonjs"), now fails with what to do: rename it to .mts or .mjs, or set "type": "module" in the nearest package.json. Before, it showed only Node's Unexpected token 'export'. On a Node too old to load TypeScript, the error says it needs Node 22.18 or later. The bring-your-agent guide and the README say a module agent must be an ES module (#305).
• sandtable run and demo --live: before the first trial, they check that the agent and the simulated user can start, and when one can't, they name it and the problem, run nothing, write nothing and exit 2. Before, a missing ANTHROPIC_API_KEY made every trial fail one by one (demo --live skipped all 50 before saying why), and when only one side lacked credentials, the other side's calls were made and billed in every trial. The checks cost nothing: Claude on the Anthropic API lists one model, Claude on Bedrock and bedrock: agents resolve their AWS credentials, region and optional package locally, and an HTTP agent must accept a connection, with no request sent (#316).
• Domain errors: a schema or cross-reference error in an included file, such as a task in tasks.yaml with an unknown key or an argument its tool doesn't have, now names that file instead of sandtable.yaml, as YAML syntax errors already did. Problems in several files are listed under each one. lint says "needs a row in orders" instead of "needs a orders row" (#317).
• Agents and callers on other models: reasoning a model writes into its reply text no longer reaches the other side of the conversation or the graders. The openai: adapter removed only <think> blocks, so gpt-oss through Bedrock's OpenAI-compatible endpoint, which writes <reasoning>, leaked its reasoning: as the agent it was scored as speech (never_disclose included), and as the simulated caller it quoted its hidden instructions to the agent. The Bedrock Converse agent (bedrock:<model-id>) removed only <reasoning>, and missed gpt-oss's occasional <analysis>. Both adapters, for the agent and for the openai: caller, now remove <think>, <thinking>, <reasoning> and <analysis> blocks, a closing tag whose block opened before the reply, and an opening tag that never closes with everything after it. usage counts the replies that had reasoning removed as reasoningStripped, shown in the terminal summary and the HTML report. Runs of such models through these adapters scored leaked reasoning as speech before; regrade can't change that, since the stored transcript holds the leaked text and the conversation would have gone otherwise, so rerun them. On the published freight leaderboard, one of gpt-oss-120b's never-disclose-ceiling failures came from an <analysis> block (#308).
• init --sop: a drafted invariant's conditions can use $in, $nin and $exists, so a rule that a call must include an argument in some cases, such as a refund by check needing the mailing address, drafts as a never with when: { method: { $in: [...] }, mailing_address: { $exists: false } } instead of a soft rule that says it can't be expressed. Two conditions on the same field, such as a lower and an upper bound, are both kept; before, the second replaced the first (#314).
• docs/quickstart: the toy HTTP agent, in Node and in Python, remembers the carrier it verified under the session_id sandtable sends with every step, so after a verified MC number it says a rep will finish the booking instead of asking for the MC number again on every turn. It still scores pass^1 36.0% on the freight example (#270).
• init --transcripts: the drafting prompt now shows each tool param's allowed values, whether it's optional and its description, as init --sop's does, so drafted references use the API's values (by_mail) instead of the transcript's wording ("By Mail"), which failed the draft-time lint. A drafted value that differs from exactly one allowed value only in case, spaces or dashes is replaced by it, with a warning, and drafted end-state conditions can use in, nin and exists like init --sop's (#336).
• Claude Code plugin: the drafting and results skills say what to do when a command fails with OpenSSL configuration error … openssl.cnf … Operation not permitted: Claude Code's sandbox blocked Node from reading Homebrew OpenSSL's config, so rerun it with OPENSSL_CONF=/dev/null in front (#334).

What's Changed

Other changes

• docs(docs): the leaderboard page records the board's re-render at v0.1.2 by @Uliseis in #264
• docs: sandtable is described as beta, not early alpha by @Uliseis in #268
• feat(grading): expect.called and expect.not_called can match tool arguments by @Uliseis in #269
• feat(grading): a critical task fails the run when any of its trials fails by @Uliseis in #271
• feat(spec): a task can patch seed rows and give the agent call context with given by @Uliseis in #272
• feat(run): run --baseline shows which tasks broke or got fixed since a previous run by @Uliseis in #273
• feat(grading)!: errored trials are skipped instead of failed, and tasks get passed, failed, flaky or skipped verdicts by @Uliseis in #275
• docs: the roadmap shows which v0.3 items are done and which are left by @Uliseis in #289
• fix(spec): validate rejects an unknown operator or $in without a list in any where clause by @Uliseis in #290
• fix(cli): a missing file names the option it came from instead of printing ENOENT by @Uliseis in #291
• fix(init): init --transcripts keeps a transcript's facts and warns when a reused row contradicts them by @Uliseis in #292
• fix(init): init --sop lists every rule it can't check as a soft rule, with the reason by @Uliseis in #293
• fix(adapters): the simulated caller says it doesn't know what its instructions don't state, and reports flag turns where it steps out of role by @Uliseis in #294
• docs: the changelog marks every change since 0.1.2 that can break an existing setup by @Uliseis in #304
• fix(plugin): the marketplace installs the plugin from the latest release's tag, so its skills match the CLI npx runs by @Uliseis in #306
• feat(cli)!: a failed gate exits 1 and an error exits 2, as grep and diff do by @Uliseis in #307
• docs(plugin): the skills confirm a grading-only fix with regrade, and rerun only tasks whose inputs changed by @Uliseis in #310
• docs(plugin): attribution doesn't blame an agent that stops because the records contradict the request by @Uliseis in #311
• fix(report): run --baseline labels a task whose test changed as changed, not fixed or broken by @Uliseis in #326
• fix(adapters): a module agent that Node loads as CommonJS says to use .mts or type module by @Uliseis in #319
• fix(run): run and demo --live check the agent's and caller's credentials before the first trial by @Uliseis in #325
• fix(cli)!: report -o takes a folder as run -o does, and domain errors name the file they're in by @Uliseis in #328
• fix(adapters): reasoning a model writes into its reply never reaches the other side, in the openai: adapter and Converse by @Uliseis in #322
• feat(spec): a reference call can expect the API to reject it, so lint proves refusal tasks solvable by @Uliseis in #323
• fix(init): init --sop drafts "a call like this must include Y" as never with $in and $exists by @Uliseis in #321
• feat(init): init prints each drafting step's token usage and estimated cost by @Uliseis in #324
• feat(init): init --transcripts drafts reference calls and lints each drafted task by @Uliseis in #327
• docs(docs): the quickstart's toy agent remembers the verified carrier per session instead of asking for the MC number every turn by @Uliseis in #320
• feat(cli): demo --live plays the caller with Claude Haiku 5.5 by default by @Uliseis in #330
• fix(docs): sandtable.dev documents the latest release, deployed from a docs-live branch each release moves by @Uliseis in #331
• docs(plugin): the drafting skill says how to install the CLI with its optional packages by @Uliseis in #312
• docs(plugin): before a paid init step, the drafting skill states the calls, the model and a cost bound by @Uliseis in #315
• docs(plugin): the review checklists check a drafted task against its transcript, and a step-only rule becomes soft by @Uliseis in #318
• fix(init): init --transcripts shows the model each param's allowed values, so drafted references use them by @Uliseis in #337
• fix(plugin): the plugin evals run the CLI in Claude Code's sandbox, and the skills say what to do on an OpenSSL config error by @Uliseis in #338
• docs(plugin): the skills never read sandtable's own code to learn what the CLI supports, and say how Bedrock model names work by @Uliseis in #340
• feat(init): init --dry-run prints the drafting plan and its cost bound, and the drafting skill runs it before a paid init by @Uliseis in #339
• chore(release): v0.2.0 by @Uliseis in #335

Full Changelog: v0.1.2...v0.2.0