v1.2.0

A minor release for agents you can't change, with no breaking changes. run --serve serves each trial's simulated world over REST, MCP and webhooks for an agent that calls its own tools, and sandtable serve grades the conversations a hosted voice platform's own test calls drive. A task's faults make tool calls time out, hit a rate limit or fail, and its events change the world mid-call. sandtable import vapi drafts a domain from a Vapi assistant's export. The freight example gains two fault tasks and a rule, so stored freight runs can't be regraded under it. The plugin's hooks cover more ways of running and installing sandtable.

Added

• Spec: a task's faults make its tool calls fail the way a flaky dependency does: a timeout, a rate_limit (a 429 with retry_after) or a server_error (a 5xx status), a latency_ms delay, or both, on the calls of a tool that call names. With after_commit: true, a timeout or server error comes after the tool's write landed, the trap where a retry books twice. Invariants check a faulted call as any other, and an invariant can catch the unsafe retry. lint replays a task's reference with its faults and warns about one the reference never reaches (#15).
• Spec: a task's events change the world mid-call, as someone else would: right after the call-th call of a tool, update sets fields of a row or create adds one, such as another carrier booking a load the agent just found. validate checks their tools, tables and rows (#15).
• run --serve [port]: each trial's simulated world is served over HTTP while it runs, for an agent that calls its tools itself. REST at POST /tools/<name> and at each tool's http route, MCP (Streamable HTTP) at /mcp, and a webhook at /webhook that answers Vapi's and Retell's tool calls. Every call is graded like an in-process one, and a task's faults answer with their HTTP status. Trials run one at a time unless you pass -c, so the base URL means the trial that is running; an HTTP agent receives its trial's own URLs as world on every request, and a module agent as ctx.world. The library exports startWorldServer and runSuite takes server (#443).
• sandtable serve <domain>: serves the simulated world for conversations another system drives, such as a hosted voice platform's own test calls, and grades each one when it ends. A conversation opens with a platform's call id, an MCP initialize, POST /sessions or the first request, and ends with Vapi's end-of-call report, Retell's call_ended, POST /sessions/<id>/end or an MCP DELETE. It is graded on the end state, the invariants, the tools called and, with the transcript, what the agent said. results.json and report.html are rewritten after each conversation, and Ctrl-C prints the summary. With run --serve, this closes black-box mode (#444, #14).
• Spec: a tool's optional http: { method, path } is its route in the customer's API, such as GET /loads/{load_id}, served by run --serve with arguments from the path, query and body. init --openapi drafts it for each operation, and the freight example has it for the five tools its API defines (#443).
• results.json: a tool call that came over run --serve's world server has via (rest, mcp or webhook), which the HTML report shows (#443).
• results.json: a tool event that a fault hit has fault (error, status, retryAfter, latencyMs, and committed for an after_commit fault), and one that a task's event followed has eventChanges. The HTML report shows both on the call. The library exports the Fault, TaskEvent and FaultApplied types (#15).
• sandtable import vapi <path>: drafts a domain from a Vapi assistant's export, with no model call. The system prompt becomes policy.md, the function, transferCall and endCall tools become tools, and each simulation scenario becomes a task. A scenario's tool mocks become that task's seed rows and expected calls, and its variable values its given.context. Everything is pending review, with notes on what the config can't say. examples/vapi in the repository is a fictional export to try it on (#34).

Changed

• Roadmap: ROADMAP.md and sandtable.dev/roadmap list what 1.2.0 ships, under In 1.2, instead of the black-box milestone under Next, and say that nothing is planned as Next yet. The comparison pages no longer call serving the world, faults and importing Vapi configs planned (#451).
• examples/freight: two tasks with faults, booking-times-out-after-it-lands and search-rate-limited. There is a new invariant, book-available-load, which catches an agent that books a load again after a booking timed out though it went through. A new policy line says what to do when a tool fails, and the scripted agents handle it: careful looks the load up before booking again and searches again after a rate limit, and naive books again. The example now has 52 tasks and 20 invariants, and the naive agent scores pass^1 9.6%. Because policy.md and the invariants changed, regrade and leaderboard resume refuse stored runs of every freight task, and run --baseline against an older freight run labels every task changed (#15).
• Plugin: the drafting skill and the plugin guide say that pnpm installs only releases at least a day old by default, so on the day of a release pnpm add -D sandtable-ai installs the one before, which is older than the skills describe. The skill has Claude compare npx sandtable-ai --version with the latest release, say so when it's older and use what that version's --help lists; the skill and the guide give pnpm add -D sandtable-ai@<version>, and minimumReleaseAgeExclude for a project that sets its own age, for getting the newest (#433).

Fixed

• Plugin: the install hook now stops the unrelated npm package named sandtable from being installed or run in sandtable's place. Asked to "install sandtable", Claude had run pnpm add -D sandtable, which installs another project. An install of sandtable with npm, pnpm or yarn is held, with the same command for sandtable-ai to run instead. Running sandtable through npx is held where no sandtable-ai is installed, since npx would fetch the other package (#441).
• Plugin: the plan hook now resolves the relative paths of npm run sandtable -- init from the package root, where npm runs the script, instead of from the shell's folder. In a project with a sandtable script, a paid run from a subfolder now gets the plan of a dry run of the same files, and no longer gets the plan of a dry run that read other files. The root is the nearest folder with a package.json or a node_modules folder, or the one --prefix names, as for npm; for a workspace's script (-w), the plan isn't known (#430).
• Plugin: the plan hook now catches a model-backed init run as npm run sandtable --silent -- init, npm run sandtable -C .. -- init or with other npm options between the script's name and the --. Before, such a run had no dry-run check and no plan message (#439).

What's Changed

Other changes

• fix(plugin): the plan hook resolves npm run paths from the package root by @Uliseis in #436
• docs(plugin): pnpm installs a sandtable-ai at least a day old by @Uliseis in #438
• fix(plugin): the plan hook catches npm run sandtable with npm options before -- by @Uliseis in #440
• feat(spec): faults and events for tasks: injected tool failures and mid-call world changes by @Uliseis in #442
• feat(run): serve each trial's world over REST, MCP and webhooks with run --serve by @Uliseis in #445
• feat(cli): sandtable serve grades conversations a hosted platform drives by @Uliseis in #446
• feat(init): sandtable import vapi drafts a domain from a Vapi assistant's export by @Uliseis in #447
• fix(plugin): the install hook stops the unrelated sandtable package from being installed or run by @Uliseis in #448
• docs(docs): the roadmap lists what 1.2.0 ships by @Uliseis in #452
• chore(release): v1.2.0 by @Uliseis in #453

Full Changelog: v1.1.3...v1.2.0

v1.1.3

A patch release with fixes to the Claude Code plugin's two hooks. The plan hook also catches init run through npm exec or npm run. It no longer shows a dry run's plan for a paid run in another folder or region. The install hook covers pnpm and yarn as well as npm. It suggests npm init -y with each of them, and a command that installs into the folder it checked. There are no breaking changes.

Fixed

• Plugin: the hook that shows a model-backed init's plan now also catches npm exec sandtable-ai -- init, npm run sandtable -- init and npx sandtable-ai -- init, which ran with no dry run and no message before. It compares the environment variables that pick the provider and region too, so a paid run under another AWS_REGION= than its dry run says the plan isn't known, instead of showing the other region's plan (#415).
• Plugin: the hook that keeps sandtable in the working folder's own project now covers pnpm add, pnpm install <package> and yarn add, which also install into the nearest project above a folder with no package.json. It tells Claude to run npm init -y first with those too, because pnpm init adds a devEngines field that makes npx sandtable-ai refuse to run. It also checks the folder that yarn's --cwd names (#415, #432).
• Plugin: when the install hook stops an install that runs in another folder, after a cd or with yarn's --cwd, the command it suggests now starts with a cd to that folder. Before, it left the cd out, so the suggestion, run as written, installed into the project above (#432).
• Plugin: the plan hook no longer shows a dry run's plan for a paid init whose relative paths name other files. It resolves each command's paths from the folder it ran in, following any cd in the command, so cd eng && npx sandtable-ai init --sop sop.md after a dry run of --sop sop.md in the folder above now says the plan isn't known. A folder change it can't follow, such as cd $DIR or a subshell, makes the plan unknown too. A bare AWS_REGION=…; before the command, which reaches init only when the variable is exported already, now matches only a dry run with the same assignment (#427).

What's Changed

Other changes

• fix(plugin): the hooks cover init after --, the region from the environment, and pnpm and yarn installs by @Uliseis in #426
• fix(plugin): the plan hook compares paths by the folder each init ran in by @Uliseis in #429
• fix(plugin): the install hook says npm init -y with pnpm and yarn too by @Uliseis in #434
• chore(release): v1.1.3 by @Uliseis in #435

Full Changelog: v1.1.2...v1.1.3

v1.1.2

A patch release with two fixes to init's messages. --sop and --transcripts with no domain yet say to draft one with --openapi first. A drafting reply cut off at the output limit, as a Claude Haiku 5.5 reply can be while thinking, says so, notes that the call was still paid for, and names a larger model to try. There are no breaking changes.

Fixed

• init --sop and init --transcripts with no domain at -o, and no --openapi, now say that they add to an existing domain and give the command that creates one, init --openapi <spec> -o <dir>, or say to pass --openapi in the same run. Before, the error said to draft the domain with init, the command that had just failed. They still exit 2 before any model call. init --help says that -o is created by --openapi (#421).
• init --sop and init --transcripts: when a drafting reply stops at the 16,000-token output limit, as Claude Haiku 5.5 can while thinking, init now says so. It says the call was still paid for and names a larger model to try, such as -m claude-sonnet-5-5. Before, --sop failed with "the model returned no parseable invariants" and --transcripts warned "no usable proposal" for each chat, with no reason. Each chat's warning now gives the reason, and one more says what to try (#422).

What's Changed

Other changes

• fix(init): --sop and --transcripts with no domain say to draft one with --openapi first by @Uliseis in #423
• fix(init): a drafting reply cut off at the output limit says so, and what to try by @Uliseis in #424
• chore(release): v1.1.2 by @Uliseis in #425

Full Changelog: v1.1.1...v1.1.2

v1.1.1

A patch release with one fix, to the Claude Code plugin: when Claude runs a model-backed init, the engineer sees the plan of its dry run, the calls and what they cost, on any model, and Claude must dry-run it first. There are no breaking changes.

Fixed

• Plugin: when Claude runs a model-backed init (--sop or --transcripts without --dry-run), a hook shows the engineer the plan of its dry run, the calls and what they cost, as a message from the plugin, and stops the command if Claude hasn't dry-run it first. The plan comes from a dry run of the same command; for a command no dry run covered, the message says the plan isn't known. Claude Opus 5.5, the default model, often ran the paid init straight after the dry run without writing the plan, though the drafting skill asks for it. The hook doesn't wait for a go-ahead; the skills still ask for that. It needs Node.js, and it lets the command run when it can't read the session's transcript (#397).

What's Changed

Other changes

• fix(plugin): a hook shows the engineer the dry run's plan when a model-backed init runs by @Uliseis in #419
• chore(release): v1.1.1 by @Uliseis in #420

Full Changelog: v1.1.0...v1.1.1

v1.1.0

A minor release for customer sign-off, with no breaking changes. run shows which rules its trials checked and which they caught broken, lint says which rules the tasks put in play, and report --audience customer writes a page a customer's operations lead signs off on before go-live, with run --without-policy as the careless agent that shows the scenarios can catch a mistake. run --tag runs only the tasks with those tags, a repeated --tasks adds its tasks, and the Claude Code plugin keeps npm install of sandtable in the working folder's own project. An empty --tasks is deprecated: it still runs every task, but warns, and 2.0 will make it a usage error.

Added

• run --tag <tags>: runs only the tasks with at least one of the comma-separated tags, such as --tag refusal,negotiation, or of every repeated --tag; untagged selects the tasks with none. With --tasks, it runs the tasks both select. A tag no task carries, an unknown task id, or --tasks and --tag with no task in common stops the run with exit 2 before anything runs (#21).
• Coverage: run shows which rules it checked and which it caught broken. A trial checks a rule when the agent calls the tool the rule guards, or, for a never_disclose rule, is shown a value it must not say; only a trial that broke a rule, and failed for it, shows that a task catches a violation of it. A broken soft rule is reported apart, since it never fails a trial. The terminal summary counts the rules checked and caught broken and names the rules no trial checked and the tools no trial called. report.html draws a rule × task matrix, with a key, that marks the trials that broke each rule and outlines the rules nothing checked. results.json keeps it as coverage, regrade adds it to older runs, and the library exports its types Coverage, RuleCoverage, CoverageCell and ToolCoverage. It never changes a verdict, a score or an exit code (#30).
• lint says how many rules at least one task puts in play, warns about each rule no task does, and lists the tools no task uses. A task puts a rule in play when its reference calls the rule's tool or its called or not_called names it. For never_disclose, a reference call or the task's context must show the agent a protected value. A domain that linted clean can now show warnings, but warnings never fail lint. The library's LintReport adds rules and unusedTools, and each TaskLint its rules (type RuleLint) (#30).
• report --audience customer writes customer-report.html: a static page a customer's operations lead can sign off on before go-live. It has a plain-language summary with the open items, each rule with its source quote, the conversations that checked it and an example, every scenario's result, what the rehearsal doesn't cover, and a sign-off block. --probe <results.json>, a deliberately careless agent's run of the same domain, adds which rules the rehearsal caught it breaking, kept apart from the rules that were only checked. It reads the run's coverage, so a run from before 1.1 needs regrade first. The library exports it as renderCustomerHtml (#31).
• run --without-policy rehearses a careless agent: a built-in model agent (claude, claude:<model>, bedrock:<model-id>, openai:<model>) prompted without the domain's policy, keeping the generic instructions, the tools and each task's context. The rules it breaks that fail a conversation are the evidence report --audience customer --probe needs to show which rules the rehearsal catches, on any domain; a soft rule it breaks is reported, not caught. results.json records withoutPolicy: true, the agent's name ends in (without policy), and the inputs hashes cover the prompt it got, so --baseline, compare and regrade never take it for a run of the real agent. Any other agent exits 2 before anything runs. The library's SuiteOptions adds withoutPolicy (#405).

Changed

• examples/freight: do-not-use-carrier and do-not-use-capacity now have a reference that looks the carrier up, so lint proves them solvable and finds the rule against telling a caller why it is flagged in play. Their caller prompts and grading are unchanged, so stored runs of them still regrade (#30).
• examples/freight: 18 of its 19 rules now quote, as source, the sentence of policy.md they come from, so the customer report shows where each rule comes from. Grading is unchanged and stored runs still regrade. A trial that broke one of them now quotes that sentence in its attribution's evidence (#31).
• run and leaderboard: a repeated --tasks adds its task ids, as --tag and --floor do, so --tasks a --tasks b runs both. Before, only the last --tasks counted (#403).
• Roadmap: ROADMAP.md and sandtable.dev/roadmap list what 1.1.0 ships, under In 1.1, instead of under Next. PR comments with pass^k deltas and Retell and ElevenLabs imports move to Later, matching their milestones, and --tag is no longer a good first issue (#416).

Deprecated

• run and leaderboard: an empty --tasks (--tasks "", as an unset CI variable gives) still runs every task, and an empty value next to task ids still adds nothing, but both now warn on stderr. sandtable-ai 2.0 will make it a usage error (exit 2), as an empty --tag is. To run every task, leave --tasks out (#403).

Fixed

• Plugin: a PreToolUse hook stops npm install of sandtable-ai in a folder with no package.json, and tells Claude to run npm init -y there first. Without that step npm installs into the nearest project above the folder, even the engineer's home folder. The skills already asked for it (1.0.1); the hook holds on any model, even when no skill loaded (#412).

What's Changed

Other changes

• feat(cli): run --tag runs only the tasks with those tags by @Uliseis in #398
• feat(report): a coverage matrix shows which rules a run checked and caught broken by @Uliseis in #399
• feat(report): report --audience customer writes a page a customer signs off on before go-live by @Uliseis in #400
• feat(run): run --without-policy rehearses a careless agent for the customer page's --probe by @Uliseis in #407
• fix(cli): a repeated --tasks adds its tasks, and an empty --tasks warns that 2.0 will refuse it by @Uliseis in #414
• fix(plugin): a hook keeps npm install of sandtable-ai in the working folder's own project by @Uliseis in #413
• docs(docs): the roadmap lists what 1.1.0 ships by @Uliseis in #417
• chore(release): v1.1.0 by @Uliseis in #418

Full Changelog: v1.0.1...v1.1.0

v1.0.1

A patch release with three fixes. A failed transcript call in init --transcripts no longer throws away the chats already drafted, and a configuration error stops it at once. init's next-step commands run as printed for any folder name. The Claude Code plugin installs sandtable into the working folder's own project, never a parent's. There are no breaking changes.

Fixed

• init --transcripts: a transcript whose call fails after the client's retries no longer throws away the transcripts drafted before it. init writes what was drafted, names each failed transcript with its error, prints the init --transcripts command that drafts only the transcripts left, and still exits 2. It goes on past an error that concerns one transcript alone (an HTTP status of 400, 413, 429 or 5xx, such as a Bedrock 503 on a long chat). It stops at one that every later call would hit too (401, 403, 404, or no HTTP status, such as missing credentials), names it once and lists the transcripts it didn't send (#402).
• init and validate: the next-step commands they print quote a folder whose name has a space or another character the shell would split or expand, such as npx sandtable-ai review 'dom three', so they run as printed (#409).
• Plugin: the drafting and results skills install sandtable-ai and its optional packages into the working folder's own npm project. In a folder with no package.json, they run npm init -y first, since npm otherwise installs into the nearest project above it, even the engineer's home folder. Afterwards they say which package.json they changed (#401).

What's Changed

Other changes

• fix(init): init --transcripts keeps the chats it drafted when one fails, and names the failed ones by @Uliseis in #406
• fix(init): init's next-step commands quote paths, so they run as printed by @Uliseis in #410
• fix(plugin): the skills install sandtable into the working folder, never into a parent project by @Uliseis in #404
• chore(release): v1.0.1 by @Uliseis in #411

Full Changelog: v1.0.0...v1.0.1

v1.0.0

sandtable 1.0. The domain spec, results.json, the CLI and its exit codes, the HTTP step protocol and the library API are now stable: they change only in a major release, as the stability policy says. There are no breaking changes since 0.3.0. Coming from 0.3.0, read 1.0.0-rc.1 as well. Since the release candidate, the drafting skill waits for a go-ahead after a dry run, and the docs settle the last contract questions.

Changed

• Status: sandtable leaves beta. The README, the docs, the docs site's banner and the PyPI classifier (Development Status :: 5 - Production/Stable) say it is stable since 1.0, and the CI guide now suggests pinning the major version (^1.0.0) instead of an exact one. From 1.0, the stability policy applies: a breaking change comes only in a major release, after a minor release that deprecates the item (#353).
• Docs: the stability policy names the files the CLI reads as covered (the leaderboard models file and the --prices file), says the bundled examples' demo agents aren't, and says the rules behind attribution, callerNotes and init's copied from notes can improve in a minor release while their fields stay. The library reference documents World's keyOf and hasTable, and the CLI reference both shapes of a --prices file (#394).

Fixed

• Claude Code plugin: after an init --dry-run, drafting-a-domain writes the plan (calls, model, provider and dollar bound) to the engineer as a message of its own, then ends its turn and waits for a go-ahead, or, when model steps are already authorized, runs the model-backed init right after that message. Before, an agent could go from the plan line straight to the paid command (#392).

What's Changed

Other changes

• docs(docs): the library reference lists every exported type, checked by a test by @Uliseis in #389
• feat(init): init --transcripts flags names, usernames, emails and phones a draft copied from its chat by @Uliseis in #390
• chore(release): v1.0.0-rc.1 by @Uliseis in #391
• docs(docs): the stability policy names the files the cli reads, the demo agents and the rule-based results by @Uliseis in #395
• fix(plugin): the drafting skill stops after an init dry run and waits for the engineer's go-ahead by @Uliseis in #393
• chore(release): v1.0.0 by @Uliseis in #396

Full Changelog: v0.3.0...v1.0.0

v1.0.0-rc.1

The first release candidate for 1.0, published under npm's next tag. The contracts are as 0.3.0 left them. It adds a local check that flags names, usernames, emails, phone numbers and street addresses an init --transcripts draft copied from its chats, sandtable review printing each item's review notes, and the full list of exported types on the library reference. There are no breaking changes since 0.3.0.

Added

• init --transcripts: after drafting, a local check with no model call flags the values a task copied from its chat:

  • names the caller gave, including a customer message that is only a name;
  • usernames, and an email's local part on its own;
  • the chat's emails and phone numbers, and emails made from its usernames;
  • street addresses.

  Each gets a # TODO(review): copied from <file>: replace "<value>" with a fictional value note above the task, saying where it is, and the summary prints how many it found, next to the PII redactions. Before, a draft could keep a caller's real name, or rebuild an email from a kept username, with nothing pointing at it. Order, account and load IDs, and values the domain already had, aren't flagged (#388).

• sandtable review prints the # TODO(review): notes init left above each pending tool, invariant and task, under the item. Before, it printed only the item and its quote, and for a tool a pointer to the comments in sandtable.yaml (#388).

Fixed

• Docs: sandtable.dev/reference/library lists every type the package exports, such as AttributionCounts, Owner, SuiteOptions and ToolSchema, grouped by what it describes, and a test fails when an exported type is missing from it (#353).

What's Changed

Other changes

• docs(docs): the library reference lists every exported type, checked by a test by @Uliseis in #389
• feat(init): init --transcripts flags names, usernames, emails and phones a draft copied from its chat by @Uliseis in #390
• chore(release): v1.0.0-rc.1 by @Uliseis in #391

Full Changelog: v0.3.0...v1.0.0-rc.1

v0.3.0

The last release before 1.0 that changes contracts. It adds:

• --fail-on-regression and sandtable compare for regressions against a baseline
• --floor per-tag gates
• a likely owner for each trial that didn't pass
• tool params that are objects and lists
• a library API cut down to what is documented, results.json versions that readers check, and a stability policy for 1.0

It also makes leaderboard and demo --live treat critical tasks and errored trials as run does, and requires Node.js 22 and Python 3.10. Read the six Breaking: entries under Changed before upgrading.

Added

• Library: parseDomain(value) checks a domain built in code as loadDomain checks a file, the schema and then the cross-references, and throws a DomainError. A new page, sandtable.dev/reference/library, documents every export of the package (#370).
• results.json records the sandtable-ai that wrote it, as sandtable: { version }, and sandtable regrade adds regradedWith. When regrade refuses a run because what the agent or the caller is given changed, and the run came from another version, it says the version that ran it and the command that regrades it with that version, such as npx sandtable-ai@0.2.1 regrade runs/results.json -d ./sandtable. A leaderboard's resumed results stay as stored unless a verdict changes (#372).
• sandtable run --fail-on-regression, with --baseline: the run exits 1 when a task that passed every graded trial in the baseline fails one now, and names each such task with its passes before and after, so a pull request that breaks a task fails CI even when pass^k stays above --fail-under. Tasks that got worse without ever passing every trial, tasks whose test changed since the baseline, and new failing tasks are still listed but don't gate. Without --baseline it is a usage error (exit 2), before any trial runs (#281).
• sandtable compare <baseline.json> <results.json>: shows what changed between two runs of one domain that already exist, with the same block run --baseline prints (broken, worse, fixed, better and changed tasks, added and removed ones, pass^1 and pass^k before and after), without calling a model. Use it for a pull request's run and the base branch's made in separate CI jobs, or for runs made without --baseline. --fail-on-regression exits 1 when a task broke, and -o <dir> writes the newer run's results.json and report.html with the baseline attached. Runs of two domains exit 2 (#285).
• sandtable run --floor <tag>=<pct>, repeatable, gates one tag's pass^k as --fail-under gates the whole suite's, so a strong happy path can't carry weak refusals over the gate: --floor refusal=100 --floor negotiation=70 exits 1 and names each tag below its floor, with or without --fail-under. A tag that no task in the run carries, a value that isn't <tag>=<pct>, a percentage outside 0 to 100 or a tag given twice stops the run before any trial with exit 2, so a typo can't make the gate pass. The By tag table in the terminal and in report.html shows each floor and marks a tag below it, and results.json records the floors as floors (#283).
• sandtable leaderboard: when the board runs critical tasks, a Critical column in the terminal table, LEADERBOARD.md and leaderboard.html shows how many each model passed in every trial, 3/3, or ✗ 2/3: expired-insurance with the ones it failed. Each row of leaderboard.json has them as critical: { tasks, failed }, and an error row from skipped trials has skipped, how many there were (#284).
• Failure attribution: each trial that didn't pass gets a first suspect in results.json, attribution: { owner, evidence }, from deterministic rules, with no model call. The first rule that applies wins: harness for a skipped trial, task-spec when lint proves the task can't pass as written, simulated-user when a caller note (a role break) comes before the trial's first hard failure, and agent otherwise, as the default. results.json counts trials per owner in attribution, the terminal summary prints them on one line (not passed, by likely owner: agent 9 · simulated user 2), and the HTML report shows each trial's owner and evidence. pass^k, verdicts and exit codes don't change. A results.json from an earlier version has no attribution until sandtable regrade re-scores it. The library exports the Attribution, AttributionCounts and Owner types, and the plugin's attribution workflow starts from the field and says when its reading of the transcript disagrees (#29).
• Domain spec: tool params can be objects and lists, so a tool can take line items or an address: type: object with properties, and type: array with items, nested to any depth. The agent gets nested JSON Schema, and an argument error names its path, such as argument "items[1].qty" must be an integer; an object or list a model sends as a JSON string is parsed. nullable: true keeps an explicit null, to clear a field. Templates read inside an object or list param ($address.zip, ${items.0.sku}), a dotted field in any where clause reads a nested value ("address.zip": "94105"), and $contains on a list matches an element, so a task can check called: [{ tool: create_order, args: { items: { $contains: { sku: A-100, qty: 2 } } } }]. A model that leaves out a required nullable argument is told (it may be null). Existing domains load, run and grade as before, and their tool schemas and inputs hashes don't change (#26).
• init --openapi: a request body's object fields and arrays become nested params, resolving $refs up to 3 levels deep, instead of being skipped. A body field it still can't map, such as one with several types or an array without items, gets a TODO(review) note on its tool as well as the warning (#26).
• Domain spec: a require invariant can say skip_missing: true, so a call naming a record that doesn't exist doesn't break that rule. A rule about a record's fields can then leave a made-up ID to a rule that the record exists (where: {}), and the made-up ID is reported once, under that rule (#163).
• Docs: a stability policy, at sandtable.dev/reference/stability, says what 1.0 will keep stable (the domain spec, results.json and leaderboard.json, CLI commands, flags and exit codes, the HTTP step protocol and the library's exports), what isn't covered, how version numbers signal a break, how long a deprecated item keeps working, and what counts as a change to a score (#352).

Changed

• results.json: report, regrade, run --baseline and leaderboard refuse a results.json whose format version is newer than the one they read, with exit 2 and a message saying to use a newer sandtable-ai, instead of reading fields that may have changed meaning. The version moves only when a field is removed or changes meaning, never for an added field, as docs/reference/results-schema.mdx now states. Every results.json sandtable has written is version 1, and a hand-made one without version is read as version 1, so none that loaded before is refused (#349).

• Breaking: Library: the package exports only its documented API, which 1.0 will keep stable. Removed:

  • the zod schemas (DomainSpec, TaskSpec, ToolSpec, ParamSpec, TableSpec, OpSpec, CheckSpec, FieldRef, InvariantSpec, StateAssertSpec, CallExpectSpec, ExpectSpec, ReferenceCallSpec, PersonaSpec, GivenSpec): use parseDomain(value) instead of DomainSpec.parse(value). The TypeScript types (Domain, Task, CallExpect, …) are still exported, so zod's version is no longer part of sandtable's API.
  • runEpisode, gradeTrial, regradeTrial, checkCall, invariantTool: use runSuite with taskIds and k: 1, or regradeSuite.
  • worldFor, seedFor, diffSnapshots: use new World(domain.world) and its snapshot().
  • agentSystemPrompt: an agent receives the same prompt as AgentContext.system.
  • userSystemPrompt, STOP_TOKEN, callerNotes, binom, DEFAULT_MODEL, DEFAULT_OPENAI_BASE_URL: internals with no replacement; a run's results.json still carries each trial's callerNotes.

  Code that imports only the documented exports is unaffected (#370).

• Breaking: Node.js and Python: sandtable-ai needs Node.js 22 or later (engines.node is >=22), and the PyPI launcher needs Python 3.10 or later. Node.js 20 reached end of life on 2026-04-30 and Python 3.9 in October 2025, and 1.0 shouldn't promise to keep supporting them. On Node 20, npm install and npx sandtable-ai warn about the engine, and an install with engine-strict fails; pip install sandtable-ai on Python 3.9 keeps installing 0.2.1, the last launcher that supports it, which runs npx sandtable-ai@0.2.1. Upgrade Node.js to 22 or later (a CI job's node-version: 22), and Python to 3.10 or later for the launcher. CI now tests Node 22, 24 and 26 (#350).

• Breaking: sandtable leaderboard: a model that failed any trial of a critical task now ranks below every model that failed none, whatever its pass^k, sinc...

v0.2.1

A patch release with no breaking changes: validate names the installed version and a command to check a newer one when a key is unknown, a model whose reply is only reasoning is asked again instead of leaving the caller in silence, a module agent's load error reads as two sentences, and the plugin checks a failed rule against its SOP quote before blaming the agent.

Added

• sandtable validate, and every command that loads a domain: an unrecognized key is followed by a line naming the installed version, such as sandtable-ai 0.2.1 doesn't know this key: check its spelling, or whether a later release added it, so a key from a newer release than the one installed is easy to tell from a typo (#341).

Changed

• Claude Code plugin: before blaming the agent for a failed invariant, running-and-reading-results and the failure-triager subagent read the rule's source.quote and check that the SOP requires what the rule checks: the same order of calls, bound and condition. A rule stricter than the SOP is attributed to the task spec, with the fix to relax it or make it soft: true and confirm with regrade, instead of the agent being blamed for following the SOP (#343).
• Roadmap: ROADMAP.md and sandtable.dev/roadmap list what 0.2.0 shipped, instead of calling it unreleased, and show the path to 1.0: 0.3.0 as the last release that may change the contracts, what 1.0 promises to keep stable, and what comes after it (#347).

Fixed

• run --agent <module>: when a JS or TS agent fails to load, sandtable's advice now starts a new sentence after Node's error, as in Unexpected token 'export'. Node read it as CommonJS, …, instead of running on from it (#345).
• Agents and callers on other models: a reply that is nothing but reasoning, with no tool call, no longer reaches the other side as silence. The openai: adapter (agent and caller) and the Bedrock Converse agent (bedrock:<model-id>) removed the reasoning and sent an empty message, so the caller heard nothing, asked "Hello?" and the turn counted against max_turns; gpt-oss-120b through Bedrock's OpenAI-compatible endpoint did this in 4 of 56 agent turns. Both now ask for such a reply again, up to twice, as Converse already did for reasoning that never closes, and the discarded replies never enter the conversation. If the third reply is still empty, it is sent as before and counted in results.json's usage as emptyAfterStrip, next to reasoningStripped, so a run shows how often the other side heard silence; the terminal summary and the HTML report show the count when it isn't zero (#332).

What's Changed

Other changes

• fix(cli): a module agent's load error starts sandtable's advice on a new sentence after node's message by @Uliseis in #354
• fix(adapters): a reply that is only reasoning is asked for again instead of reaching the other side as silence by @Uliseis in #358
• fix(plugin): attribution checks a failed invariant against its sop quote before blaming the agent by @Uliseis in #361
• docs(docs): the roadmap lists what 0.2.0 shipped and the path to 1.0 by @Uliseis in #360
• feat(cli): validate names the installed version after an unrecognized key by @Uliseis in #356
• chore(release): v0.2.1 by @Uliseis in #379

Full Changelog: v0.2.0...v0.2.1