v1.2.0

A minor release for agents you can't change, with no breaking changes. run --serve serves each trial's simulated world over REST, MCP and webhooks for an agent that calls its own tools, and sandtable serve grades the conversations a hosted voice platform's own test calls drive. A task's faults make tool calls time out, hit a rate limit or fail, and its events change the world mid-call. sandtable import vapi drafts a domain from a Vapi assistant's export. The freight example gains two fault tasks and a rule, so stored freight runs can't be regraded under it. The plugin's hooks cover more ways of running and installing sandtable.

Added

• Spec: a task's faults make its tool calls fail the way a flaky dependency does: a timeout, a rate_limit (a 429 with retry_after) or a server_error (a 5xx status), a latency_ms delay, or both, on the calls of a tool that call names. With after_commit: true, a timeout or server error comes after the tool's write landed, the trap where a retry books twice. Invariants check a faulted call as any other, and an invariant can catch the unsafe retry. lint replays a task's reference with its faults and warns about one the reference never reaches (#15).
• Spec: a task's events change the world mid-call, as someone else would: right after the call-th call of a tool, update sets fields of a row or create adds one, such as another carrier booking a load the agent just found. validate checks their tools, tables and rows (#15).
• run --serve [port]: each trial's simulated world is served over HTTP while it runs, for an agent that calls its tools itself. REST at POST /tools/<name> and at each tool's http route, MCP (Streamable HTTP) at /mcp, and a webhook at /webhook that answers Vapi's and Retell's tool calls. Every call is graded like an in-process one, and a task's faults answer with their HTTP status. Trials run one at a time unless you pass -c, so the base URL means the trial that is running; an HTTP agent receives its trial's own URLs as world on every request, and a module agent as ctx.world. The library exports startWorldServer and runSuite takes server (#443).
• sandtable serve <domain>: serves the simulated world for conversations another system drives, such as a hosted voice platform's own test calls, and grades each one when it ends. A conversation opens with a platform's call id, an MCP initialize, POST /sessions or the first request, and ends with Vapi's end-of-call report, Retell's call_ended, POST /sessions/<id>/end or an MCP DELETE. It is graded on the end state, the invariants, the tools called and, with the transcript, what the agent said. results.json and report.html are rewritten after each conversation, and Ctrl-C prints the summary. With run --serve, this closes black-box mode (#444, #14).
• Spec: a tool's optional http: { method, path } is its route in the customer's API, such as GET /loads/{load_id}, served by run --serve with arguments from the path, query and body. init --openapi drafts it for each operation, and the freight example has it for the five tools its API defines (#443).
• results.json: a tool call that came over run --serve's world server has via (rest, mcp or webhook), which the HTML report shows (#443).
• results.json: a tool event that a fault hit has fault (error, status, retryAfter, latencyMs, and committed for an after_commit fault), and one that a task's event followed has eventChanges. The HTML report shows both on the call. The library exports the Fault, TaskEvent and FaultApplied types (#15).
• sandtable import vapi <path>: drafts a domain from a Vapi assistant's export, with no model call. The system prompt becomes policy.md, the function, transferCall and endCall tools become tools, and each simulation scenario becomes a task. A scenario's tool mocks become that task's seed rows and expected calls, and its variable values its given.context. Everything is pending review, with notes on what the config can't say. examples/vapi in the repository is a fictional export to try it on (#34).

Changed

• Roadmap: ROADMAP.md and sandtable.dev/roadmap list what 1.2.0 ships, under In 1.2, instead of the black-box milestone under Next, and say that nothing is planned as Next yet. The comparison pages no longer call serving the world, faults and importing Vapi configs planned (#451).
• examples/freight: two tasks with faults, booking-times-out-after-it-lands and search-rate-limited. There is a new invariant, book-available-load, which catches an agent that books a load again after a booking timed out though it went through. A new policy line says what to do when a tool fails, and the scripted agents handle it: careful looks the load up before booking again and searches again after a rate limit, and naive books again. The example now has 52 tasks and 20 invariants, and the naive agent scores pass^1 9.6%. Because policy.md and the invariants changed, regrade and leaderboard resume refuse stored runs of every freight task, and run --baseline against an older freight run labels every task changed (#15).
• Plugin: the drafting skill and the plugin guide say that pnpm installs only releases at least a day old by default, so on the day of a release pnpm add -D sandtable-ai installs the one before, which is older than the skills describe. The skill has Claude compare npx sandtable-ai --version with the latest release, say so when it's older and use what that version's --help lists; the skill and the guide give pnpm add -D sandtable-ai@<version>, and minimumReleaseAgeExclude for a project that sets its own age, for getting the newest (#433).

Fixed

• Plugin: the install hook now stops the unrelated npm package named sandtable from being installed or run in sandtable's place. Asked to "install sandtable", Claude had run pnpm add -D sandtable, which installs another project. An install of sandtable with npm, pnpm or yarn is held, with the same command for sandtable-ai to run instead. Running sandtable through npx is held where no sandtable-ai is installed, since npx would fetch the other package (#441).
• Plugin: the plan hook now resolves the relative paths of npm run sandtable -- init from the package root, where npm runs the script, instead of from the shell's folder. In a project with a sandtable script, a paid run from a subfolder now gets the plan of a dry run of the same files, and no longer gets the plan of a dry run that read other files. The root is the nearest folder with a package.json or a node_modules folder, or the one --prefix names, as for npm; for a workspace's script (-w), the plan isn't known (#430).
• Plugin: the plan hook now catches a model-backed init run as npm run sandtable --silent -- init, npm run sandtable -C .. -- init or with other npm options between the script's name and the --. Before, such a run had no dry-run check and no plan message (#439).

What's Changed

Other changes

• fix(plugin): the plan hook resolves npm run paths from the package root by @Uliseis in #436
• docs(plugin): pnpm installs a sandtable-ai at least a day old by @Uliseis in #438
• fix(plugin): the plan hook catches npm run sandtable with npm options before -- by @Uliseis in #440
• feat(spec): faults and events for tasks: injected tool failures and mid-call world changes by @Uliseis in #442
• feat(run): serve each trial's world over REST, MCP and webhooks with run --serve by @Uliseis in #445
• feat(cli): sandtable serve grades conversations a hosted platform drives by @Uliseis in #446
• feat(init): sandtable import vapi drafts a domain from a Vapi assistant's export by @Uliseis in #447
• fix(plugin): the install hook stops the unrelated sandtable package from being installed or run by @Uliseis in #448
• docs(docs): the roadmap lists what 1.2.0 ships by @Uliseis in #452
• chore(release): v1.2.0 by @Uliseis in #453

Full Changelog: v1.1.3...v1.2.0