Skip to content

v0.3.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 22 Oct 14:00
· 51 commits to main since this release
v0.3.0
515b0cb
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

The last release before 1.0 that changes contracts. It adds:

  • --fail-on-regression and sandtable compare for regressions against a baseline
  • --floor per-tag gates
  • a likely owner for each trial that didn't pass
  • tool params that are objects and lists
  • a library API cut down to what is documented, results.json versions that readers check, and a stability policy for 1.0

It also makes leaderboard and demo --live treat critical tasks and errored trials as run does, and requires Node.js 22 and Python 3.10. Read the six Breaking: entries under Changed before upgrading.

Added

  • Library: parseDomain(value) checks a domain built in code as loadDomain checks a file, the schema and then the cross-references, and throws a DomainError. A new page, sandtable.dev/reference/library, documents every export of the package (#370).
  • results.json records the sandtable-ai that wrote it, as sandtable: { version }, and sandtable regrade adds regradedWith. When regrade refuses a run because what the agent or the caller is given changed, and the run came from another version, it says the version that ran it and the command that regrades it with that version, such as npx sandtable-ai@0.2.1 regrade runs/results.json -d ./sandtable. A leaderboard's resumed results stay as stored unless a verdict changes (#372).
  • sandtable run --fail-on-regression, with --baseline: the run exits 1 when a task that passed every graded trial in the baseline fails one now, and names each such task with its passes before and after, so a pull request that breaks a task fails CI even when pass^k stays above --fail-under. Tasks that got worse without ever passing every trial, tasks whose test changed since the baseline, and new failing tasks are still listed but don't gate. Without --baseline it is a usage error (exit 2), before any trial runs (#281).
  • sandtable compare <baseline.json> <results.json>: shows what changed between two runs of one domain that already exist, with the same block run --baseline prints (broken, worse, fixed, better and changed tasks, added and removed ones, pass^1 and pass^k before and after), without calling a model. Use it for a pull request's run and the base branch's made in separate CI jobs, or for runs made without --baseline. --fail-on-regression exits 1 when a task broke, and -o <dir> writes the newer run's results.json and report.html with the baseline attached. Runs of two domains exit 2 (#285).
  • sandtable run --floor <tag>=<pct>, repeatable, gates one tag's pass^k as --fail-under gates the whole suite's, so a strong happy path can't carry weak refusals over the gate: --floor refusal=100 --floor negotiation=70 exits 1 and names each tag below its floor, with or without --fail-under. A tag that no task in the run carries, a value that isn't <tag>=<pct>, a percentage outside 0 to 100 or a tag given twice stops the run before any trial with exit 2, so a typo can't make the gate pass. The By tag table in the terminal and in report.html shows each floor and marks a tag below it, and results.json records the floors as floors (#283).
  • sandtable leaderboard: when the board runs critical tasks, a Critical column in the terminal table, LEADERBOARD.md and leaderboard.html shows how many each model passed in every trial, 3/3, or ✗ 2/3: expired-insurance with the ones it failed. Each row of leaderboard.json has them as critical: { tasks, failed }, and an error row from skipped trials has skipped, how many there were (#284).
  • Failure attribution: each trial that didn't pass gets a first suspect in results.json, attribution: { owner, evidence }, from deterministic rules, with no model call. The first rule that applies wins: harness for a skipped trial, task-spec when lint proves the task can't pass as written, simulated-user when a caller note (a role break) comes before the trial's first hard failure, and agent otherwise, as the default. results.json counts trials per owner in attribution, the terminal summary prints them on one line (not passed, by likely owner: agent 9 · simulated user 2), and the HTML report shows each trial's owner and evidence. pass^k, verdicts and exit codes don't change. A results.json from an earlier version has no attribution until sandtable regrade re-scores it. The library exports the Attribution, AttributionCounts and Owner types, and the plugin's attribution workflow starts from the field and says when its reading of the transcript disagrees (#29).
  • Domain spec: tool params can be objects and lists, so a tool can take line items or an address: type: object with properties, and type: array with items, nested to any depth. The agent gets nested JSON Schema, and an argument error names its path, such as argument "items[1].qty" must be an integer; an object or list a model sends as a JSON string is parsed. nullable: true keeps an explicit null, to clear a field. Templates read inside an object or list param ($address.zip, ${items.0.sku}), a dotted field in any where clause reads a nested value ("address.zip": "94105"), and $contains on a list matches an element, so a task can check called: [{ tool: create_order, args: { items: { $contains: { sku: A-100, qty: 2 } } } }]. A model that leaves out a required nullable argument is told (it may be null). Existing domains load, run and grade as before, and their tool schemas and inputs hashes don't change (#26).
  • init --openapi: a request body's object fields and arrays become nested params, resolving $refs up to 3 levels deep, instead of being skipped. A body field it still can't map, such as one with several types or an array without items, gets a TODO(review) note on its tool as well as the warning (#26).
  • Domain spec: a require invariant can say skip_missing: true, so a call naming a record that doesn't exist doesn't break that rule. A rule about a record's fields can then leave a made-up ID to a rule that the record exists (where: {}), and the made-up ID is reported once, under that rule (#163).
  • Docs: a stability policy, at sandtable.dev/reference/stability, says what 1.0 will keep stable (the domain spec, results.json and leaderboard.json, CLI commands, flags and exit codes, the HTTP step protocol and the library's exports), what isn't covered, how version numbers signal a break, how long a deprecated item keeps working, and what counts as a change to a score (#352).

Changed

  • results.json: report, regrade, run --baseline and leaderboard refuse a results.json whose format version is newer than the one they read, with exit 2 and a message saying to use a newer sandtable-ai, instead of reading fields that may have changed meaning. The version moves only when a field is removed or changes meaning, never for an added field, as docs/reference/results-schema.mdx now states. Every results.json sandtable has written is version 1, and a hand-made one without version is read as version 1, so none that loaded before is refused (#349).

  • Breaking: Library: the package exports only its documented API, which 1.0 will keep stable. Removed:

    • the zod schemas (DomainSpec, TaskSpec, ToolSpec, ParamSpec, TableSpec, OpSpec, CheckSpec, FieldRef, InvariantSpec, StateAssertSpec, CallExpectSpec, ExpectSpec, ReferenceCallSpec, PersonaSpec, GivenSpec): use parseDomain(value) instead of DomainSpec.parse(value). The TypeScript types (Domain, Task, CallExpect, …) are still exported, so zod's version is no longer part of sandtable's API.
    • runEpisode, gradeTrial, regradeTrial, checkCall, invariantTool: use runSuite with taskIds and k: 1, or regradeSuite.
    • worldFor, seedFor, diffSnapshots: use new World(domain.world) and its snapshot().
    • agentSystemPrompt: an agent receives the same prompt as AgentContext.system.
    • userSystemPrompt, STOP_TOKEN, callerNotes, binom, DEFAULT_MODEL, DEFAULT_OPENAI_BASE_URL: internals with no replacement; a run's results.json still carries each trial's callerNotes.

    Code that imports only the documented exports is unaffected (#370).

  • Breaking: Node.js and Python: sandtable-ai needs Node.js 22 or later (engines.node is >=22), and the PyPI launcher needs Python 3.10 or later. Node.js 20 reached end of life on 2026-04-30 and Python 3.9 in October 2025, and 1.0 shouldn't promise to keep supporting them. On Node 20, npm install and npx sandtable-ai warn about the engine, and an install with engine-strict fails; pip install sandtable-ai on Python 3.9 keeps installing 0.2.1, the last launcher that supports it, which runs npx sandtable-ai@0.2.1. Upgrade Node.js to 22 or later (a CI job's node-version: 22), and Python to 3.10 or later for the launcher. CI now tests Node 22, 24 and 26 (#350).

  • Breaking: sandtable leaderboard: a model that failed any trial of a critical task now ranks below every model that failed none, whatever its pass^k, since run would block its release; within each group, models rank by pass^k as before. A board with critical tasks can therefore order its rows differently from 0.2: a script that reads the top row of leaderboard.json as the best model should check its critical.failed too (#284).

  • Breaking: sandtable leaderboard: a trial that ended in an error now counts as run counts it. One that ended in an error after the agent broke a hard rule or called a forbidden tool counts as failed, so the model is ranked with it instead of shown as an error row, and its results are stored and reused like any complete run. Only a skipped trial, one that ended in an error before the agent broke anything, still makes the model an error row, whose message now reads N of M trial(s) skipped after an error: <the first error> instead of N of M trial(s) crashed: … (#284).

  • Breaking: sandtable demo --live now applies run's gates: it exits 1 when any trial of one of the freight example's critical tasks failed, and 2 when a trial was skipped after an error. Before, it exited 0 whatever the agent did, unless every trial ended in an error. A script that runs demo --live and expects 0 should accept 1, as with run: npx sandtable-ai demo --live || [ $? -eq 1 ] (#284).

  • Breaking: examples/freight: four tasks' callers changed, so a live run fails them less often through no fault of the agent. In two-loads-rounds-restart, rounds-exhausted-then-alternative and load-already-covered, the caller now knows where its truck is empty and when it can pick up. Before, a live caller asked about it made a date up, such as "I can roll tomorrow morning", and a correct agent then passed on a load that picks up later (#346). In walk-away-above-ceiling, the caller answers a low offer without repeating the agent's number. Before, a caller that misheard $1,350 as "fourteen fifty", the load's ceiling, led a correct agent to say $1,450 while correcting it, and never-disclose-ceiling failed (#383). Since a task's caller instructions are part of a run's inputs hash, sandtable regrade refuses a stored freight run that includes these tasks, and sandtable leaderboard stops before running anything when a model's stored freight results include them. Regrade such a run with the version that made it (npx sandtable-ai@0.2.0 regrade …); for a board, move those models' folders away or pass --no-resume, which reruns every model and costs model calls. run --baseline against an older freight run lists these four tasks as changed.

  • examples/freight: booking a made-up load ID now breaks the new book-existing-load invariant instead of rep-confirm-not-booked, so a leaderboard's rule-violation counts no longer list an invented load as a rep-confirm booking. The trial fails as before. sandtable regrade re-files stored freight trials without a model call. Since the invariants are part of every task's grading hash, run --baseline against a freight run from an earlier version lists every task as changed and leaves it out of the before-and-after scores, once; the exit code is unchanged (#163).

  • Claude Code plugin: when reviewing drafted tasks, drafting-a-domain first checks that each task keeps its chat's case (the customer's account and membership, the order, the item and the request, with only names and identifiers made fictional), and only then that the expected end state is what the SOP requires. Before, a drafted task that turned a gold member into a guest could be explained away as "the rep did something else" (#377).

  • Roadmap: ROADMAP.md and sandtable.dev/roadmap list what 0.3.0 shipped, and show what is left for 1.0: a release candidate tested end to end, and leaving beta (#353).

Fixed

  • Failure attribution and the Claude Code plugin: a trial left with the default agent owner after a failed rule now quotes the rule's source in its evidence, or says the rule has none, so the rule gets checked before the agent is blamed. running-and-reading-results and failure-triager treat the stored attribution as a starting point, never as evidence: when their check finds another owner, such as the task spec for a rule stricter than the SOP, that owner is the answer instead of a caveat (#385).
  • Docs: the CLI reference lists NO_COLOR, and the spec reference describes source: { file, quote } on invariants and tasks. Tests now fail when a reference page misses a command's option, an environment variable, a spec key or a results.json field, so the pages stay complete (#371).

What's Changed

Other changes

  • test(docs): reference pages fail a test when they miss a flag, an env variable, a spec key or a results field by @Uliseis in #376
  • feat(report): results.json readers refuse a format version newer than theirs by @Uliseis in #362
  • feat(cli)!: the library exports only its documented api, adds parseDomain, and has a reference page by @Uliseis in #374
  • build!: require node 22 and python 3.10, since node 20 and python 3.9 are end of life by @Uliseis in #368
  • feat(report): results.json records the sandtable-ai that wrote it, and regrade names it when it refuses a run by @Uliseis in #380
  • feat(cli): run --fail-on-regression fails ci when a task broke since the baseline by @Uliseis in #364
  • feat(cli): sandtable compare shows what changed between two stored runs by @Uliseis in #367
  • feat(cli): run --floor gates a tag's pass^k, such as refusal=100 by @Uliseis in #366
  • feat(run)!: leaderboard and demo --live treat critical tasks and skipped trials as run does by @Uliseis in #369
  • feat(grading): each trial that didn't pass names a likely owner, from deterministic evidence by @Uliseis in #373
  • feat(spec): tool params can be objects and lists, and tasks grade them with $contains and dotted fields by @Uliseis in #378
  • fix(examples)!: four freight callers know their truck's availability or don't repeat the agent's numbers by @Uliseis in #363
  • fix(grading): a made-up load id is filed under its own freight rule, and require can skip missing records by @Uliseis in #365
  • fix(plugin): reviewing a drafted task checks it keeps its chat's case before checking its end state by @Uliseis in #381
  • docs(docs): a stability policy says what 1.0 keeps stable and how it changes by @Uliseis in #375
  • fix(plugin): a stored attribution is a starting point, and a failed rule's evidence points at its source by @Uliseis in #386
  • chore(release): v0.3.0 by @Uliseis in #387

Full Changelog: v0.2.1...v0.3.0

Contributors

Uliseis
Assets 2
Loading