Poisson Prove 0.2.2

A release of the docs alone: the package's code is the same as 0.2.1's.

Added

• A new page, docs/what-a-declaration-checks.md: what a [declare] table records, stamps and
  prints, and what never reached means and does not. It held the second half of
  docs/what-capture-stores.md before, and the sweep spec now links to it for what steps and
  gates report.

Changed

• docs/recipes/in-ci.md gives every file a GitHub repository needs to gate its pull requests on
  Prove. make_comment.py is given in full: it reads the check --json file named on its command
  line and prints the comment, and an empty file, which a refused check leaves, gets a comment
  that says so. The main branch's workflow, prove-baseline.yml, is given in full beside the pull
  request's, prove.yml, which now also posts the comment. Every action in both is pinned to a
  full commit SHA. One line says that unresolved and timeout are the support bot's events, an
  example: on the quickstart's runs unresolved is refused with exit 1, so name your own.
• docs/artifact-format.md and docs/what-capture-stores.md are in short sections under headings
  you can jump by, with lists and tables where a paragraph listed, and an example of each stored
  format a paragraph describes, from a real run. No fact was cut.

Poisson Prove 0.2.1

Fixed

• With --json, stdout is the one JSON document whatever the target writes (#28). Before, a
  target that printed put its text in stdout beside the document and the sweep still exited 0, so
  a caller got success and a document it could not parse. Now everything the target writes to
  stdout goes to stderr instead, once, as it is written: a print in the rollout or at import, a
  write to sys.stdout.buffer, os.write(1, ...), the C library, a program the target starts,
  an atexit hook, and all of these under --episode-timeout. The same holds for
  ui --open --json, whose browser inherits stdout, and for what the Modal client or a Burla
  sweep's own process writes. The document, the exit codes and everything without --json are
  as they were.
• What that means for a caller of sweep --json: read both streams, because a caller that pipes
  stdout and stderr and reads only stdout can hang on a target that prints a lot (200 kB a
  rollout is enough; subprocess.run(capture_output=True) reads both); look for the line that
  starts error: , not stderr's first line, which can be the target's text; and with stderr a
  terminal the target sees sys.stdout.isatty() true, so a framework may draw there.
• A target that calls sys.exit, at import or in a rollout, no longer ends sweep --json with
  exit 0 and an empty stdout: the sweep exits 3 with one error: line. Without --json it ends
  the process as before.
• A target that closes its stdout no longer costs sweep --json its document.
• main() called in the caller's own process with --json and no stdout= leaves file
  descriptor 1 pointing at stderr for the rest of that process, so a host script that prints
  afterwards prints to stderr; Prove's own output from later calls still goes to the process's
  real stdout. A caller that passes stdout= or uses contextlib.redirect_stdout is untouched.
• Under --episode-timeout, text a rollout printed and had not yet flushed is no longer lost
  when the episode's process is ended. This holds without --json too, where it was lost in some
  runs.
• On Burla, a rollout that writes to stdout through the file descriptor (a flushed print,
  os.write, a child process, or any print under --episode-timeout) no longer fails the
  sweep with an UnpicklingError, with or without --json. What a rollout writes stays with the
  worker.
• The dashboard no longer cuts a chart off at its panel's edge (#31). On the comparison page the
  heatmaps' last column was cut at 1,280, 1,440 and 1,920 wide, and on a phone the differences
  and limits charts and the charts on a condition's page were cut too. A chart now leaves room
  for its own padding inside its panel; a chart in a panel wider than the chart is drawn as
  before.

Poisson Prove 0.2.0

Changed

• diff reads in plain words by default: the verdict block, then one table of the conditions in
  the order of their axis values, each with its change and the smallest rise and fall it could
  have detected, and what the comparison left out, in words. The machine sections (the per-event
  tables by hash, the coverage table, the omnibus, the fault lines and the bound) moved behind
  --full, which prints what diff printed before. A tool that read them from the text passes
  --full or reads --json, which did not change.
• Saved comparison ids change at this minor: an id hashes the tool's major and minor version, so
  the same comparison saved again under 0.2.0 gets a new id. Comparisons saved by 0.1.x stay
  readable.
• inspect prints identical errors as one traceback with a count of the episodes that have it,
  each listed episode on a line of its own under it; an error none of the listed episodes has is
  counted in one line. --json is unchanged.
• The dashboard's comparison page shows its heatmaps on the first screen, between the verdict's
  detail lines and what the verdict rests on; the differences and limits charts stay after the
  conditions.
• The limits line of diff, check, report, the dashboard and the terminal app counts the
  rows with no limit by direction, so a row with no limit on its fall is not said to have none on
  its rise; "no change of any size" is said only of a row with no limit either way.
• The look file gains the banner section the Poisson Labs tools share, and its static UI's type
  rule lets labels (table headers, detail labels, captions) take the system's sans.

Poisson Prove 0.1.3

Fixed

• When every rollout of a sweep errored, the sweep's next: line suggests inspect with
  --limit 3, so the first look shows three of the failing episodes rather than every one.
• diff, report and check of a run in which every episode errored name that run in plain
  words and suggest inspect of its commonest error with --limit 3, one line per such run, in
  place of an error about the event set. report writes nothing for it.
• report ends with a next: line, poisson-prove ui, on the store that was typed, so the chain
  of next: lines has no dead end; --json carries the same command.
• runs, the dashboard and the terminal app show a complete run in which every episode errored as
  stopped, as the sweep did. Its manifest and runs --json keep complete, with rows and
  no_outcome carrying the fact; a partial run stays partial. The app's comparison screen takes
  complete runs, so it no longer offers such a run.

Changed

• Before a release, and on every pull request, CI runs Poisson Tasks' own Prove check against
  Prove built from the tree: the check from Tasks' latest release, unedited, on a wheel built under
  a version the package index cannot hold, so the check cannot quietly test the release already
  out. A red check stops the release.

Poisson Prove 0.1.2

Fixed

• prove ui prints its address as soon as the viewer is listening. It no longer waits on a reverse
  lookup of its own host name, which took 35 seconds on some machines, among them GitHub's hosted
  macOS runners.

Poisson Prove 0.1.1

Prove's Burla backend now runs on Burla 1.8.4, the first Burla client with private mode, in which
a cluster in your own cloud account never contacts Burla's servers.

Changed

• The [burla] extra pins exactly Burla 1.8.4, in place of 1.8.2. The backend is still
  experimental.
• The plan's Traffic line says that code, inputs and results go through Burla's relay unless the
  cluster's head runs in private mode (BURLA_PRIVATE=True), and directly between your machine
  and the nodes when it does. Burla 1.8.4 no longer has the DISABLE_BURLA_RELAY switch the line
  used to name.
• When no node is running, the error also says how to start a cluster in private mode: through
  the coordinator's localhost API, POST /v1/settings, then POST /v1/management/cluster/start.
• Prove neither sets nor checks private mode. Its Burla client is private when the environment
  Prove runs in sets BURLA_PRIVATE=True.

Poisson Prove 0.1.0

The first public release of Poisson Prove.

Prove tells you whether a change to an AI agent really changed how often it fails. It replays the same cases before and after the change, on the same seeds, compares the two runs condition by condition, and gives a verdict: safer, worse, trade or no significant change, with stated error bounds and detection limits. When no change counted and at least one condition could not have shown a change of the size you accept, the verdict is too small to tell. Before a sweep runs, its plan refuses a design that small and names the number of replicates that would do.

Install and first run

Python 3.11 to 3.14, on macOS and Linux. The example needs no key and no account.

python3 -m venv .venv && source .venv/bin/activate
pip install poisson-prove
poisson-prove init --example
poisson-prove sweep --yes
poisson-prove sweep --yes --target poisson_prove.examples.mock_agent:rollout_patched --pair-with latest
poisson-prove diff

The example sweeps a mock agent over three tasks and three tool failure rates, 194 episodes in each of the nine conditions, then sweeps a patched version of it on the same seeds. The comparison reads trade: the patched agent takes a forbidden fallback less often, and in one condition where tools fail most it runs out of tool calls more often. The mock agent is there to learn the commands on; its counts measure nothing and move from one pair of sweeps to the next.

What it looks like

The terminal app as it opens, from a fresh install of 0.1.0. poisson-prove with no verb, on a terminal, shows Prove's banner for a second, then your runs and comparisons.

What's in it

• The poisson-prove command with ten verbs: init, sweep, diff, check, report, paired, serve, inspect, runs and ui. check is a gate for CI: passed exits 0, failed exits 4.
• A terminal app: poisson-prove with no verb, on a terminal, opens your runs, your comparisons and their verdicts, a sweep of the folder's prove.toml, and every condition and episode behind a verdict.
• A dashboard: poisson-prove ui builds static pages over the store, opened from disk, with nothing served and nothing loaded from the network. In 0.1.0 the pages do not update while you look at them; the commands that write to the store rebuild them.
• Setup by your own coding agent: poisson-prove init --agent writes a skill, in the folders where Claude Code, Codex and Cursor look for skills, and a one-line prompt. The agent reads your code, writes prove_rollout.py and prove.toml, checks them with poisson-prove sweep --dry-run, which runs no episode, and stops for you to review. Prove makes no model call and reads no key.
• The statistics: a test per condition and outcome. The flags are held to a false discovery rate (Benjamini-Hochberg), and the verdict counts only the changes that also hold at the family-wise level (Holm). Every condition that did not move says how large a change it would have caught 4 times in 5, computed for the test the comparison used, or that it could not have caught one at all. Runs paired at sweep time are compared with the paired test (McNemar) by default, and every output names the method it used.
• Backends: your machine by default; pip install "poisson-prove[modal]" for parallel Modal containers on your own account, with a compute estimate in the plan; pip install "poisson-prove[burla]" for your own Burla cluster in your own AWS, Google Cloud or Azure account, pinned to Burla 1.8.2 and experimental.
• Fault injection through a [chaos] table, capture of an agent's OpenTelemetry spans with [capture] (pip install "poisson-prove[agent]"), and a Python API that gives the same comparisons the command line prints.

What leaves your machine

Prove never phones home: no telemetry, no analytics, no update check. serve binds to 127.0.0.1 only. With a remote backend, your rollout's code and conditions go to the account you choose, and the episodes come back to the local store.

What it does not claim

Prove compares two runs; it does not claim that the change caused anything. It says nothing about conditions you did not sweep, and it cannot check that an outcome label means what you meant by it. no significant change means no change counted and every condition could have caught one of the size you accepted; a smaller change can still be there.

Next

A live local dashboard over the same store, then setting up and running sweeps from either the terminal app or the dashboard.

Links

• Docs: https://poissonlabs.ai/docs/prove/
• The Prove page, with the launch video: https://poissonlabs.ai/tools/prove/
• PyPI: https://pypi.org/project/poisson-prove/
• Changelog: https://github.com/poisson-labs/prove/blob/main/CHANGELOG.md

Stores written by Prove's private builds are read unchanged. MIT licence.