TestPion 0.49.0

Explain and fix failures with AI

• Explain and fix failures with AI… on every failed run — the run's ⋯ menu (test, collection, flow and monitor runs), a run's right-click menu in Runs, and the failed run in the Problems panel and Home's Needs attention. TestPion first groups the failures without AI (the same failed check, status and endpoint), so 500 failures from one expired token are a few groups; the dialog shows the failures and groups, the model and the token estimate before anything is sent. Your model sees each group's examples with the evidence — the request, the response, the failed check as the test writes it, and what changed since the same test last passed — and answers with causes (one cause may explain many groups), each with its evidence, a confidence and a fix. (Explain and fix failures)
• Every fix is proven before you see it: a changed check must pass on the failing response, must not pass on any response (an empty body or an error page), and is run on the last passing response too (it follows an API change when it would have failed there); an edited test file must lint and an edited script pass in the sandbox; a variable fix must be one the failing steps missed and never a secret. When the test is right and the API regressed, the fix is a bug report to copy, with what changed since the last pass.
• A check fix cannot miss its check: the model names the failing check by its result and its place in the test (checkIndex, as the evidence numbers them) instead of quoting text; a fix that still cannot be made goes back to the model with the reason and the checks it may name (up to twice). For a stale expected value TestPion also works out the update itself from the evidence (the value the API answers now, or a stable form such as type $.id = "string" for a value the server makes up; never for a 5xx, 401 or 403) — it stands in when the model's fix cannot be used and is offered beside a different one under Safer:, so a stale test is never left without a fix (task triage-run v2).
• You decide: nothing changes before Apply selected fixes; then Rerun the affected tests and Undo.
• Privacy: failure bodies are masked like every AI context, and Metadata only sends the method and URL (query values as …), statuses, check types and names, the checks as the test writes them, header names, the shape of bodies (field names and types, a text as string (N chars)) and which headers changed since the last pass; the body's changes since then go as shapes only (how many; each change's path and kind as string (N chars), not which fields) — no body values, no failure messages. Names only sends the URL's path, and expected values and most numbers as their type. Without a model you get the grouping with the likely cause of each group.
• CI: testpion test|run|run-collection … --triage adds a Triage section after a failed run, writes triage.md / triage.json beside the results and — when a model answered — a Triage (AI) section in the JUnit, HTML and Markdown reports. It never changes the exit code. testpion ai triage [--run <id|last>] [--privacy metadata] [--hint …] [--no-ai] [--dry-run] [--apply [--items …]] [--json]; MCP ai_triage_run. (CLI, MCP, in a pipeline)

Why is this flaky?

• Why is this flaky? on a flaky test (the Problems panel, a result's History tab): TestPion reads its results across runs — pass rate, flips, retries, the environments and statuses of its failures, its time when it passes and when it fails — and your model names the pattern (timing, ordering, shared state, a time-dependent value, an external dependency, a rate limit, a data collision) with evidence and a fix that makes it stable without hiding a failure: a wait (until:), a stable check, isolated data; a retry only as a last resort, always flagged. Skipping or quarantining the test is never offered. (Why is this flaky?)
• The pattern starts from a signature TestPion computes: how the failing responses differ from passing ones ($.status: "done" when it passes, "running" when it fails) and the pattern it points to (an intermediate state → timing and a wait; 429 → rate limit; 409 → data collision; changed dates → time-dependent; 5xx, timeouts or slow failures → external dependency). The model gets a guide to each pattern's signature and the whole test-file format, and a fix that breaks the YAML or cannot be made goes back with the lint explanation or the check to name (task analyze-flaky v2).
• Attempt to fix applies the fix and runs the test 5 times through the ordinary runner; the review shows the verdict (Stable: passed 5/5, Still flaky, Failing), then Keep the fix or Undo. testpion ai run analyze-flaky --item test:<path>[:<step>], testpion ai attempt-fix <id> [--times 5] [--allow-production]; MCP ai_analyze_flaky, ai_attempt_fix.

Better generated tests

• Generate tests now tells the model the exact tp.* script API, read from the script sandbox itself (what tp.response, its assertions and a tp.expect chain really have, with one test that runs), and a script that misuses it goes back naming the right member — tp.response.json().id for a field read from tp.response.body (the raw text), tp.response.json() for pm.response.json(), to.be.ok for Jest's toBeTruthy(), status() for an invented statusCode() — so fewer repairs are needed (task generate-pm-tests v4).

Behaviour changes: what works differently

• testpion run-collection on a workspace collection records what it ran (the collection, the requests and folders picked, the iterations), so Run again, Run in CI, ci --run last and the triage find its saved requests.
• Attempt to fix and the triage obey the production guard: in an environment marked as production, Attempt to fix asks first like any run, testpion ai attempt-fix needs --allow-production for data-changing requests, and the MCP tool ai_attempt_fix refuses such an environment unless the server was started with --allow-production.
• A fix is never applied over a newer file: when the test file or saved request changed after the proposal was made, applying it is refused with it changed since the proposal was made (run the analysis again).
• Damaged run files are named as such: a run whose summary.json cannot be read says so (with what to do) instead of showing a JSON parser error, and the triage, the flaky analysis, the history and the reports read a damaged or edited results file without failing.
• Rerunning failed requests brings the requests they need: run-collection --rerun-failed, Rerun failed and Rerun the affected tests also run, first, the earlier requests whose scripts set a {{variable}} the failed ones use (the token request), marked [setup]; their results are reported but do not count toward the rerun's totals.
• Test-file edits change only what they change: a fix (triage, flaky analysis) or a flow designer edit rewrites only the edited step's text; flow-style lists, quoting, comments and spacing elsewhere stay byte for byte (CRLF files too).
• --triage outside a workspace says so: triage needs a workspace (-w); nothing recorded (stderr and the JSON envelope's warnings); testpion ai run analyze-flaky without a model (or with --no-ai) prints TestPion's own finding and exits 0.
• testpion ai run shows a JSON task's answer as its proposal (findings and items) instead of the raw JSON.

Also

• The proposal review shows what TestPion changed in a model's answer (notes), long verification labels stay on their row, and a text item (a bug report) has Copy.
• Each AI call from the window now releases its engine context when it ends, so a long session with the assistant no longer holds on to them.

Downloads

See the download page for which file to choose and the installation guides.

The installers aren't code-signed yet: Windows SmartScreen and macOS Gatekeeper ask you to confirm the first launch.