TestPion 0.54.0

0.54 is the everything release. Everything you make in TestPion can now be saved, exported, imported and put under git: a bundle holds any kind of item (collections, environments, API definitions, datasets, MCP servers, prompts, monitors, tests and more), imports come from every other API tool with a preview, a report and Undo, and git shows, restores, stages and merges each item rather than a whole file. AI now checks its own work (tests from a spec, tidying an import, security tests, contract drift, search in your own words), TestPion tests RAG systems with 138 metrics and a cost shown before every run, and the app and the command line start and run faster, with safer saves, imports and downloads.

Save, export, import

• Workspace export is a bundle now. The workspace switcher's Export…, the Export dialog's The whole workspace and testpion workspace export write <name>.testpion.zip (a small workspace of text only: .testpion.json), the same bundle testpion export --all writes: it opens as a new workspace with Import in the workspace switcher, or merges into an open one with a preview, conflict choices and Undo. The workspace's own variables (the Globals) are not part of a bundle; the export says so. The one-file export of 0.53 and earlier is still read, and still written by File ▸ Export ▸ Workspace (older format)… and testpion workspace export --legacy, for two more releases. --format zip|json chooses the file form.
• Cancelling a save dialog no longer downloads the file anyway. Exporting a collection, a workspace or an environment goes through the one save path: written where you choose, downloaded in a browser, nothing at all when you cancel.
• Thunder Client, Yaak, HTTPie and Apidog import. Collections and environments of Thunder Client (GUI tests become assertions or test code, tc.* scripts and {{#guid}}-style generators are converted), Yaak (the workspace export or a git-sync folder; ${[ var ]} templates, gRPC and WebSocket requests), HTTPie for Desktop and Apidog (endpoints become requests under {{baseUrl}}, bodies from examples or schemas, responses as examples). Each is one module of the import pipeline with fixtures written from the tools' public documentation.
• Postman v1, v3 and Native Git. The first export format (requests + order), the file-based v3 format of Postman's Git integration (a folder of *.request.yaml files and .resources/definition.yaml, read from a folder, a ZIP or a link) and the Native Git layout (collections, environments and globals; the cloud id in .postman/resources.yaml identifies the collection for updates). Postman globals become the workspace's globals (secrets to the secret store); without a globals store (the CLI) they come over as an environment.
• Importing again updates. An import of a source imported before shows what it would do to the collection it made, item by item: replace what the source changed, new what it added, skip what is unchanged or what you edited here too (a conflict stays yours unless you choose Replace). The match is the tool's own id of the request or folder where there is one, else its path. The collection keeps its ids, an update is one undo step, an environment of the same name gets only its missing variables. testpion import updates by default (--new-copy adds a copy, --update-decisions file overrides single items), MCP import_definition takes update and decisions.
• Simple tests become assertions (a status, a response time, a header, a body text, a field of the JSON; about fifteen exact shapes, in pm.* or tp.*): the report counts them, Lift N simple tests into assertions (--lift-tests, MCP lift_tests) moves them into the request's Tests tab and keeps the rest of the script.
• Script conversion for more tools. Hoppscotch pw.* / hopp.* (Jest-style matchers become chai) and Thunder tc.* are converted to tp.*; Bruno's req, res and res('path') now become tp.request / tp.response too, bru.sendRequest is an adapter over tp.sendRequest, bru.cookies.jar() is tp.cookies.jar() (with Bruno's method names), req.setTimeout / setMaxRedirects with a number move to the request's settings, bru.getProcessEnv names a value to enter, and pm.state is tp.state (kept as globals named state.<key>). pm.datasets and pm.mock have no equivalent and say what to use instead. testpion scripts convert --from bruno|insomnia|thunder|hoppscotch|postman (--file for one script, --report) and MCP convert_scripts / convert_script_code.
• More comes over from Insomnia and Bruno: Insomnia unit tests (each becomes a tp.test of the request it sends), the cookie jar (to the workspace's jar), Hoppscotch environments with secret flags, Bruno response examples, Bruno gRPC and WebSocket requests and OpenCollection's (as the collection's gRPC calls and connections).
• Export to Insomnia and OpenCollection. testpion export --format insomnia (v5 YAML), insomnia-v4, opencollection (Bruno's YAML folder) and opencollection-file, with the workspace's environments, scripts written in the target tool's API, assertions written as tests, and not exported: … for what the format cannot hold. Round trips are property-tested over generated collections; MCP export_collection. A name like CON or aux.bru can no longer make an export write a Windows device file.
• Links to repositories. A public GitHub or GitLab repository or folder link imports as the folder's files (Bruno, OpenCollection, Postman Git, Yaak sync folders …): https only, only those two services (a redirect elsewhere is refused), no credential sent or stored, at most 500 files / 5 MB each / 25 MB, nothing executed.
• Migrate from… for every tool, and generated guides. Home's Migrate from… offers Postman, Insomnia, Bruno, Hoppscotch, Thunder Client, Yaak, HTTPie and Apidog with how to export. docs/migrate/* (one page per tool, an overview with the capability matrix) is generated from the importers' support table and the converter's tables (npm run docs:migrate); a test fails when a page is out of date, an importer has no page or a detected format is in no table.
• GraphQL schemas import with a folder per operation type and a request for every root field (up to 60 per type), also from an introspection result (JSON).
• The folder editor has a Settings tab (timeout, retries, redirects, TLS, URL encoding, cookies: the defaults of the requests inside) and a description; the collection editor a Settings tab. One component for request, folder and collection settings.
• Tests and speed. A corpus of 80 collections (eight API shapes in ten export formats) and 265 script snippets (five script APIs) with golden output, idempotence and behaviour checks; hostile inputs (zip bombs, path traversal, __proto__ keys, YAML alias bombs, 3000-level nesting, scripts that would do something if run, byte order marks); a 10,000-request collection previews in under 1.5 s and imports in under 3 s. Big YAML files (an Insomnia v5 export of thousands of requests) are read with js-yaml, several times faster; a file nested thousands of levels deep is a message, not a crash.
• The registry's import capability is true for requests (copied commands and request files); the kinds that cannot import yet are planned for 0.58.
• Preview before anything is written. The Import dialog has three steps: where it comes from (several files, a folder, a ZIP, a link, pasted text, or something dropped on the window), a preview of everything it will create (the tree of collections, folders and requests, environments, API definitions, flows, clashes with what the workspace has) and the compatibility report, then the result with the same report and Undo import. Undo moves what the import made to Recently deleted and removes its flow files; it refuses while an imported collection was edited. Home has Migrate from… with how to export from Postman, Insomnia and Bruno.
• A compatibility report for every import: per item imported fully, with changes, or not imported and why, grouped by code (auth, template tags, script APIs, settings, values to enter …), downloadable as JSON or Markdown, kept for the last 30 imports (testpion import-report, MCP get_import_report). The notes list stays in every JSON result.
• One intermediate model. Every importer (Postman, Insomnia, Bruno, OpenCollection, Hoppscotch, OpenAPI, HAR, cURL, WSDL, AsyncAPI, GraphQL SDL, .http, .env, Arazzo) parses to one import document; one pipeline converts scripts, makes ids unique and builds the report. importers.ts is split into one module per format.
• Scripts: an AST converter. pm.*, the old postman.* / tests[] / responseBody, insomnia.* and bru.* become tp.* by the syntax tree (scope-aware: your own pm variable is left alone; strings, comments and formatting survive; idempotent). What has no equivalent (pm.datasets, bru.sendRequest …) gets a // TODO(import) line and a finding; a script that does not parse is kept and reported with its line. testpion scripts convert and the app's Convert scripts use it too. New: tp.execution.runRequest("name") runs another saved request of the collection (Bruno's bru.runRequest).
• New auth types: NTLM, Hawk and Akamai EdgeGrid are signed by the engine (NTLM v2's three-message handshake on one connection) and are imported from Postman, Insomnia and Bruno instead of becoming "No auth" with a placeholder. Tested against RFC 1320, [MS-NLMP] 4.2.4, the Hawk README and the Akamai client test data.
• Folder and collection headers and settings are real fields (a Headers tab on folders and collections; folder description; settings inherited by the requests inside). Insomnia and Bruno folder headers stay on the folder instead of being copied into every request; Postman's protocolProfileBehavior (redirects, cookies, URL encoding, TLS versions …) is imported at request, folder and collection level and exported back.
• Bruno's OpenCollection YAML (the default format since Bruno 3.1) is read as a folder or one file; Bruno settings blocks, custom http methods, SPARQL and file bodies, collection-level post-response variables; Insomnia design documents (kept in specs/), folder scripts, v5 request settings and v5 gRPC / WebSocket requests; Postman Vault references, switched-off bodies and several files in one form field are reported.
• ZIP and folders: a Postman data export (every collection and environment) imports in one go; Postman environment secrets go to the secret store. The reader refuses zip bombs, entries that leave the archive, encrypted and multi-part archives. testpion import takes several inputs, previews them first (--apply), and writes the report (--report); import-undo takes it back.
• Postman by API key: postman:collection/<id> (or environment, workspace) with the user's own key, kept in the secret store.
• About fifty Postman dynamic variables that were missing ($randomBs, $randomIban-style finance values, images, database, grammar, file, job and direction names) now render.
• MCP: import_preview, get_import_report, undo_definition_import; import_definition returns report and importId.
• Imports of OpenAPI examples honour the documented limits (a minimum, maxLength …), so the generated example body satisfies its own schema.
• The rest of the outputs have the down arrow: History (HAR, CSV, JSON), Traces (OTLP JSON), a Debugger exchange (raw text, HAR, each body as the file it is; the grid's "Request body…" and "Response body…" now use the Save as dialog), MCP resources, prompts and the protocol trace, the AI Playground answer, a model comparison, an analysis's findings and a proposal under review, Git changes (patch or Markdown), the API reports (lint, fuzzing, coverage, the security review, the comparison of two versions), every chart (PNG, SVG when the plot is a drawing), the environment matrix and the comparison of two environments, a monitor's runs, the runs list, the workspace report on Home, collection documentation and Settings ▸ About's diagnostics. Redaction is the registry's: outputs masked when they are recorded never offer Include secrets….
• testpion output <output> --format <format> --out <file> saves any output that is built from the workspace (--list shows them), and --out / --format work on send, history list, monitor results, git diff and eval matrix. The file is byte for byte what the app saves (one set of builders in the engine, used by the window, the CLI and MCP).
• MCP save_output: the same outputs for an AI agent, returned inline or written to a new file inside the workspace (never over an existing file or a definition). Secrets stay masked.
• Checks: the output registry's cli field is a command that exists (tested), and the output save no longer calls other handlers by name.
• A bundle for every kind: Export… and Import… are on every list header and every item's menu. A bundle (.testpion.zip, or .testpion.json when small and all text) holds collections, environments, API definitions, datasets, MCP servers and mocks, protos, script packages, test and flow files, saved gRPC calls and connections, prompts, load tests, evaluations, monitors and (only when asked for) AI providers. It is a selection, not a nesting: nothing moves on disk. Export and import
• Export with everything it uses… (Export ▸ on a collection): adds the environments that define its variables, the API definitions, datasets, protos and mocks its text names, its MCP servers, script packages, saved calls, monitors, load tests and the test and flow files that refer to it; what is named but not in the workspace is listed as missing.
• Preview before import: every item is New, Already here (importing a bundle twice changes nothing), Conflict or Refused, with Keep both (the default; the new one is renamed and references to it follow), Replace or Skip per item and for all. Scripts and programs an MCP server starts are listed under Review before you run; nothing runs on import and an imported stdio server stays untrusted.
• Undo import: one step takes back everything an import made and restores what it replaced, byte for byte; a file edited since is left alone and named. Datasets over 10 MB go by reference (or all of them with Always include the data).
• No secret value in a bundle: the existing detector replaces every typed-in secret (headers, auth, MCP headers and env, scripts, dataset columns, connection strings) with a {{variable}} and lists each one to fill in.
• A bundle is untrusted input: the reader refuses paths that leave their folder, links, names that differ only in case, encrypted and zip64 archives, too many or too large entries, zip bombs and a wrong checksum; __proto__, constructor and prototype keys are dropped; a higher format version asks you to update.
• Command line: testpion export <kind:id…> | --section <kind> | --all [--uses] --out <file>, testpion import <bundle> [--preview] [--on-conflict keep-both|replace|skip] [--only] [--decisions] --json, testpion import --undo <id> / --list, testpion bundle inspect <file>.
• MCP: export_bundle, describe_bundle, import_bundle_preview (read-only, returns a plan_id), import_bundle_apply (the only write, needs the plan and consent) and undo_import.
• App: File ▸ Export ▸ Bundle…, File ▸ Import Bundle…, the command palette, and a .testpion.zip or .testpion.json dropped on the window. Workspace exports of earlier versions are read as bundles too (merge into the open workspace). Export Workspace stays for two more releases.
• Export a folder or a request. Export… on a folder or a request of a collection takes that part as a collection of its own, with the folders above it and their headers, auth, variables and scripts (Clinic / Pets); testpion export folder:clinic/Pets, request:clinic/"List pets", MCP export_bundle items. The registry's export is true for the folder and the request kinds.
• Export a folder of a list. Export… on a folder of API definitions, datasets, MCP servers, saved items or test files takes the items in it (testpion export --folder spec:Public, MCP folder). An API definition or a dataset keeps its folder through a bundle (the item's folder).
• Import here… on a collection and on a folder adds the bundle's requests under it (new ids where an id is taken, the collection variables the requests use added when you have none of that name, your own settings untouched), with one Undo: testpion import file --into clinic/folder-id, MCP import_bundle_apply into. The registry's import is true for the folder kind.
• The workspace variables (Globals) travel in a bundle (the manifest's workspace section, format version 1; a whole-workspace export carries them, a selection with Include the workspace variables, --globals or MCP globals). Secret values are blank. The preview counts them (new, the same, different); the import adds the new keys, takes the bundle's values only when asked, and never blanks a value you have; Undo puts them back unless you edited them since. A workspace opened from a bundle gets them too.
• The Export dialog shows why each item is in (what uses it) and lets you untick the ones that are only there because something uses them; they are listed as left out (--exclude kind:id, MCP exclude).
• A collection for Postman or Bruno with what it uses. The Export dialog's Postman package with what it uses (.zip) and Bruno collection with what it uses (.zip) add the environments it uses, the workspace variables as Postman globals, API definitions, data files and an .mcp.json; what those tools cannot hold is listed in the zip's README. testpion export clinic --format postman --extras, MCP export_bundle package. Secret values are never in them.
• Import compatibility. Yaak: an auth type or a template function from a Yaak plugin is named as a plugin's (TestPion does not run plugins), Yaak's own functions with no TestPion variable are told apart, random.range(a, b) and Akamai EdgeGrid are read. HTTPie and Apidog: an encrypted password or variable is left empty and named, never copied in as a credential. Postman v3: $ref to another file of the folder is followed (a #/pointer too); one to a file outside the folder, a missing one or a loop is named and left out. The update preview shows the script lines that differ, per item.
• One zip reader. The import's reader for a Postman data export or a zipped folder is now a policy over the hardened reader that bundles use (junk files left out, backslashes as separators, links skipped, its own limits); hostile-archive checks (checksums too) are the same everywhere.

Git

• A commit no longer fails while the app reads Git in the background. Reads (status, diff …) no longer take .git/index.lock, so a commit or a stage that came at that moment could fail with "Unable to create index.lock".
• The Git view shows the file again right after Undo even when the file watcher reports the write a moment later (it used to put the refresh off by two seconds, again for every report).
• Git writes you can take back. Resolving a conflict (in the dialog and with Keep mine / Keep theirs beside it, now confirmed) and discarding a file or Discard all (which now lists the files) have Undo; a resolve is put back as a conflict. Pressing Resolve with these choices before choosing anything asks first. One verb, Keep theirs, everywhere.
• Fix git setup. One guided, previewed, undoable action (the Git view's strip and ⋯ menu, the workspace switcher's ⋯ menu, the command palette; testpion git fix [--dry-run] [--hook] [--untrack] --json; MCP git_fix_setup). It completes .gitignore and .gitattributes (one line-ending policy, the merge driver and the new diff driver on every JSON kind; a pattern you already have gets the missing attributes on its own line, an attribute you set stays yours), registers the drivers in the repository's config, and rewrites existing files once into the stable format (the dialog lists them and says which only change line endings; nothing is rewritten when a workspace is opened). Optionally the pre-commit secret check and git rm --cached of tracked files the ignore rules ignore. It notes big files and CRLF documents. Idempotent; works without git or outside a repository (the file steps still run) and in a folder of a bigger repository (nothing outside the workspace is written). Every file is copied aside first: Undo. Make ready for git stays as the quick version.
• A readable git diff (the diff driver): plain git diff, git log -p and git show print collections, environments, library files, MCP servers and providers item by item, with the item named in the hunk header; installed by Fix git setup, and two git config lines for people who work without the app (documented). The app's own line diff stays plain. The app and the CLI are the driver (TestPion --textconv, testpion git textconv).
• Big files. A file over 5 MB is noted, over 50 MB offered Git LFS (a rule in .gitattributes; says plainly when git-lfs is not installed) or by reference (datasets/x.csv.ref with the size and sha256, the data git-ignored and removed from git); over 100 MB a commit is refused (GitHub rejects it) until one of the two. Each shows what it will write and has Undo. testpion git large [--check|--track f --lfs|--reference], MCP git_large_files / git_track_large. --check says per reference whether this computer has the right data.
• Share session. A Debugger capture session stays git-ignored; the Debugger's Session menu ▸ Share "session name" with the team… (testpion git share-session, MCP git_share_session) puts a copy, masked once more with the redaction every export uses, in debugger/shared/ where git tracks it. The watcher no longer reports the constantly rewritten debugger/*.har files.
• CODEOWNERS. ⋯ ▸ Create CODEOWNERS… (testpion git codeowners, MCP git_codeowners) drafts one line per kind folder of the workspace (the owners are comments to fill in, or one owner for every line); written only when the workspace is the repository, never over an existing file.
• Stage one item. Staging an item of a file with several items (a collection, MCP servers, providers, a library) now puts only that item's change in the index (built with the same serializer the app saves with, verified by putting the old item back; a file whose staged copy is not in the stable form, a new file, or a file that is one item stages whole and says so). Each item row in the Git view has its own Stage / Unstage and shows staged; the file shows partly staged. Commit takes just the staged items.
• The secret scan's allowlist. Allow… on a finding writes .testpion/secrets-allow.json (committed, read in review): the file, item, field and a fingerprint of the value (never the value) with the reason you must give; the same text elsewhere, or a changed value, is found again. testpion git allow-secret, disallow-secret, allowed-secrets. Deliberately not an MCP tool: an agent cannot switch the guard off.
• One-click Fix for every kind. Fix all in the staged-secrets panel now also fixes MCP servers (headers, environment), AI providers (key and headers), saved items (gRPC metadata, WebSocket headers, prompts, monitors), test and flow files and any other text file: the value becomes a {{variable}} (the file in the stable form) and a secret variable of the active environment, in the OS secret store; Undo restores the files. testpion git fix-secrets and MCP git_secrets_fix_plan show the plan.
• Kind filter chips in the Git view (MCP server 3, Dataset 1 …), one shared FilterChips control. MCP, API definition and dataset folders (library/mcp-folders.json, spec-folders.json, dataset-folders.json) are changes of their kind's list, so the folder rows and the section count show them.
• Kinds now versioned: the workspace settings, .proto files, script packages, MCP mocks and the Debugger rules (history, discard, stage, conflicts, secret scan; the RAG targets already were). Capture sessions say why they are not (kept on this computer; Share session). A full-registry gap-matrix test checks every versioned kind: stable form, row, history, Discard, Stage, conflict screen.
• Fixes. testpion git history <plain word> is an error that says what to name instead of an empty list. The watcher reports datasets as datasets (they were reported as test files) and ignores debugger/*.har; the dataset view and the Collection Runner follow changes made outside.
• Performance budget (a test): at 9,000 items, reading the changes by item takes under 1 s and making the marks under 100 ms.
• Stable files for every kind. Collections, environments, workspace settings, library files (saved gRPC calls, connections, prompts, load tests, evaluations, monitors), MCP servers, AI providers and the HTTP Debugger's rules are written with a fixed key order (the same on every computer), LF line endings, one final newline and no volatile field (a library item no longer carries updatedAt; workspace.json keeps its own time stamp when only its variables change). Saving twice, or from another key order, leaves the same bytes. A test file keeps your text and only gets LF and a final newline. Older files are read as they are and written in the stable form the next time their item is saved; nothing is rewritten when a workspace opens.
• A change mark on every row. Requests, folders, collections, saved gRPC calls and connections, environments, test files and flows, MCP servers, AI providers, prompts, load tests, evaluations, monitors, API definitions and datasets show M / A / D / R / ! on their own row when git sees a change; a folder and a section header (MCP servers, Datasets …) show how many changes are inside. One mechanism: a row says what it is (git on the shared tree row) and the mark follows.
• The Git view lists changes by item. MCP server Weather: URL, AI provider OpenAI: model, Prompt Greeting: moved, settings, API definition petstore.json: 2 operations added, 1 breaking change, Dataset users.csv: 12 rows added, column dob removed, Workspace settings: variables (added region), with the kind's icon, Compare for every kind (secret values are hidden), and the file as muted text (Stage and Discard still act on the file). The down arrow of the list saves the changes as Markdown by item or as a patch.
• testpion git status --by-item [--kind <kind…>] --json and the MCP tool git_changes return the same rows (kind, change, details, ref, link, file, staged).
• Merging: the lists of MCP servers and AI providers merge server by server (mcp-servers.json merge=testpion, providers.json merge=testpion); workspace.json.bak-* and *.tmp-* files are ignored by the generated .gitignore. The order of keys no longer makes a difference between two versions of a file.
• Unreleased registry: versioned is now true for saved gRPC calls, connections, prompts, load tests, evaluations, monitors, MCP servers and AI providers.
• History in git… for every versioned item. Requests, folders, collections, environments, test files and flows, API definitions, datasets, MCP servers, AI providers, saved gRPC calls and connections, prompts, load tests, evaluations and monitors: the item's menu and its row in the Git view open the commits that changed that item, each with what it changed ("URL", "headers (added X-Team)") and compared with now part by part. Restore this version shows what would change, writes just that item (the rest of its file stays) and offers Undo.
• Who changed what. The same dialog has a Who changed what tab: for each field of the item the author, the commit and the date of its last change, or "not committed yet".
• Discard changes… for one item. Only that item goes back to the last commit; the other changes in the same file (a second MCP server, another request) stay, and an item that was staged is unstaged in the index too. A new item is removed, a new file deleted. The result is written in the stable form, with Undo (also Ctrl+Z) and through .local/git-undo/ for the CLI.
• Stage changes. Staging an item stages its file and says which other changed items come along ("it also holds 2 other changed items"); the Stage button of a file with several changes says it too.
• Set changes aside (stash) with a name, listed by item in the Git view, with Put back and Drop.
• Conflicts of every kind. A file git could not merge opens Resolve…: collections, environments, library files, MCP servers, providers, the workspace settings and the Debugger rules are resolved item by item and, inside an item changed on both sides, field by field (only the clashing fields are conflicts; the others are merged); test and flow files, API definitions and datasets by clashing hunk, each named by the step it is in; a database whole. Each one: Keep mine, Keep theirs or Edit; the result is written in the stable form.
• The commit scans what is staged. The secret guard reads the staged files (the blobs a commit would publish), kind by kind, with the detector the exports use. A certain secret (a secret name with a typed value, a key-shaped value) refuses the commit with a message naming the item and the {{variable}} to use, and there is no "commit anyway"; a probable one is listed and can be committed after a confirmation. Fix moves a collection, environment or workspace value to a secret variable and stages the fix.
• Command line and AI agents. testpion git history | blame | restore | discard-item | stage-item | unstage-item | undo | stash push|list|show|apply|pop|drop, git conflicts [--resolve ours|theirs], git resolve --pick/--edit, git check --staged (all with --json; --dry-run where they write). MCP: git_history, git_blame, git_restore_item, git_discard_item, git_stage_item, git_undo, git_stash; git_conflicts, git_resolve and git_propose_commit know the new shapes.
• Merging by field: two people changing different fields of the same request, server or prompt no longer conflict; the conflict message names the field (request.url). Registry: git-history, git-discard, git-stage and secret-scan are true for every versioned kind.
• Git. Staging one item now works for a request that is new (placed under its folder in the staged copy, a new folder made with only that request), for a new file and for a deleted file, and unstaging takes the item and the folders only it needed out again. The changes between two commits (testpion git diff, stash show) are read with a handful of git processes like the working folder's. A conflict in a test or flow file is settled by step (one choice per step; a hunk of its own wins; hunks outside the steps stay hunks). git diff prints test and flow files step by step (tests/**/*.yaml diff=testpion from Fix git setup). Fix git setup lists the files to untrack with the .gitignore lines it is about to write, and removes exactly those. The Git view reads status and changes from the one shared read.

AI that checks its work

• The cost before the send. AI items of menus (Suggest checks, the script AI items, What to do about a problem) open the assistant on Ready to send with the estimate and What will be sent…; Ask… shows its estimate under the box and waits for Ask; a judged metric's Run examples says how many judge calls and asks; the estimate of an AI button shows on keyboard focus too. One AiEstimateLine (announced to screen readers) is the estimate line of every AI dialog.
• Fix what the import left (AI). The scripts that still carry // TODO(import) are rewritten with tp.* as a reviewed proposal (task import-leftovers): one item per script, each checked statically and run in the sandbox on the request's saved response, never the real API. In the import report (Fix what the import left…), testpion ai run import-leftovers, MCP ai_run_task.
• Security tests. Generate security tests with AI… (a collection's or folder's AI menu, or Security tests in the Generate with AI dialog; testpion security generate; MCP security_tests_generate) writes OWASP API probes as an ordinary test file: BOLA (a second user repeats the owner's request and must be refused and must not see their data), broken authentication (no credentials, an invalid token), mass assignment (privileged fields such as role added to a body) and function-level authorization (a lower-privilege user calls an admin function). The model only chooses which saved requests to probe and which variables hold the two users' credentials (names, never values); TestPion writes the steps and every verdict is a check computed from the responses, never the model's opinion. Safe by design: GET/HEAD/OPTIONS only unless write or delete probes are asked for, paced, capped, a production environment refused. testpion security run, MCP security_tests_run and the window run it one step at a time and list findings first; an owner setup that fails makes the probe inconclusive, never a pass.
• Contract drift explained. Check traffic against this spec with AI… (an API definition's AI menu and header; testpion openapi-drift; MCP contract_drift) compares History, a HAR file or a run with an OpenAPI document and lists, without a model, what differs: undocumented endpoints, statuses, content types and fields, changed types, missing required fields, nulls the schema forbids, enum values. With a model it says who is right per finding and proposes the fix: a spec patch TestPion makes from what was observed (comments and order kept) or a contract check pinned to the saved request. Every patch is verified against the same traffic (the drift gone, no other drift, the document linting no worse); Undo restores the file.
• Script helper. Script with AI in every script editor (Pre-request and Post-response): Write a script with AI…, Fix this script with AI… (from the error) and Explain this script with AI. The script uses the tp.* API as the sandbox really has it (never pm.*) and is run once in the sandbox before it is offered; the review says what it did (variables set, headers changed, tests run). testpion ai script, MCP write_script.
• Checks from a run's stored response. Generate checks from this response on a result of a run (Tests and collection runs) asks the verified-assertions task for checks written for the response the run stored; accepted checks join the step of the test file or the saved request that produced the result. Also Suggest checks with AI… on a saved request's menu.
• Prefer stable checks (Settings ▸ AI assistant; --prefer-stable, MCP preferStable): suggested checks assert types, formats and presence instead of literal ids, times and tokens.
• The flow review shows what the model changed: the Vs draft tab lists, per step, what the model added, dropped or changed against TestPion's own deterministic draft.
• One table for the right-click "… with AI" items (packages/shared/src/ai-actions.ts): a node with several AI items keeps them in an AI ▸ submenu; testpion kinds and list_kinds show them; the ai capability is true for collections, folders, requests, API definitions, runs and tests.
• Hide all AI actions (Settings ▸ AI assistant): removes every AI button, menu item, command, the assistant and the Ask search from the window; AI Lab, the CLI and the MCP tools are unaffected. A test lists every entry point of the window.
• Tests from a spec, driven by the gaps. Generate tests from this spec… (the API definition's header, its right-click menu, and From a spec in the Generate with AI dialog) first shows what the tests and collections never prove about the document: operations never called, documented statuses never asserted, no negative, boundary or security test, no contract check, parameters never sent. That report needs no model (testpion generate gaps --spec, MCP spec_gaps). The tests that close the gaps are positive, negative, boundary and security cases: TestPion builds them from the document itself (testpion generate tests --spec … --no-ai), a configured model adds more ideas, which are built from the document's valid example too — an operation the document does not have is dropped and reported, an undocumented status becomes its class (4xx), a request that should be accepted must satisfy the schema. The checks come from the document (status, the openapi contract check, a shallow JSON Schema), every test is validated against it, and the proposal says the coverage before → after, computed. Task tests-from-spec; CLI generate gaps|tests; MCP spec_gaps, spec_tests_generate.
• Tidy this import. After importing Postman, curl, HAR, Insomnia or Bruno (a notice offers it), or on any collection (Tidy with AI…), the clean-up is proposed as separate changes: names and folders, a host or a repeated token typed into requests as collection variables, a hard-coded id that an earlier response returns chained with a tp.* script, tracking, static and duplicate requests dropped, optionally a flow. TestPion finds the hosts, tokens, noise, duplicates and the chains a saved example confirms without a model; the model names and groups, and its extra drops and unconfirmed chains start unticked. Each change is verified: the result is a valid collection and every remaining request sends exactly what it sent before (variables resolved). Apply is one step and Undo puts the collection back. Task tidy-import; CLI testpion tidy <collection> --ai; MCP tidy_import. A HAR import now keeps the recorded response as a saved example (small JSON or text bodies), which lets the tidy confirm chains.
• testpion tidy --dry-run reports what --remove-* would remove without changing anything.
• Ask: search in your own words. Type a question — failed payment calls last week over 2 s, flaky tests in staging, traces where a tool timed out — in Quick Open (~, or the last Ask: “…” row), in the top bar's search, or with the sparkle button beside the filters of History, Runs and Traces, and TestPion searches History, runs, test results and traces. The model only translates the question into a validated search query (new AI task nl-search, small tier); it sees the question, the time and the field names, never your data. TestPion checks the query against a fixed schema (known fields and operators, bounded values and limits, no code) and runs it itself, read-only; the dialog says how the question was understood (the query in words, what the model guessed) and shows it as editable filters you change and run again with no model. Without a model, or when it fails, a cap refuses it or its query is invalid, the same box searches the words as typed and says why.

RAG testing

• 138 metrics (was 134). answer-regex, answer-length and answer-json-schema (DeepEval's JsonCorrectness) score the shape of an answer 1 or 0 with no model; judge-cost is what judging the other metrics of the sample cost, in dollars (lower is better, max gates it). negative-rejection takes mode: judge for a refusal the pattern does not know. Ragas' one noise_sensitivity takes mode: relevant | irrelevant. A jury now also votes on the claim-level metrics, claim by claim.
• A dag kind of custom metric (a decision tree in metrics/<id>.yaml) with a Decision tree editor in the metric editor: the questions in the order they are asked, where each answer leads (a question or a score and why), add, remove and rename nodes; every change is written back into the YAML. The tree is checked like the built-in dag metric's, and the catalogue row says the most judge calls a sample can cost.
• Test sets get a voice and a meaning check. Personas (--persona) and styles (--style: formal, casual, terse, verbose, typos, non-native) go round the slots; each record keeps them in its tags. --dedupe embedding (--embedder) also finds repeated questions by meaning, redacted and with the words as the fallback that says so; the estimate counts the embeddings.
• Captured datasets: reference answers and more protocols. Draft reference answers… (testpion rag draft-references, MCP rag_draft_references, the AI task rag-draft-references) drafts a reference for each record that has none from the passages it retrieved, not from the system's answer, checks every draft against them and offers one proposal; accepting replaces the record by id (one Undo). Capture reads gRPC and MCP calls of History (questions only: History keeps no response) and a WebSocket message log (--messages).
• Calibration: a jury's members have their own rows (kappa, accuracy, TPR, TNR against your labels), safety and conversation judges calibrate like any other (a label belongs to the whole transcript, a disagreement shows the last turns), the report downloads as Markdown or JSON (the new output surface rag.calibration; rag calibrate --out), and Make an example of this turns a disagreement into an example of a custom metric (rag metric example, MCP rag_make_metric_example).
• Authoring: the gates of a drafted evaluation are a field of the saved evaluation (a Gates field in the editor; rag run, a monitor of the evaluation and CI use them, a --min for the same metric wins), and --slice-file also writes the slice it runs to datasets/<name>.jsonl.
• Conversations: Conversation is a field of the dataset column mapper (the question becomes the last user message), and a server-session target can keep its session in a cookie (conversation.cookies): the cookies each answer sets go with the next turn of the same conversation only, never shared and never stored.
• The bundle carries custom metrics. Export… and Import… handle metrics/*.yaml (metric is a bundle kind; a script or judge metric is listed in the import's Review). Under the hood: eval/rag/custom/file.ts and metric-matrix.ts use the shared isRecord and percentile.
• Test sets from documents. AI Lab ▸ RAG ▸ Data ▸ Generate a test set from documents…: documents (md, txt, html, https pages) are chunked deterministically, the AI task rag-generate-testset writes single-hop, multi-hop, abstract, unanswerable and adversarial questions with reference answers and chunk ids, and every sample is verified (lexical and judge entailment support, a keyword-retrieval check that unanswerable questions are unanswerable, duplicates) and reviewed in the shared proposal review before it is added to datasets/<name>.jsonl. Estimate, caps and What will be sent come first; document URLs are https only, public addresses only, same-host redirects, 2 MB. testpion rag generate-dataset, MCP rag_generate_testset.
• Datasets from real calls. Create dataset from captured calls… (History, the Debugger, AI Lab ▸ RAG ▸ Data): the question is read from the request where the RAG target writes it, the answer and contexts with its mapping; secrets and personal data are masked before anything is written; dedupe, seeded sampling, preview. testpion rag capture, MCP rag_capture_dataset.
• Judge calibration. Pass/Fail labels on a result's metrics (the matrix drill-down) stored in labels/*.json with provenance; Calibrate judge shows Cohen's kappa, accuracy, TPR/TNR, the confusion matrix, bootstrap intervals, the numbers per judge model, disagreements, and the pass mark that agrees most (applied to the evaluation or the metric file). testpion rag label|calibrate (--min-kappa gate), MCP rag_label, rag_calibrate.
• Draft an evaluation with AI. rag-author-evaluation proposes a complete RAG evaluation (metrics with pass marks and reasons, judge options, gates); metrics that cannot run on the data are dropped with a note, and a dry run on a few samples (free metrics for real, judged prompts counted, nothing sent) is the badge on the proposal. testpion rag author, MCP rag_author_evaluation.
• Trends, nightly runs, CI. testpion rag trend / MCP rag_trend / a Trend view with the shared chart output control (a change of judge or prompt is marked, not read as a regression); monitors get a rotating nightly sample, a per-run budget and a regression gate against a baseline, with the alert carrying the metric table; the CI dialog and ci get a scheduled nightly RAG workflow and a regression section in the PR comment. The metric editor has Save as…; the redactor now masks a phone number that ends a sentence.
• New proposal item kinds dataset-row and evaluation; AI tasks that read a folder or a target (prepareContext) run without a workspace item.
• A SQL metric can only read the databases its saved definition names. datacompy-score and sql-equivalence read a SQLite file only inside the workspace (and fail closed where no workspace folder is known), never a database an agent or a request passes in its options (rag_try_metric and the Try dialog take an evaluation instead and use its database), and a PostgreSQL or MySQL connection that a workspace file names (url, or connectionEnv pointing at any environment variable, even marked readOnly: true) is n/a until you allow that target on this computer: testpion rag trust-db "<target>" (the n/a message prints the exact line). The decision is kept in the app's data folder, not in the workspace, names the host and database (a changed variable asks again), and no agent can make it.
• The SQL checker reads each dialect the way its server does. A MySQL --x, a # comment, /*! … */ and /*M! … */ comments, a backslash outside a string, an unusual space or control character and a quoted function name ("pg_read_file"(…)) are refused instead of stripped; a PostgreSQL line comment ends at a carriage return; a number runs into no keyword. A hostile-input list per dialect (SQLite, PostgreSQL, MySQL) is a test.
• A SQLite query cannot take the computer's memory. x||x in a recursive query, printf('%1000000000c') and group_concat stop at SQLite's own size limits (16 MB a value, 30 compound terms, 32 function arguments), the heap is capped, and a query that makes the process grow by 1 GB is killed.
• The RAG MCP tools that send requests or spend judge tokens ask first. rag_run, rag_rescore, rag_probe, a real rag_try_metric (a dry run that asks no target needs none), rag_validate_metric with run_examples, and rag_import_dataset with save_as need confirm: true (or a yes to the client's question).
• The metrics matrix of a 5,000-sample, 84-metric run opens in 0.1 to 0.2 s instead of 3 s and pages, sorts, filters and pass-mark edits take about 20 ms instead of a full rebuild each: the built matrix is kept per run, the 95% intervals are asked for the columns on screen (once each), only the rows on screen are drawn, and typing a pass mark no longer redraws the table.
• The exact estimate of a big run keeps counts, not prompts (9 MB for 1,200 samples became 0.01 MB; 300 to 800 MB at 5,000 samples × 10 metrics), a judged sample builds its prompt once per model call instead of twice, the judge cache is indexed by time, custom metric files are read once while the metrics/ folder is unchanged (the lists reload only when a metric file changed), and a search over trace files lets the app breathe between the reads.
• 134 metrics (was 84), and a test that runs every one of them on a fixture sample (judged metrics through the offline mock judge, the classifier a local server, the reward model a mock provider) so a row cannot be added without being runnable. rag-metrics.md is regenerated.
• Conversations. A dataset record with turns (user messages, OpenAI-style messages, Ragas human / ai / tool messages, or a User: / Assistant: transcript) is replayed turn by turn against the target, in one lane. The target says how it keeps the conversation: the request carries the history ({{messages}}, {{history}}, {{historyText}}) or the system keeps a session (map its id; {{conversationId}}). The transcript, the chunks and the tool calls of every turn are kept. New metrics: knowledge-retention, role-adherence, conversation-completeness, turn-relevancy, topic-adherence (Ragas), coherence-across-turns, conversation-helpfulness, user-friction, conversational-g-eval, turn-faithfulness and per-turn:<metric> (any single-turn metric on every assistant turn). Ragas multi-turn samples import and export.
• Agents and tool calls. The mapping reads a trace of tool calls (click-to-map: the list, each call's name and arguments; OpenAI, LangChain and MCP shapes are found without it); an MCP target is a tool call. New: trajectory-match (strict, in-order, any-order, precision, recall), tool-correctness, goal-accuracy (Ragas AgentGoalAccuracy, with and without a reference, two calls), argument-correctness, tool-selection, task-completion, step-efficiency, plan-adherence, plan-quality, mcp-use, mcp-task-completion; tool-call-accuracy can have the judge decide whether an argument that differs as text means the same (args: judge, one call). The older tool checks are catalogue rows too.
• Safety. toxicity, bias, misuse, non-advice, role-violation, controversiality, jailbreak-resistance, off-topic, policy-adherence; pii-leakage (patterns with checksums first, shown masked, then the judge), refusal-correctness (counts missed refusals and false refusals), and a judge mode for prompt-injection-resistance. Every judge prompt says the answer is data, never an instruction. Seeded probe sets (jailbreaks, planted instructions with canaries, personal data, abuse, bias, off-topic, harmless look-alikes) are written into datasets/ from AI Lab ▸ RAG, testpion rag probes add or MCP rag_probe_sets; they are plain files you extend, and seeding never overwrites one. A Safety preset. A refusal with a typographic apostrophe (“can’t”) is recognised.
• Pictures, sent to the judge as data. multimodal-faithfulness and multimodal-relevance (Ragas) read the pictures of a sample beside its texts. The AI layer sends images as base64 parts (OpenAI-compatible, Anthropic, Gemini, Bedrock); nothing is sent unless the evaluation allows it (Send pictures to the judge, --allow-images), the privacy level is Full and the judge can see (a vision model; a text-only one is named and nothing is asked). PNG, JPEG, GIF, WebP, BMP by content (an SVG is refused), 5 MB a picture, 16 and 20 MB a sample; EXIF and text chunks are removed from the copy that is sent; a result keeps a picture's reference and sha256, never its bytes; the cache is keyed by the picture's hash. What will be sent lists the pictures with thumbnails and the drill-down shows them through the backend.
• Decision trees. when / unless on a metric run it only when another metric of the sample gave a result ("faithfulness < 0.8", "faithfulness fails"); the metric that is not run costs nothing and is n/a with the reason; a circle or an unknown metric is refused before the first question. In the picker (run only when / unless), --when / --unless, MCP when / unless. The dag metric (DeepEval) is a tree of judgements inside one metric (yes/no, choice, task and deterministic rule nodes; only the path taken is asked; checked for cycles and depth).
• More judge designs. g-eval (steps written once and cached; samples: n averages ratings, because the AI layer has no token probabilities), pairwise (blinded, both orders, win rate against baseline_answer), summarization (DeepEval and Ragas), faithfulness-hhem (claims by the judge, entailment by a classifier you serve, e.g. Vectara HHEM), reward-score (Nemotron reward models on an OpenAI-compatible endpoint). The classifier and the reward model obey privacy level, redaction, Local only and the ledger like any judge.
• SQL production guard. A database marked production: true is asked about once before the generated SQL touches it: the same question and consent as a request to a production environment (the app asks, the CLI takes --allow-production).
• The picker edits text options (a role, topics, rules, a criterion, a tree as JSON), marks the required ones, and has the decision-tree control. The Try dialog takes pictures and the other fields of a record (a conversation, tool calls, should_refuse).
• Tests. Golden mock-judge cases for every worked example (Ragas' topic adherence 0.8, the Tesla multimodal pair, goal accuracy in two calls, G-Eval 0.75 / 0.625, DAG paths, pairwise orders, summarization 0.8, HHEM 0.75, reward 0.8), a conversation run against local servers (stateless history, server session, tool calls, per-turn scores, routes, probe sets, a picture kept by hash only), image/EXIF/limit/consent/redaction tests, and the 5,000 × 40 matrix perf budget (rag-matrix-perf).
• 84 metrics (was 30). New judged metrics on the judge service (cache, ledger, budget, jury and swap as before): context-precision and context-recall with the judged variants, context-entity-recall, factual-correctness, answer-correctness, response-groundedness, hallucination, correctness, completeness, factuality, citation-accuracy, negative-rejection, the claim-level family (claim-faithfulness, claim-recall, claim-hallucination, noise-sensitivity-relevant / -irrelevant …; one claim pass is shared between them), and the judge templates (pass/fail, simple criteria, rubric, instance rubric, checklist, prompt alignment, coherence, conciseness, helpfulness, comprehensiveness, harmfulness, maliciousness). Without a judge in a RAG target run they are n/a, never a guessed score.
• Free metrics added: hole@k, graded ndcg@k (from relevance in the record), bleu, chrf, numeric-diff, json-diff, label-f1, quoted-spans, semantic set metrics, context-size, context-usage, error-rate, throughput, retrieval-score-stats, prompt-injection-resistance, tool-call-f1, tool-call-accuracy. Each has known-value tests.
• Changed meaning, said at the place it matters. hallucination is now DeepEval's metric: the share of unsupported claims, lower is better. It used to be another name of faithfulness; a test file that uses it gets an info line saying so, and faithfulness is the higher-is-better metric. answer-correctness reads claims both ways; the older sentence-overlap behaviour remains for a test file without a judge. In a RAG target run context-precision, context-recall and context-entity-recall are strict (no judge, no score); a test with contexts written in the file keeps the heuristic.
• Custom metrics. metrics/<id>.yaml: a judge criterion, a checklist, deterministic rules, a formula over other metrics (its own expression language, no eval), an embedding set or a sandboxed script, each with examples that act as the metric's tests. AI Lab ▸ RAG ▸ Metrics edits them (live check, Run examples, Try on a sample, usage before delete), Draft with AI writes one as a reviewed proposal (task rag-draft-metric, its examples run before it is shown), the metric picker lists them, testpion rag metric new|list|validate|export|import, MCP rag_list_custom_metrics, rag_validate_metric, rag_save_custom_metric. The metric item kind is live (a git file; copied by rag metric export/import, not yet carried by the workspace bundle). Documented in docs/ai-testing/custom-metrics.md.
• SQL metrics and a read-only executor. datacompy-score (what the generated query returns against the reference query's result) and sql-equivalence (a judge's opinion). The executor accepts one SELECT only (no PRAGMA, ATTACH, writes, extensions or stacked statements, in both readings of a backslash), caps rows and time, opens SQLite read-only on a worker thread that is terminated on timeout, and runs PostgreSQL and MySQL in a read-only transaction with a statement timeout (the source must be marked readOnly: true). Hostile-input tests cover it.
• Ragas parity. --compat ragas (and compat: ragas in a saved evaluation) reads Ragas metric names with Ragas options; rag export --names ragas and rag matrix --names ragas name the columns as Ragas does; rag matrix --ragas-result file brings the scores of a Ragas result table in beside yours as external columns. --repeat-target n calls the RAG system n times per question and scores the mean with the spread.
• The results, more ways to read them. The matrix exports (CSV, JSON, HTML; rag.matrix output surface, testpion rag matrix), a By tag view, Charts, Compare against another run, a Runs × metrics pivot, Score again… from the matrix, Free metrics first in the estimate dialog, the raw call to the RAG system in a sample's drill-down, and Explain this sample (AI task rag-explain-sample, reads only that sample, proposes nothing).
• Monitors and CI. The monitor editor runs a saved RAG evaluation (gates, a number of questions, a share of the samples that are judged, a daily judge budget). The Run in CI dialog has RAG evaluations with gates, a preset, a baseline, a judge budget, metric names, calls per question and, for GitHub, a pull-request comment with the metrics table (--markdown).
• Housekeeping. The flow-format and checks references name the new check types (token budgets raised for that and nothing else). The SQL metrics reach the executor through a hand-over (eval/rag/sql-host.ts), so the metric engine still imports nothing from the runner at run time.
• One catalogue of 30 RAG metrics (RAG metrics, generated from the engine): faithfulness, answer relevance, context relevance, context precision and recall, factual correctness, entity recall, citations; the IR measures (recall@k, precision@k, hit-rate@k, mrr@k, ndcg@k, map@k, r-precision) and the lexical context precision and recall against the gold texts; string similarity; semantic similarity; the limits. A metric says whether it needs no model, a judge or embeddings, what it reads, its direction and its calls. Every older name still works (groundedness, answer-correctness, rag-faithfulness) and so do Ragas' (LLMContextRecall, response_relevancy, nv_context_relevance, ragas: applies Ragas' own definition) and recall@5. In a RAG target run the judged ones are strict: no judge, no score.
• New judged metrics. context-relevance in four modes (NVIDIA's two prompts, DeepEval's statements, TruLens' chunks rated 0 to 3, promptfoo's sentences), Ragas' response_relevancy (questions generated from the answer, compared by embeddings), and a shared claims-v1 pass so metrics that judge the answer's claims split it once.
• The metric picker in the evaluation: search by any name, nine presets (RAG triad, Ragas core, DeepEval RAG, NVIDIA RAG, Retrieval only, Free, Answer vs reference, Ragas NL comparison, Ops), pass marks and options on each row, the inputs the dataset lacks said in amber, formulas and worked examples, and Try on one sample showing the exact judge prompt (nothing sent) before it scores.
• The cost, first. A dialog before a run that asks a judge: judge calls, tokens, a cost range, the calls to your RAG system, the limits it would break, and what the judge receives. rag run --estimate, rag_estimate; exact when re-scoring a run. The run summary has judgeCostUsd.
• The samples × metrics matrix next to Matrix: a column per metric grouped by type, editable pass marks that decide pass and fail again, mean, p10, pass rate, n/a, a 95% interval and the distribution, filters by why a sample failed (retrieval miss, unfaithful, false refusal, ranking, noise, off topic, incomplete, citation, format …), and a drill-down per metric with the claims, the lane of retrieved contexts and Try again with another judge.
• Judge options. A jury (majority, mean, lowest), both document orders (a disagreement is a tie), repeats with the flip rate, skip judging when retrieval failed, shared claims, and testpion rag cache clear.
• More ways in. Presets for OpenAI-compatible (also streamed), LangServe, LlamaIndex, Haystack, Dify and Flowise targets, Suggest mapping with AI (applied to your sample before you see it), limits on the calls an evaluation makes (at once, per minute, timeout, retries), the column mapper for a dataset with your own column names, imports of BEIR, NVIDIA retriever data and promptfoo tests, and contexts: "{{var}}" in a test file.
• Terminal, CI and agents. testpion rag metrics|metric|presets|run|rescore|compare|connect|cache: gates (--min faithfulness=0.8), a metric-by-metric comparison with a paired bootstrap, JUnit and HTML, --two-phase (collect answers with the free metrics, then judge them), rescore (no call to your system). MCP rag_metrics, rag_try_metric, rag_estimate, rag_run, rag_rescore, rag_compare and the prompt evaluate_rag_system.
• Test a RAG system that runs somewhere else. AI Lab has a new RAG tab: connect an endpoint over HTTP, a streamed answer (server-sent events), WebSocket, gRPC or an MCP tool, with the protocol clients TestPion already has. Ask it one question (Probe), then click the answer and the list of retrieved documents in its response (and each document's text, id, score and source, token counts, latency): every click writes a JSONPath into the mapping, which stays plain text you can edit, so the same response always gives the same sample. The target is saved in the workspace (library/rag-targets.json) like a prompt or an evaluation.
• An evaluation of type RAG calls the target for every question of its dataset (Evaluations ▸ RAG ▸ choose the target), scores each answer, and shows the usual runs, results and matrix. Records are question, reference, reference_ids (the gold documents) and reference_contexts; Import Ragas / DeepEval… and Export as… convert their documented dataset shapes (data only, nothing is executed).
• Metrics: normalized-match, token-f1, rouge (1, 2, L), context-recall-ids and context-precision-ids need no model; rag-faithfulness (the share of the answer's claims the retrieved documents support) and rag-answer-relevance ask a judge model claim by claim, with the evidence kept in the result. Latency, tokens and cost of the system under test are scored as before.
• The judge is never guessed. It is the evaluation's Judge, else the AI settings' large model (--judge provider/model, TESTPION_AI_PROVIDER); without one the run stops before the first question and lists what to configure. Judge calls are the AI task rag-judge: counted in AI usage, limited by the daily caps and --max-judge-cost, redacted, shown in "What will be sent", and cached by the content of the prompt, so running again costs nothing. Local only is enforced.
• Terminal and agents: testpion rag targets|probe|import|export (all with --json), eval run --judge … --max-judge-cost …, and the MCP tools rag_list_targets, rag_probe, rag_import_dataset, rag_export_dataset. A demo RAG system for trying it: node examples/servers/rag-server.mjs.
• The RAG target is now a real kind of item (rename, duplicate, move to a folder, delete with Undo, search, links).
• testpion search --ask "<question>" [--in …] [--no-ai] [--dry-run] [--json] and testpion search --query-json <spec> (a query run directly, no model); MCP tool search_ask (a question or a query; read-only). One query engine behind the dialog, the CLI and MCP.

Everyday polish

• History's download control is always reachable. The count row of History (count, RAG dataset, Redacted, Copy, Download and the format arrow) could push the arrow out of view at the default sidebar width; RAG dataset is now an icon (like Ask…) and an output control with no tools of its own never shrinks.
• History says so when the repository is missing. A workspace whose .git was moved or deleted showed "No commits changed this yet" for up to three seconds instead of the error with Try again: the remembered repository root is now believed only while its .git is still there.
• A bundle dropped on the window opens one dialog (it used to open two: the general Import and the bundle preview). The window has one drop listener, which routes by file type; the general Import also hands a bundle over to the bundle preview. A file that is not a bundle ends in Choose another file or Import another format…, not in a disabled button.
• Dialogs that wait for you. Import has Cancel at every step, Choose files… as the main button, switches instead of checkboxes, an announced progress and result, and Coming from another tool?; the metric editor asks before it drops edits and delete has Undo; presets and Clear in the metric picker have Undo; the History dialog says when it could not read the history and offers Retry; modal footers wrap in narrow windows; a disabled main button says why in the footer.
• Plain words. Stage and Set aside are explained in one line, a bundle is defined once, the Git changes rows keep the item name and wrap their buttons, the metric matrix cells open from the keyboard and say passed or failed in a mark (and lowest 10%, spread (min to max) explain themselves), the setup and large-file dialogs say what they do instead of naming git internals.
• Easier to find. Create CODEOWNERS, Share a Debugger Session with the Team and Migrate from Another Tool are in the command palette; Home's report control says Share report; File ▸ Export has Workspace as a Bundle… and Workspace (older format)….
• pace: in test files: a rate limit (pace: 250ms is four requests a second) for steps that carry it, across the whole run and whatever the concurrency.
• A dataset-driven test file can be edited in the Flow view. A test file with a dataset: (one test per record when it runs) is one step on the canvas, labelled "× each row of …", instead of one step per record; moving, editing, connecting, adding or removing steps now writes to the right step instead of failing with "No step … in the flow". A test in a tests: list may carry a dataset: too, and runs once per record. The step's latest result is the worst of its rows'.
• The GitHub Action's job summary shows the run's Markdown report and, with the new triage: true input (adds --triage), the failure triage from triage.md.

Faster and safer

• testpion starts in about half the time (a command went from about 2.0 s to 0.9 s on a loaded machine): the build also writes the command line, the engine and the shared code as one file (dist/bundle.js), which bin/testpion.js loads while it is newer than the build; TESTPION_UNBUNDLED=1 loads the files as before. --version and --help were already instant.
• Opening a workspace with tens of thousands of files no longer stalls. The change watcher walked every file with a stat before returning (3 s for 50,000 files, all blocking); it now walks in 8 ms slices, a file that changes before the walk reaches it is still reported, and a long burst of changes (a build, a branch switch) is reported in batches at the latest 5 s after the first. The test-file tree of 50,000 files takes 54 ms (was 454 ms).
• Faster runs. A run of HTTP tests spends about 13% less CPU per test; a run of 1,000 steps × 100 iterations takes 0.67 s instead of 1.3 s (ids come from a pool of random bytes, secret-key rules are remembered per name, the list of check types is cached, a JSON-path check evaluates its path once). Registering a thousand secrets takes 0.2 ms instead of 88 ms. The history of one test over 60 runs of 5,000 results is 2 to 3 times faster; test generation from a 4,000-operation OpenAPI document takes 1.6 s instead of 6.9 s, and its lint no longer scans the whole document for every problem.
• Long lists come in pages (limit, cursor, truncated, total): a run's results (runs.results: at most 1,000 a page), the flaky tests (new flakyTestsPage), and a search (offset in the query, truncated and nextOffset in the answer).
• Every long list is paged the same way on the CLI and over MCP: --limit / --cursor / --offset (CLI) and limit / cursor / offset (MCP) with truncated and nextCursor in the answer (the envelope's page with --json) for history list, history flaky, the new history results <runId> and MCP run_results, search (plain, --ask, --query-json) and search_ask, search_workspace, history_list, recent_failures, request_history, requests, env list, datasets show and read_dataset; a list that fits is unchanged, and a cut one says more: --cursor …. See the CLI reference and MCP server pages (Paging).
• A collection of more than 1,000 requests runs to the end (the guard against a setNextRequest loop counted requests per iteration with a fixed limit of 1,000; it is now four times the collection's size, at least 1,000).
• Scans remember files. The exposed flows (MCP tools list), the webhook flows and the spec gap analysis read a test file again only when its size or time changed; connection pools, OAuth tokens and test-set verifier state are bounded.
• Switching tabs redraws about half as much. The highlight of the open item is read by the rows that show it, not by the whole Explorer, so a switch redraws the two rows whose highlight changes (20 switches over a 500-request workspace: 11,700 component renders before, 6,200 now). The Import dialog loads when you import, not at start (24 KB less on the start path).
• Under the hood, no behaviour change: the 60-file import cycle is cut, dead code deleted, a check type registered twice is an error, MCP tool modules live in one folder, the biggest files are split, and architecture tests keep it so. The old workspace export (workspace.export, storage/workspace-bundle.ts) is deprecated in favour of bundles and stays for two more releases.
• Git changes by item read the 'before' of every changed file through two git processes, not one or more per file (300 modified files: 311 processes and 28 s before, 2 and about 1 s now), remember the rows of files that did not change since the last refresh, and give the app a turn while thousands of new files are read. The status bar, the explorer and the environments list no longer render again when a refresh finds nothing new.
• Bundles and imports of big libraries. Exporting or previewing a bundle of 2,500 saved items took 20 to 26 s (the library file was read and parsed once per item); it takes under half a second. "Keep both" for 2,500 conflicts applies in 1.5 s (it was 35 s) and Undo of it in 0.4 s (it was over two minutes). Import journals of bundles keep the newest 30.
• Importing a folder of files into a workspace that already holds 10,000 requests reads the workspace's collections once, not three times per file (100 files: 32 s to 5 s; the preview 15 s to 1 s). The list of earlier imports no longer reads their receipts, identical scripts of a collection are converted once, and saving a collection with thousands of requests orders its keys in one pass.
• testpion --version and testpion --help answer from an index written at build time, without loading the command line.
• A Fiddler SAZ file is opened by the hardened zip reader. Importing a .saz (Debugger) used a zip reader of its own that checked nothing: a zip bomb, an entry count in the hundreds of thousands, a name like ../x or a damaged entry now stop the import with a message, the same limits as a bundle's.
• MCP save_output only writes under exports/. save_to is an allow-list now: a new file under exports/ or .local/exports/, never over a file, never in a hidden folder (.husky/, .vscode/) or as a program file, and it asks for confirmation (confirm: true). export_bundle follows the same rule (a numbered name instead of overwriting the default file).
• More MCP tools ask first: export_traces (sends traces to an endpoint), security_tests_run, import_definition (when it updates) and undo_definition_import.
• Imported AI providers never bring secret references. A bundle's provider or remote MCP server is imported without its {{$secret…}} / {{$env…}} key, headers and variables (you enter the key yourself) and with only the fields TestPion knows; Review says where it sends things and warns when Replace would move one of yours to another address.
• Importing from a link follows the network policy. The link and every redirect hop are checked (https only where private networks are blocked, at most 5 hops); IPv6 spellings of private addresses (::ffff:7f00:1, NAT64, 6to4, IPv4-compatible) are recognised.
• The pre-commit hook no longer expands the workspace folder name ($(…), backticks, quotes, line breaks stay text).
• Git paths: the .git folder is refused in any spelling (.GIT, GIT~1, trailing dots) and at any depth; a file name with a line break cannot be tracked by LFS or by reference; the secret scan's allowlist fingerprints are salted and slow (older allowlists keep working).
• Import limits: at most 8 import previews are kept (30 minutes), uploads are measured before they are decoded, zip bombs are caught for entries of any size and as a whole, and a bundle holds at most 512 MB once unpacked.
• RPC parameter checks for the RAG, bundle and large-file calls, and a test that fails when a new call has none.

Downloads

See the download page for which file to choose and the installation guides.

The installers aren't code-signed yet: Windows SmartScreen and macOS Gatekeeper ask you to confirm the first launch.