Skip to content

v0.13.0

Choose a tag to compare

@testsprite-cli-release testsprite-cli-release released this 28 Sep 03:47

Added

  • --env on test create --run and test create-batch --run, and stricter --local. Both create commands accept --env <name> when they trigger the run (--env without --run exits 5). With a backend at v0.18.0 or later: a misspelled or deleted --env name fails with the available names instead of running against the default environment; --local refuses a backend test, or a named environment with a public URL, before a tunnel is opened; and existing API keys can run --local without being re-issued.
  • auth status and usage say when the workspace is paused or payment-pending. GET /me reports the workspace's billing standing, and only --output json carried it; the text renderers printed a plan and balance for a workspace whose next command was going to be refused. auth status now prints a workspace: line and usage a state: line whenever the state is not ok: paused — <notice> <cta>: <billing url>, or pending — a scheduled plan change or a payment retry is in progress; the workspace keeps working meanwhile. An older backend that sends no workspace status renders exactly as before.
  • Run history shows when a plan hides older runs. test result --history and schedule run list print the hidden count, retention window, and pricing and billing links when the backend reports omitted runs. JSON output passes through the backend's history metadata, and the empty-steps earlier-run suggestion names its limited window.
  • test run --all --target-url <url> applies a run-time target to the batch. Frontend tests use the URL override; the CLI probes it before dispatch, checks the backend echo on every dispatch, and cancels accepted runs if the backend ignores it.
  • --target-url remains supported for run-time frontend targets. test run, test create, and test create-batch show it in help without a deprecation warning. A run reuses an environment at the URL's origin or gets a temporary one that TestSprite removes when the run finishes; --env supplies its settings to the override URL.
  • Run many tests against your local app through one tunnel. Pass testsprite test run <id> <id> … --local <port> or testsprite test run --all --project <id> --local <port>; the latter runs frontend tests and reports backend tests as skipped. One binding serves the whole invocation, with up to 5 runs at once by default (--max-concurrency, at most 10). Ctrl-C stops queuing, cancels unfinished runs on an owned tunnel by default, and prints partial results; tunnel loss also stops queuing and exits 10. JSON output is one { tunnel, maxConcurrency, results, skipped, summary } object in test order. Telemetry records the concurrency limit and observed peak as counts.
  • tunnel list and tunnel stop --all --confirm recover tunnel slots. See every live client id, connection status and expiry, then revoke one binding or explicitly stop all of them. A dry-run bulk stop previews sample ids; it does not inspect or revoke your real tunnels.
  • The tunnel-limit error names your tunnels. When the account's live-binding limit is reached, text output lists the client ids and the commands to inspect or stop them; JSON output carries the ids in the error details.
  • tunnel start and test run --local print the tunnel client id before connecting. The id appears on stderr as soon as the binding is minted, so you can identify it — and stop it — even if the connection fails or the process is killed before the online receipt.
  • testsprite ci doctor / testsprite ci connect — find the broken link before it silently costs you a gate. A workflow triggered by deployments only fires if the repo actually receives GitHub Deployment events, and when the deploy platform isn't git-linked to the repo, nothing is ever written — so the gate never runs, with no error, no annotation, and no check to notice. ci doctor reads the repo's deployment history through gh and turns that invisible state into a verdict: linked (a gate will fire, exit 0), none (no events — either not linked or linked-but-never-deployed, which GitHub cannot distinguish, so the output names both), production-only (a PR gate keys on preview deployments and would never run on a PR), or stale. Failing to read the history (no gh, not authenticated) exits 10 rather than 1, because "I couldn't check" must not be reported as "not linked". The linked/stale call is made on the newest preview record specifically — a repo that ships production daily but whose last preview deployment is months old is stale, not healthy. ci connect fixes the common case by running the platform's own CLI as a subprocess (npx vercel link / vercel git connect), strictly non-interactively and never with --yes — on an unlinked directory it requires --project <name> (an existing project) rather than letting the Vercel CLI create a stray project named after the directory; your Vercel credentials stay with the Vercel CLI, and this command never asks for, stores, or transmits a platform token. Since linking only affects future pushes, ci doctor --wait closes the loop by watching for a new deployment record to land — any environment counts, because a new record is the hard fact that events reach the repo.
  • testsprite setup says where an API key comes from. A line naming the API-keys settings page precedes the key prompt; the empty-key, --from-env and non-interactive errors carry the same page as their next action; and the README / DOCUMENTATION link points at that page instead of the marketing homepage. Against an API endpoint the CLI cannot map to a Portal, the hint names the page path without an origin rather than guessing at one.

Changed

  • A fresh test run lets the server heal a drifted test; --no-auto-heal opts out. test run and test run --all now send the opt-out only when you pass --no-auto-heal; whether a run heals by default is decided and rolled out server-side. When the opt-out does not take effect (a server that predates the field, or one that declines it), the CLI says so on stderr instead of silently healing. Rerun and heal messages no longer quote V2-era prices.
  • Ctrl-C lets an in-flight request finish unwinding instead of exiting from the signal handler. Every command now drains the request it was making before exiting (the pattern that crashed libuv on Windows is gone everywhere, not only under --wait); exit codes 130/143/129 are unchanged, a second signal gives the request up to 2 s, a third exits. setup no longer installs agent skills after an interrupted identity lookup, batch delete/cancel stop at the interrupted item, and an interrupted download removes the partial file.

Fixed

  • test run --wait no longer attaches to an in-flight run on a different environment. When a run for the test is already in flight, test run --wait checks that run's environment against --env (or the project's default) and attaches only on a match. When the environments differ, or the server cannot report the in-flight run's environment, it exits 6 with test wait <run-id> as the next step instead of reporting another environment's verdict. A run read that carries no environment at all (a V2-served run, or an older backend) is still attached as in 0.12.0 when you omit --env, with the same --target-url check.
  • Ctrl-C stops test flaky during a rerun trigger. An interrupted request now exits with the signal code instead of being counted as a failed attempt and starting more reruns.
  • test run --all and test rerun --all no longer count a retried test twice. A deferred retry that came back with a test the first dispatch already ran was polled twice and inflated summary.total; duplicates are now merged, and when a retry did dispatch an extra run the CLI names it with the testsprite test cancel <run-id> to stop it. test run --help documents the in-flight run cap.
  • Billing refusals now show a usable upgrade path across CLI commands. Plan exclusions, plan limits, paused workspaces and insufficient credits keep their server text, gain pricing and billing links in human and JSON errors, and retain their billing exit codes in batch runs; rollout gates receive no upgrade link. The CLI reads the workspace gate's refusals by the gate's names — paused, not_entitled, limit_exceeded — and still reads the deprecated billing_hold / plan / plan_limit from older backends; an error the CLI builds itself (a batch refused because the workspace is paused, a raw gate body without a reason) now carries the gate's name in details.reason.
  • Plan, steps, and password files are read through one opened descriptor. --steps, --plan-from, --plans and --password-file open the path once and read through that descriptor, and the size cap is enforced on the bytes actually read, so a file that changes or grows between the check and the read cannot bypass it. Error wording is unchanged. The dry-run route matcher also escapes every regex metacharacter in its route templates.
  • JUnit reports count every non-dispatched member as skipped. testlist run --report junit and test run --all --report junit used to build the XML from polled results only, so a member that was never dispatched (not in the list, an in-flight conflict, an unresolved deferral) was counted by the summary but missing from the report — tests="1" skipped="0" next to a summary saying total 2, skipped 1. Those members now appear as <testcase> elements with <skipped message="…reason…"/>, on every path including the all-conflict and zero-dispatch tails, so tests and skipped agree with the summary.
  • The in-flight auto-resume advice names --env instead of --target-url. When a second test run attaches to a run already in flight, the advisory tells you to re-trigger with --env <name> (plus --local <port> for a loopback environment) when the run's environment is known; a run that records none gets a plain "re-trigger" hint instead, since there is nothing to name.
  • test list checks credentials before it demands --project. With no key configured the command now exits 3 with the setup next step instead of --project is required; with a key the exit-5 validation is unchanged.
  • /me and usage responses are checked before they are printed. A response missing a required field now fails with a clear error instead of rendering blanks or crashing; doctor's connectivity probe stays lenient. Community contribution (#291).
  • Saving credentials on Windows no longer fails while another testsprite holds the lock. Creating or removing credentials.lock while another process is closing it fails on Windows with EPERM/EBUSY/EACCES instead of EEXIST; those codes on the lock file are now treated as contention and retried within the normal wait, so a concurrent command no longer loses a profile. POSIX permission errors stay fatal.
  • The live progress line fits the terminal width. It is truncated to the terminal's columns, so it redraws in place instead of wrapping onto new lines. Community contribution (#309).
  • Agent-skill read failures are logged under --debug. The skill nudge used to swallow a failed read silently; with --debug the error now appears on stderr. Community contribution (#317).
  • TESTSPRITE_PROFILE selects the profile again — it was ignored by every command. The documented order is --profile > TESTSPRITE_PROFILE > default, and only the flag worked: each command resolved its own profile as flag ?? 'default' before calling into config loading, so the concrete string 'default' arrived where undefined was needed and the environment variable was never read. Exporting TESTSPRITE_PROFILE therefore kept using the default profile's key — silently, with no error, when both profiles held a valid key: a project created under a personal key while the environment said otherwise. All commands now resolve the profile through one shared helper, which keeps the existing normalization (a blank or whitespace-only TESTSPRITE_PROFILE falls back to the default profile instead of failing the profile-name guard on every command). doctor reports the profile it actually resolved, so the active profile is visible, and a selected profile that has no stored credentials now says so by name: the AUTH_REQUIRED message is unchanged, but its next action reads No credentials are stored for profile "x". Run testsprite setup --profile x… and details.profile carries the name, rather than sending you to configure the default profile you already have. --profile and the per-profile TESTSPRITE_API_KEY / TESTSPRITE_API_URL overrides behave exactly as before. Behaviour change: commands that write credentials follow the same order, so with TESTSPRITE_PROFILE exported, setup / auth configure write to — and auth remove deletes — that profile rather than default. If you exported the variable earlier, found it had no effect and left it in a shell rc, unset it or pass --profile default.