Skip to content

v0.16.0

Latest

Choose a tag to compare

@testsprite-cli-release testsprite-cli-release released this 01 Nov 07:59

Added

  • test run takes several test ids. testsprite test run <id> <id> … dispatches any number of ids as batches: ids are grouped per project (read from each test) and sent 50 per request, one chunk after another, through the same wave-ordered batch as --all, with --wait, --report, --summary-file and the shared exit-code precedence. If a later chunk fails, the command exits with that chunk's exit code and the output (and its nextAction) still lists every run id the earlier chunks accepted; without --wait those runs are not polled, with --wait they are (see Changed). A JUnit report across several projects uses each row's own project id as the classname. Backend-only ids go through the batch even when the project's default environment is a loopback URL, since backend tests do not use the tunnel.
  • test run --all --status <list> runs only the test cases whose latest status (the status of their latest result, as in the Portal's Latest Status column) matches one of the comma-separated values (passed|failed|blocked|cancelled|not_run; not_run is a test case that never ran). draft, ready, queued, running and unknown are rejected (exit 5). --env <name> only picks where to run; it does not change which test cases are selected. test run --all --status failed,blocked runs what did not pass. An empty, whitespace-only or commas-only --status is a usage error (exit 5) on test run and test rerun, before any request, including under --dry-run. --allow-empty now covers --filter/--status matching nothing.
  • test list --status and test delete-batch --all --status take not_run (a test case that never ran: draft or ready) and list the same five values as test run --status (passed|failed|blocked|cancelled|not_run). The older values (draft, ready, queued, running, unknown) still work until 1.0 but are no longer listed; test rerun --status is unchanged.
  • test run --skip-dependencies (backend test cases only) runs the named test cases without the test cases they depend on, on a single id, several ids and --local runs. The backend confirms it: a response to POST /tests/{id}/runs or POST /tests/batch/run carries skipDependencies: true exactly when the request did and the backend handled the field, and omits the key otherwise. A batch whose response lacks the echo is cancelled and exits 7 (skip-dependencies-not-honored); a single or --local run prints an [advisory] and carries on (a Portal run of a test case also includes the test cases it depends on), as it does for an unconfirmed --target-url.

Changed

  • No more 61-second resend; runs queue on the server. test run --all, test run <id> <id> … and test rerun --all no longer resend deferred ids after 61 seconds under --wait, and the CLI's own in-flight limit is gone: runs queue on the server as Portal runs do. Each request starts one run, so a large batch is accepted whole and its results show queued (text progress and JSON) until they start. deferred[] can still appear in rare cases (a request spanning several projects where one project's lock stayed busy, or a test case that went in flight during dispatch); those ids are listed (stdout deferred, stderr line, not-dispatched, JUnit and summary rows), never resent, accepted runs are still awaited under --wait, and the exit code is 7. The 60 run triggers per minute per API key and the 409 for a test case that already has a run in flight are unchanged; ci run is unaffected.
  • test run --wait: one rule when a request fails part-way through a batch (a one-time behaviour change). This covers several ids and --all with --filter/--status. A request that fails after earlier requests were accepted now stops the dispatch (nothing further is sent) and, under --wait, polls every run already accepted (and every in-flight run it joined) to its verdict before exiting: stdout, --summary-file and the JUnit report carry those real verdicts, and the ids never sent are printed on stderr and written as not dispatched rows (stdout lists them in notRunTestIds). The exit code is the failed request's own, and it outranks rate-deferred (7), a failed verdict (1), credits (12) and conflict (6). What differs from before: a failed later chunk under --wait used to list its earlier chunks' runs as not awaited errors without polling them; it now exits with that request's code, for example 11 for a rate limit or 6 for a precondition. Without --wait nothing changes: the accepted run ids and the ids never sent are listed and nothing is polled. A run that sends a single request and fails is still rethrown unchanged. ci run, test rerun and test flaky are not affected.
  • Docs and shipped skills match the current CLI. The testsprite-verify skill now accepts a loopback environment --url and treats --expected-version as optional, the README and DOCUMENTATION.md cover ci, completion, test open, tunnel and auth remove, and docs links point at docs.testsprite.com.
  • --help is accurate and shorter. About 50 help texts were corrected against the code and the current backend, with no behavior change, and every command has a one-line summary so parent tables list one line per command (test --help 149 → 42 lines, with the deprecated rerun and flaky no longer listed). Highlights: test cancel and test run agree on Ctrl-C (a run through a tunnel the CLI opened is cancelled by default); test run lists exits 12/13 and testlist run gains an exit-code list; --page-size alone returns one page; schedule delete keeps past runs and schedule update --env keeps the pin when omitted; project auto-auth is "(paid plans)"; the root help lists the environment variables it reads.

Deprecated

  • test rerun is deprecated and removed in 0.17.0. It keeps working with the same requests, output and exit codes; each invocation now prints one [deprecated] line to stderr (before argument validation, also under --dry-run), a success JSON object on stdout ends with a deprecations array of { command, replacement, removedIn } (replacement is a template: test run, to be used with your own ids and flags), and its own --help starts with [deprecated]. It is a hidden alias: test --help no longer lists it. The deprecations key appears in stdout JSON only: not in --summary-file, JUnit reports or error envelopes, and not in text output. A server-sent deprecation advisory on a rerun response is dropped so the notice appears once.

  • test flaky is deprecated and removed in 0.17.0, with the same three signals. It has no replacement; run test run <test-id> again to sample a test.

    Instead of Use
    test rerun <id> [--env] [--wait] … test run <id> with the same flags
    test rerun <id> --no-auto-heal test run <id> --no-auto-heal
    test rerun a b c test run a b c
    test rerun --all --project <p> [--filter] [--status] test run --all --project <p> with the same flags; --status takes only passed|failed|blocked|cancelled|not_run
    test rerun --all --skip-terminal test run --all --status not_run
    test rerun <be-id> --skip-dependencies test run <be-id> --skip-dependencies
    test flaky <id> --runs N no replacement; run test run <id> again

    Differences you will notice after switching: test run starts a fresh run, so a user-authored case replays instead of being re-authored (the server's auto-heal default applies); the JSON has no closure, notFound or advisories; an unknown id fails the whole multi-id batch (exit 4) instead of appearing in notFound[]; and new runs carry no createdFrom: rerun:<id>.

Fixed

  • A batch run names the no_producer and producer_in_flight refusals. A backend test case that needs a variable no test case captures, or whose producer is running in another run, was reported as 1 not dispatched (1 not dispatched) and described as already in flight. It now reads no producer for its variables or producer running in another run, with the server's message where it sends one. Under --wait such a test case is now reported with that reason as blocked before dispatch (exit 6), as a single test run reports it, instead of being polled and printed as a bare blocked (exit 1). Its latest status is blocked either way: the server records the refusal as a blocked result, as the Portal shows. A no_producer test case is told to add a producer instead of to wait.
  • test delete-batch --all --status '' no longer deletes every test in the project. An empty --status was read as no filter. An empty, whitespace-only or commas-only --status is now a usage error (exit 5) before any request, including under --dry-run, as on test run.