Skip to content

v0.17.0

Latest

Choose a tag to compare

@testsprite-cli-release testsprite-cli-release released this 19 Nov 08:06

Writing tests now goes through the project's agent: plan asks it for tests and apply saves what it proposes; run executes them and result get shows one test's evidence. If the agent is not yet enabled for your account, the agent commands print The TestSprite agent is not enabled for this account yet. and exit 13; stay on 0.16.x until it is. For a command-by-command migration, see Upgrading from 0.16.

Breaking changes

  • One launch entry: testsprite run. The former test run, test wait,
    test cancel and testlist run command paths are removed without aliases;
    old invocations exit 5. Migrate project launches from
    test run --all --project P to run --project P --wait, and named launches
    from test run T1 T2 to run T1 T2. Resume with run wait R1 R2; cancel
    with run cancel R1 R2. Replace testlist run L with run --testlist L,
    and testlist run L --case T1 --case T2 with run T1 T2 --testlist L.
    Root run rejects --all and --case. Update custom CI scripts and agent
    guidance, and reinstall skills reported stale by testsprite agent status.
    Use test list and test get to inspect cases. Test-list management and
    ci run retain their paths.

Added

  • A run card names its environment, and apply --env <name> picks another. The conversation shows which environment a proposed run goes to (the one the agent named, else the project's default), and testsprite apply --env staging starts it elsewhere, as the portal's card lets you pick one; apply says where the run started. Needs a backend that carries the card's environment; an older one shows the card without it.

  • apply --answer answers the agent's question card. When the agent asks one to four questions, each with options, the conversation shows them numbered with lettered options, and testsprite apply --answer 1=b --answer 2=a,c answers them (your own words in quotes). An apply short of an answer is refused and the card keeps waiting. Needs a backend that carries the card's questions; an older one still shows the card as an action.

  • The project's agent writes tests: plan, apply, reply, reject, status, stop and discard. testsprite plan "<what to test>" asks the project's agent for tests (it explores the app, or reads --file documents and your --diff); apply saves what it proposed and reject turns it down with what to change; reply answers it in the same conversation, status shows where it is, stop stops it and discard drops its task. They replace the test-writing commands (see Removed).

  • A backend environment's API sign-in is set on project env. project env create and update take it the way the project's agent writes it in environments/<name>.md: --auth none|bearer|basic|apikey with --credential-file (or --credential) for a static credential, --header "Name: value" (repeatable) and --header-file for the extra headers every request carries, and --auto-auth-file for a sign-in that refreshes the credential. They work on any environment, not only the default, and are saved through the same server path as the Portal's environment card; the auto-auth file is sent as it is and the server refuses a field it does not know. project env get / list show a backend environment as the agent reads it: its auth, whether each secret is set, and its headers with secret values hidden. Saving a sign-in needs a plan with advanced environment settings, as in the Portal. Which endpoint needs which header stays a Portal decision.

  • setup keeps .testsprite/ out of Git. Inside a Git repository, testsprite setup appends .testsprite/ (where failure artifacts are written) to the project's .gitignore when it is not already ignored; skip it with --no-gitignore, and --dry-run only reports it. The entry is appended rather than rewritten, a symlinked .gitignore is not written through, and a failure here is a warning that never fails setup (it also runs when the agent skill install fails after credentials were saved). testsprite doctor gains Git repository and Gitignore safety checks: inside a repository it warns (never fails) when no .gitignore between the project directory and the repository root ignores .testsprite/, and outside a repository both are informational. Thanks to @sandman-sh (#257, closes #258).

  • result list and result get read test results. They replace test result, test steps, test failure get, test failure summary and test artifact get (see Removed). testsprite result list --run <run-id> lists one run's results; testsprite result list --test <test-id> lists one test's results across runs, newest first (--status, --env, --limit 1-100; cancelled results only when --status names them). testsprite result get <result-id> reads one result: the verdict, the error, its cause and fix, the failed step with its screenshot, and every step in --output json. --out <dir> downloads it instead: result.json, the video, steps/001.png / steps/001.html …, and sign-in/ for a result blocked at sign-in; --failed-only keeps the failed steps and one on each side, and every step when none failed (text mode says so on stderr). In an existing <dir>, result.json, video.*, steps/ and sign-in/ are replaced and nothing else is touched, so use one directory per result (.testsprite/results/<result-id>). A file that cannot be downloaded leaves the rest written and exits 10; when <dir> already holds a download of the same result, that file keeps its earlier copy (kept: true in failed, and result.json names it). Both commands exit 0 whatever the verdict.

  • result get shows a backend test's output and traceback. Text prints the last 20 lines of its Python traceback (trace) and the first 20 of its captured output (apiOutput), each cut to 2,000 characters, with a line saying how many lines and characters were left out (above the traceback, below the output) and that --output json and result.json hold them; --output json and --out <dir>'s result.json carry both whole. Both are null for a frontend test, and an older backend sends neither.

  • ci run gates pull requests whose platform posts only a commit status — shape C. Some platforms create no GitHub deployment for a pull request (Vercel with its deployment_status events switched off, builds skipped by an Ignored Build Step, Cloudflare Pages) and leave only a status or check on the head commit, whose link is the platform's dashboard rather than the site; a pull_request workflow then has no event carrying the preview URL and starts before the build finishes. Two new ci run flags cover it. --wait-for-status <name> holds the run until the status context or check run of that name succeeds on the head commit — both GitHub APIs are polled and matched exactly, so which one the platform writes does not matter — for up to --status-timeout (default 900s, its own budget); a failed, errored, skipped or cancelled check, or a success whose description says Ignored / Canceled / Skipped, is a ::notice:: plus a one-line job summary, zero step outputs and exit 0 (nothing was tested), and a check still pending at the deadline is exit 7 naming every status and check seen on the commit. --target-url-template <pattern> derives the preview address from the pull request with the GitHub App's own placeholders ({pr}, {pr-number}, {branch}, {branch-slug}, {sha}, {short-sha}; same branch-slug rule, so a Portal trigger's pattern renders the same URL); a pattern the run cannot fill is refused with exit 5 naming the placeholder, never guessed, and ci run never falls back to the environment's URL or production; a placeholder in the host part must render to letters, digits and - (a raw {branch} named attacker.example/x would move the host, so it is refused the same way). The wait is bounded as a whole: each GitHub read gets the remaining budget as its own timeout, both lists are read to their last page, a round in which one endpoint did not answer decides nothing, and a rate-limited answer's cooldown is honored inside the deadline. The ci init github template's placeholder comment and the testsprite-ci skill describe shape C and how to read the name and the pattern off a recent pull request with gh.

Removed

  • run --no-auto-heal. Whether a run heals drifted generated code is the server's decision for each run, and code a person wrote is never re-authored, so the opt-out is gone: run sends nothing about healing and prints no advisory. A script that still passes the flag exits 5 (unknown option). The launch's autoHeal in --output json still says what the server decided. A 0.16 CLI that still passes the flag gets the strict replay it asked for, and its --output json launch carries a deprecation notice saying the flag is gone.

  • test code get and schedule run list. Inspect generated test code and a schedule's execution history in the Portal. Both commands, including their now-empty test code and schedule run groups, are deleted without aliases or a deprecation period; old invocations exit 5 as unknown commands. Test reads, run/wait/cancel, and schedule CRUD remain available.

  • test wait --max-concurrency. The former option is removed and is not supported by run wait. Waits start one independent results-listing loop per distinct execution under one shared deadline. Set --max-concurrency on run launches only; it controls server dispatch per execution.

  • Tests are written by the project's agent; the CLI no longer writes them. A coding agent runs tests, reads their results and fixes product code; new, changed and removed tests go through the agent (testsprite plan "<what to test>", then testsprite apply; testsprite reply / testsprite reject to iterate). These commands are deleted, with no deprecation period; stay on 0.16.x if you still need one:

    Removed Use instead
    test create, test create-batch, test scaffold, test lint, test plan generate, test plan accept testsprite plan "<what to test>", then testsprite apply
    test update, test delete, test delete-batch, test code put, test plan put testsprite plan "<test> should …" (or reply in an open conversation), then testsprite apply
    project docs upload attach the document to the request: testsprite plan --file <path>
    test rerun run <id...>, or run --project <id> for --all; run takes no --status or --skip-terminal
    test flaky no replacement; run run <id> again
    test open the Dashboard: link that run prints
    project sign-in get, project sign-in set an existing environment's sign-in is set in the Portal; project env create still takes the sign-in flags
    project credential project env create / update with --auth and --credential-file
    project auto-auth project env update --auto-auth-file <json>; --auth switches it off
    project create --url, --local, --username, --password, --password-file, --skip-preflight project env create <id> --name default --url … (or --local <port>), which takes the sign-in flags
    project create --description nothing: a project has no description
    project update --url, --local, --username, --password, --password-file, --skip-preflight project env update <id> <name> …
    test list --status, --env, --created-from nothing on test list: a case carries no result; result list --test <test-id> --limit 1 reads the latest one
    test run --all --status run <id…> with the caseKey of each result result list --run <run-id> --status failed,blocked lists
    ci doctor, ci connect none; read deployment history with gh api repos/<owner>/<repo>/deployments and link the platform with its CLI
    init, auth configure, auth whoami, auth logout (hidden aliases) setup, setup, auth status, auth remove

    The first-project hints after setup, project create, an empty test list and an empty run --project <project-id> now point at testsprite plan, then testsprite apply. The schemas/plan.schema.json plan-file schema is no longer shipped. auth status and doctor no longer print the routing advisory (its one open gap was test rerun).

    Skills installed by an earlier CLI name removed commands: run testsprite agent status, then testsprite agent install <target> --force for each target it reports stale (own-file targets keep a .bak; the codex section is replaced by a plain re-install). testsprite doctor warns while the verify skill is stale.

  • result list and result get replace the six read commands. They are deleted with no alias; an old invocation is an unknown command (exit 5). Stay on 0.16.x if you still need one:

    Removed Use instead
    test result <test-id> result list --test <test-id> --limit 1, then result get <result-id>
    test result <test-id> --history result list --test <test-id> (--status, --env, --limit)
    test steps <test-id> (--run-id) result get <result-id> (its steps)
    test failure get <test-id> result get <result-id>; --out <dir> downloads the evidence, --failed-only narrows it
    test failure summary <test-id> result get <result-id> (cause, fix)
    test artifact get <run-id> result get <result-id> --out <dir> (the id test artifact get took is the result id)
    test diff <run-a> <run-b> none

    Against test artifact get, result get --out <dir> has no default directory (it was ./.testsprite/runs/<run-id>/), names step files steps/001.png / steps/001.html (was steps/01-screenshot.png / steps/01-snapshot.html), no longer writes meta.json, failure.json, code.<ext> (inspect generated test code in the Portal), .partial or the per-step NN-evidence.json, and keeps going when one file fails.

    Not carried over: the --source, --since and --rerun filters and the cursor paging of test result --history (result list --test takes --status, --env and --limit 1–100 instead), and, from test result and the failure bundle, the --include-analysis flag (result get always includes the cause and fix, as cause / fix), failureKind, codeVersion and snapshotId.

    The JSON fields, from the removed commands to result get / result list:

    Before (test result, test result --history, test failure get) Now
    status / verdict (lower case) testStatus, upper case: PASSED, FAILED, BLOCKED, RUNNING, CANCELLED
    executionStatus processStatus
    runIdIfAvailable; history runId resultId
    testId caseKey
    test failure get's result.*, failure.apiOutput, failure.trace the top level of result get
    analysis.rootCauseHypothesis, failure.rootCauseHypothesis cause
    analysis.recommendedFixTarget, failure.recommendedFixTarget { kind, reference, rationale } fix, plain text (there is no fix.kind)
    failure.evidence[]; steps[].screenshotUrl, steps[].htmlSnapshotUrl each step's screenshotSignedUrl / htmlSignedUrl (steps[], failedStep)
    failure get's code inspect generated test code in the Portal
    failedStepIndex (a number) failedStep.stepIndex (a string)
    history createdAt created
    environment { id, name } envId (result get), environmentName (result list)
    startedAt, finishedAt, targetUrl, targetUrlSource, summary, failureAnalysisUrl; history source, isRerun, createdFrom gone; created / modified are the result's own record times, not the run's start and end
    history { runs[], nextCursor } { results[], hiddenRunCount, retentionDays }, with no cursor
    exit 4 from test failure get when the test had no failing result, and from test artifact get when the result was not ready, not failing or cancelled result get exits 0 whatever the verdict: read testStatus
  • Per-account platform routing. Every account now runs on the same platform, so auth status drops its routing: line and doctor its Routing check, and the org and workspace lines render whenever the backend reports the org wallet. run --wait, run wait and run --project <project-id> --wait no longer fall back to the test's latest result (GET /tests/{id}/result) for a backend run whose row never finished, because backend run rows now finish; a row that does not finish times out (exit 7). The run --project <project-id> advisories and help about frontend tests skipped by the old backend-only engine are gone.

  • A removed command or option says where it went. A removed command is still an unknown command (exit 5), and its error adds one line naming what to use instead, or that there is none (in --output json, the envelope's nextAction), for example `test create` was removed in 0.17.0; tests are written by the project's agent: testsprite plan "<what to test>", then testsprite apply. The removed options of project create, project update and test list, and the test run, test rerun and testlist run options that run does not take (--all, --case, --status, --skip-terminal), do the same on their unknown option error. The line names the option, never the value given with it, and takes the place of the (Did you mean …?) guess, which could name a similarly spelled command that is not the replacement (test lint → list).

Changed

  • test list and test get return the test case, not its latest result. The object is the case as it is written: id, slug, title (was name), description, type, category, priority, version { hash, label }, stepCount and, on get, steps (was planSteps). status, createdFrom, statusByEnvironment, headlineEnvironment and details are gone, and with them test list --status, --env, --created-from and test run --all --status, which selected by a result. Text mode lists ID / SLUG / TITLE / TYPE / CATEGORY / PRIORITY / VERSION / STEPS / UPDATED. The latest result of a case is result list --test <test-id> --limit 1 (its status is testStatus, upper case), and result get <result-id> reads it. Needs a backend that answers CLI 0.17.0 with this shape.

  • A project is created without an environment. project create no longer takes a URL or a test account and no longer requires one for a frontend project; add where the app runs with project env create, and the first environment becomes the project's default. project get / list no longer report targetUrl or originMode (read the environment with project env get), the hint after project create names the environment step first, and --instruction is accepted on backend projects too. A secret handed to one of the removed flags is never echoed back in the unknown-option error, and neither is a URL that carries credentials (--url=https://user:password@host).

  • usage shows the organization wallet only. The per-user credits & plan block, its per-run cost lines and its ~N runs estimate are gone: the backend's top-level credits / subPlan / creditsPerRun are a flattened copy of the same wallet (still passed through in --output json).

  • Dashboard links are only the ones the server sends. The CLI no longer builds https://www.testsprite.com/dashboard/tests/… links itself when a run, batch or project response carries no dashboardUrl; those old routes only redirect to the dashboard home.

  • Portal links use the current routes. API-key, billing and credit-refusal hints point at /dashboard-v3/settings/apikey and /dashboard-v3/settings/billing instead of the old /dashboard/settings/… pages.

  • A run already in flight is attached only when its environment is verified. On a 409, run --wait and run --project <project-id> --wait no longer attach to an in-flight run whose read carries no environment (an older run shape); like a null environment, it is reported as a conflict (exit 6) with run wait <run-id> to follow it.

  • Update scripts that consume launch or wait JSON. A runId identifies one execution. A resultId identifies one test's result in it. Use the two top-level arrays on every run selector and on run wait:

    Read Use
    runs[].runId Resume or cancel that execution. A selection spanning projects can return multiple runs.
    results[].resultId Read one test's verdict and evidence. Results can be empty immediately after launch.
    summary on a wait Counts for this command. A completed failure exits 1. An incomplete timeout or interruption includes partial: true.
    runFailures[], when present Execution-level diagnostics, including a cancellation or a timeout without a matching test result.
    --summary-file / JUnit The CI sidecar keeps per-test rows in results[]. JUnit testcases carry resultId where available.
  • run wait and run cancel take execution IDs. Waiting follows the whole execution, including integration results added after the first tests finish. Cancellation stops all its queued and running work. A finished or already-cancelled execution can be cancelled again safely. Read the per-result refund field for the outcome. A failed refund can be retried by cancelling the same run again.

  • Project, named-test and list launches return any known test outcomes in results[]. A list that refuses members reports their IDs and reasons. Project launches that lose the acknowledgement of a request report it under unconfirmedProjects. Use the returned retryIdempotencyKey to retry the same selection.

  • A request's retry wait ends at its deadline. A 429 Retry-After or backoff wait inside the HTTP client now ends when the request's own deadline passes (--timeout on run wait and on each poll), as well as on Ctrl-C.

  • Several --local tests share one tunnel and start one run per project. run <id> <id> … --local <port> and run --project <project-id> --local <port> start one run per request (per project, up to 5,000 named tests) through one tunnel, instead of one run per test.

    For project and named-test launches, --max-concurrency controls dispatch per execution (default 5 with a tunnel, maximum 15). The default wait through that tunnel is 1,200 seconds. Saved test lists use their own limits. If the launch fails partway through, keep its run receipts and inspect any unconfirmed projects before retrying.

  • Text output never carries a terminal escape from the server. In text mode, everything a command prints on stdout and every line it writes to stderr drop the escape sequences and C0/C1/bidi control characters that server text (titles, names, errors, URLs, the agent's replies) can carry; line breaks and tabs stay, and a carriage return becomes a line break. An escape sequence ends on its own line, so it never removes the lines the CLI wrote between two server values. An id, a name, a title or a URL inside a line or a table cell stays on that line, and so do an error's message and request id, a --verbose line, the tunnel client's log lines and a prompt that names a server value (the password prompt of apply). The completion script is printed as it is, on a terminal and on a pipe. --output json on stdout and stderr keeps the server's text and writes DEL, C1 and the bidi controls as \uXXXX: the same JSON value, with no 8-bit CSI or OSC byte and no hidden reordering; a file written with --out is plain JSON.

  • testsprite test --help describes the group as "List and inspect tests", and the line printed on Ctrl-C points at testsprite run wait <run-id> or testsprite result list --run <run-id>, or, with no run id printed yet, testsprite result list --test <test-id>, instead of testsprite test list.

  • run cancel --help no longer promises a refund, nor does the line a --local run prints when the tunnel it borrowed is no longer registered and its run was cancelled. For a frontend run cancelled before a terminal verdict the server attempts one, and each result's refund says what happened.

  • A batch --wait reports its runs in the order they were dispatched, not the order their polls finished: text rows, JSON receipts, the JUnit report, --summary-file and the Resume/Cancel line, so two runs of the same batch print in the same order.

Fixed

  • Interrupted runs and result history retain the latest observed verdicts. result list --test <id> --status <list> --limit <n> requests matching results within the accessible history window.

  • Help for an unknown subcommand of a group is an error. testsprite test result --help, testsprite help test result and testsprite test help result printed the test group's help and exited 0, so a removed command looked like it ran. They exit 5 (VALIDATION_ERROR) and name the unknown command, at every level, as testsprite <typo> --help already did. So do testsprite test --help result, testsprite test -h result and testsprite --help config: a help flag before the name does not hide it.

  • ci run's skip notice keeps a third-party check's text on one ::notice line. A %, carriage return or line feed in that text is encoded, so GitHub shows it as sent and a line break in it can no longer start another workflow command.

  • A --local run whose tunnel is lost sends its cancel once. When the run had already finished (409), the cancel was sent again a second later, spending half the grace a second Ctrl-C allows.

  • A report gives each test that was left out its own reason. A positional test ID selected with --testlist <list-id> that is not a member of the list reads not a member of this list (not dispatched) also when a run --testlist <list-id> through a tunnel stops part-way, not test not found. A backend test that run --project <project-id> --local skips keeps backend-test as its reason in --summary-file and the JUnit report when no test was sent, instead of the no-runnable-tests reason.

  • A JUnit report that cannot be written no longer replaces the exit a waited batch owes. Under run --project <project-id> --wait, run <id> <id> … --wait and run --testlist <list-id> --wait, when a test failed, timed out or was not dispatched, a report that cannot be written (its directory removed during the run, say) is a line on stderr and the exit stands. With nothing else owed it is still the error, also on a batch with --allow-empty that dispatched nothing.

  • A batch whose first request is refused names the tests it never sent. When run <id> <id> … stops because its first request did not confirm a field it was sent (--target-url, --skip-dependencies), it prints [run] Dispatch stopped (…). N tests were not dispatched: … on stderr, stdout carries the partial result with their ids in notRunTestIds (--output json; in text mode stdout stays empty, as nothing was accepted), and the error's next action names them, as --local does; under --wait, --summary-file and the JUnit report list them as skipped. The stop line no longer starts its hints with an empty sentence when nothing had been accepted.

  • A . or .. id or name can no longer redirect a request. project env delete <project-id> .. --confirm resolved to DELETE /projects/<project-id>/ and deleted the whole project. Any path segment that is . or .. (also as %2E%2E) is now refused with VALIDATION_ERROR (exit 5) before a request is sent; project env also refuses . and .. as an environment name (create, get, update, --rename, delete, set-default, dry runs included) before any request. Names that merely contain dots (v1.2, ..staging) are unaffected. Thanks @RK-625 (#716).

  • A write sent without an idempotency key is no longer replayed after a 5xx. The server may have applied the write before it failed, so the automatic retry could repeat it: the tunnel mint behind tunnel start and --local could create a second tunnel. Such a request is now sent once and its error is reported; reads, requests carrying an Idempotency-Key, run cancel and tunnel delete still retry, and 429 and 409 answers are handled as before. A 408, 502 or 504 with a JSON body that is not a TestSprite error envelope (an API gateway's own timeout message) is now treated the same way. Thanks @crypticsaiyan (#624).

  • A connection that drops while a response is being read is retried. A reset or close after a successful response's headers had arrived was reported as a non-JSON response (INTERNAL, exit 1) and never retried. It is now a transport failure, retried like a failed connection (reads and requests carrying an Idempotency-Key); when the retries run out, or the request may not be retried, the error is UNAVAILABLE (exit 10) instead of INTERNAL (exit 1). Thanks @Davidson3556 (#608).

  • --wait paces its status reads when the server answers them at once. A long-poll read is re-issued immediately because the server is expected to hold it for up to 25 seconds; a server or proxy that answered without holding turned the wait into back-to-back requests, which could trip the rate limit and end a healthy wait with RATE_LIMITED. Reads that come back in under 2 seconds are now spaced 2 seconds apart, within the --timeout deadline. Thanks @cipoklean (#581).

  • An auth failure during a batch --wait carries its real error code. When polling a run in run --project <project-id>, run <id> <id> …, run --testlist <list-id> or ci run failed with an auth error, the JSON error envelope's error.code (and telemetry) said CLI_ERROR. It is now the run's own AUTH_REQUIRED, AUTH_INVALID or AUTH_FORBIDDEN; the exit code (3) and the message are unchanged. Thanks @tujack39 (#592).

  • An empty --filter is refused instead of selecting every test. ci run --filter "" and run --project <project-id> --filter "" (or --filter=, for example from an unset shell variable) were read as no filter and ran every test in the project. An explicitly empty --filter is now VALIDATION_ERROR (exit 5) before any request on ci run and run, also under --dry-run and on run <id> --filter "", which used to run the id; omit the flag for an unfiltered project launch. On run, --filter is project-only: use run --project <project-id> --filter <substring> without named test IDs or --testlist. Thanks @WUWTest (#660).

  • run --project <project-id> --wait interrupted while it checks in-flight runs prints the runs it already started. Ctrl-C during the step that reads the in-flight runs the batch might attach to (also on run <id> <id> … and ci run) exited with nothing on stdout, so the accepted ids were lost. stdout now includes the acknowledged run IDs and any results seen so far, and stderr names the run wait and run cancel commands for them; the signal's exit code (130/143) is unchanged. Thanks @walter-w-white (#565).

  • The --target-url reachability check can no longer wait indefinitely on DNS. The DNS lookup that precedes the HTTP probe had no time limit of its own, so a resolver that did not answer held the command before anything was dispatched. The lookup and the probe now share one 8-second budget: a lookup that does not finish in time is an [advisory], not a refusal, and an interrupt during the lookup stops the command without dispatching a run. Thanks @lxcario (#625).

  • The re-trigger hint quotes the environment name. When run <id> --wait attaches to a run already in flight, its [advisory] on stderr suggests testsprite run <id> --env <name> for a later run; a name with spaces was printed bare and could not be pasted. The name is now always single-quoted (--env 'My Staging'). Thanks @bagusardin25 (#713).

  • Every text run card carries the run's Dashboard: link. run, run --wait and run wait print Dashboard: <url> under each run on stdout when the server sent that run a link; it opens the run's page in the Portal, and JSON carries it as runs[].dashboardUrl. A saved list launched without --wait (run --testlist <list-id>) is sent no link for its runs, so that receipt prints none; run wait <run-id> prints it. 0.16 printed a single Dashboard: line on stderr holding the test-case link or, when the server sent none, the run's execution link under that label. A launch now returns one link per run, so there is no separate Execution: line and no dashboard / execution card line. Thanks @bagusardin25 (#712).

  • A failed backend run gets the same hint under --output json as in text. In JSON mode the stderr hint after a failed or blocked run --wait offered a download that has nothing for a backend run. It now matches text mode (testsprite result get <result-id>, without the --out <dir> offer for a backend test); when the test's type is not already known it comes from one extra GET /tests/{id} (5 seconds, no retries), made only for a failed or blocked run. Stdout and exit codes are unchanged. Thanks @Davidson3556 (#576).

  • run cancel exits 3 on a rejected key and 14 on a too-old CLI. These errors were recorded per test and the command exited 1 (4 when another id was not found). The summary is still printed, then the command exits with that error, and the JSON error envelope carries its code instead of CLI_ERROR. 3 outranks 14, and both outrank a result not found (4) and a failed request (1). Thanks @protonjls (#595).

  • --out <dir> creates missing parent directories. An explicit --out needed an existing parent and exited 5 otherwise, so the --out ./.testsprite/… commands in the verify skill, the README and DOCUMENTATION.md failed in a fresh project. result get --out now creates them once the result has been read, so a failed read still leaves nothing on disk; an --out at an existing file, or one that runs through a file, still exits 5 (#729, closes #654). Thanks @naufalfx805-source (#729).

  • run --testlist <list-id> --wait interrupted with Ctrl-C reports the runs it dispatched. An interrupt during the wait exited with nothing on stdout, so the ids were lost. stdout now carries the usual result object, with execution receipts in runs[] and the latest observed test outcomes in results[], and stderr names the run wait and run cancel commands for the unfinished runs; the signal's exit code (130/143) is unchanged. A list run through a tunnel keeps its existing interrupt handling. Thanks @luckyzd (#564).

  • schedule list checks --columns before the request. In text mode an unknown column was only noticed while rendering rows, so on an empty list the command printed No schedules. and exited 0. An unknown column is now VALIDATION_ERROR (exit 5) before any request, whatever the list holds; --output json does not use --columns and is unchanged. Thanks @grzdev (#580).

  • schedule update --env <name> --dry-run exits 0 instead of 7. The dry-run sample did not echo the requested environment, so the command's check that the environment pin was confirmed failed with UNSUPPORTED (exit 7). The sample now echoes --env as environment with environmentMode: "pinned", and --start / --end as startAt / endAt. The sample schedule that schedule list, schedule get and schedule update print under --dry-run also gains environment: null and environmentMode: "inherit". Thanks @grzdev (#584).

  • A credentials file can no longer change objects the CLI shares. A [__proto__] section in ~/.testsprite/credentials wrote its api_key/api_url onto every object in the process, so a profile without its own key picked that one up, and [constructor], [toString] and similar sections wrote onto built-ins. Such sections are now ignored, profile names that spell an inherited member (toString, hasOwnProperty) work as ordinary profiles, and --profile __proto__ or --profile constructor reads as no profile and is refused with exit 5 when setup would write it.

  • API requests no longer follow a redirect to another origin. A redirect from the configured API endpoint to a different scheme, host or port used to be followed with the x-api-key header attached. It is now refused before anything is sent there and ends as exit 10 (UNAVAILABLE) without a retry; same-origin redirects still work, up to 20 hops (#446). Thanks @Ololadestephen (#629).

  • run --testlist <list-id> now enforces the same report-flag rules as run. Orphan --report-file and --report-suite-name flags could dispatch every member of the list (billed), poll, and exit 0 without writing any JUnit report — the CI upload step then found no file or uploaded a stale one while the gate was green. These combinations are now rejected with exit 5 before anything is dispatched, through the shared validator. (#569) Thanks @zhengqiuyang (#723).

  • Several tests in one project share an idempotency key. Named tests and whole-project launches use one project request instead of one request per test. The request uses the supplied --idempotency-key, or the auto-minted key printed to stderr under --output json, --verbose or --debug; a selection spanning several projects derives a deterministic key per project. Keep the same key and request, including any borrowed --tunnel-client, when retrying. This key mapping does not guarantee that a changed request or a newly created tunnel can replay the original receipt (#492). Thanks @Shubhdeep12 (#494).

  • An empty or whitespace-only positional argument is refused on every command that takes one. testsprite project get "", run " ", a blank element in a run cancel, run wait or testlist add list and the like used to reach the API or the dry-run preview. They now exit 5 (VALIDATION_ERROR) before the command's action runs, live and under --dry-run, and the message names the argument, for example Argument `<test-id>` is invalid: cannot be empty or whitespace-only. for test get "" (a list argument is shown as <run-id...> or [test-id...]). Commands that already refused a blank positional keep exit 5 with the new message. agent install "" now exits 5 instead of dropping the blank and installing for the detected agents (or prompting on a terminal). The project's agent commands follow the same rule: plan "", reply "" and reject "" exit 5; plan and reject with no argument work as before (#368). Thanks @grzdev (#585).

  • run and run --testlist <list-id> check --report-file and --summary-file paths before a run is dispatched. A path with a missing parent directory, a directory as its target, or a trailing separator used to surface only after the runs had finished and been billed: a bad --report-file failed the command with exit 5 and the report was lost (a relative reports/ could instead be written to a file named reports), and a bad --summary-file only printed a could not write --summary-file warning and left no file. Both are now refused with exit 5 before anything is dispatched. ci run is unchanged (#455). Thanks @ronkenx9 (#594).

  • A stalled environment preflight read is reported as a timeout. When the 5-second preflight read before a --local run timed out, the CLI exited 1 with UNCAUGHT_EXCEPTION. It now exits 7 (REQUEST_TIMEOUT) and the message names the preflight read (#397). Thanks @tujack39 (#659).

  • --debug shows two errors setup used to swallow silently: a failed profile read while resolving the endpoint, and agent-install output that is not JSON. Each writes one [debug] line to stderr; without --debug nothing changes (part of #186). Thanks @naufalfx805-source (#354).

  • project get/project list validate the response shape. A malformed response now exits 1 with an INTERNAL "Response shape mismatch" error (request id and the first mismatched paths, never the body) instead of a raw TypeError, or a bad body printed as a success. Additive server fields and new project type values still pass through unchanged, and an omitted project nextToken counts as the last page. Thanks @jerrybate-sketch (#340).

  • ci run on deployment_status gates only an open pull request whose head is the deployed commit (#550). A redeploy of a merged or closed pull request's head commit, or an open pull request that merely contains the commit, is now skipped with a ::notice:: and exit 0 (no_pull_request) instead of cloning an environment and gating it. Thanks @PugarHuda (#562).

  • The deployment_status pull-request lookup honors --request-timeout and survives a non-JSON reply (#551). A GitHub API that never answers is exit 10 within the request budget instead of hanging the job, and a 200 with an HTML body (proxy, captive portal) is exit 10 with a clear message instead of an uncaught exception. 401/403 keep exit 5. Thanks @PugarHuda (#562).

  • Two workflows generated by ci init github no longer cancel each other. Every generated workflow used the concurrency group testsprite-<pull request number or deployment sha>, and groups are repository-wide, so two generated workflows in one repository (a second one written with --path, for another project) cancelled each other's jobs. The group now includes the workflow's file name, URI-escaped with uppercase letters escaped too (testsprite-testsprite.yml-…). Existing workflow files are not changed; edit them by hand or regenerate with ci init github --force (the old file is kept as .bak). Thanks @Ololadestephen (#655).

  • The workflow from ci init github skips Dependabot pull requests. Dependabot pull requests run without the repository's Actions secrets, so the gate started with no TESTSPRITE_API_KEY and failed. The generated if: now also requires github.actor != 'dependabot[bot]', next to the existing fork and draft checks. Existing workflow files are not changed. Thanks @Smile-Catpy (#619).

  • ci init github writes action versions that run on Node 24. The generated workflow now uses actions/checkout@v7, actions/setup-node@v7 and actions/upload-artifact@v7 in place of the @v4 releases, which still declare the Node 20 runtime that GitHub Actions flags with a deprecation warning on every job; the job's own node-version stays '22'. setup-node also gets package-manager-cache: false: the job installs only the CLI, and without it the step fails in an npm repository whose package.json names a packageManager but has no lockfile. Existing workflow files are not changed. Thanks @lidxgoesgiri (#715).

  • ci init github refuses to write the workflow through a symlink that leaves the repository. The workflow path was written without being inspected, so a symlinked .github, workflows directory or workflow file could put the file (and with --force its .bak) outside the directory the command runs in. A symlink on the path that resolves outside that directory, or cannot be resolved, is now exit 5 before any request; a symlink that stays inside is still followed. Thanks @ayush-that (#623).

  • Generated files converted to CRLF are no longer treated as edited. When Git's line-ending conversion turned a generated skill file or workflow into CRLF, agent install (and setup) reported it blocked, agent status reported modified, and ci init github exited 5 asking for --force. The comparison now ignores CRLF versus LF: agent install reports skipped, agent status reports ok, and ci init github treats the file as already written. Thanks @yeziR4 (#593).

  • Agent detection ignores the CLI's own numbered backups. agent install --force keeps earlier versions as .bak, .bak.1 … .bak.99, but only .bak was recognized as the CLI's own file, so after a second forced reinstall the numbered backups counted as evidence of that agent in setup and agent install. All of them are now ignored, like the skill files themselves. Thanks @Satianurag (#557).

  • The testsprite-ci skill's frontmatter is valid YAML. Its description contains on: deployment_status and on: pull_request, and every own-file target wrote it unquoted, so YAML readers took the : for a nested key and rejected the frontmatter: Kiro skipped the skill, and Cursor, Windsurf and Copilot could drop the whole block, including trigger and applyTo, while agent status still reported ok. A description that is not safe as a plain YAML value is now written double-quoted; the testsprite-verify and testsprite-onboard frontmatter is unchanged. A skill installed by an earlier CLI reads stale and is refreshed with agent install <target> --force, as described above. Thanks @Smile-Catpy (#617).

  • Arrow keys and pasted escape sequences no longer end up in a prompt answer. At an interactive prompt (for example the API key in setup), an arrow, Home, End or Delete key or a bracketed-paste marker lost only its ESC byte and left the rest ([D, [200~) in the answer, which silently corrupted the key. Whole escape sequences are now dropped, including one split across two input chunks. Thanks @cipoklean (#615).

  • setup's key check honors TESTSPRITE_REQUEST_TIMEOUT_MS. The request that verifies the API key before it is saved read only --request-timeout and otherwise waited the 120-second default. It now uses the same precedence as every other command: --request-timeout, then TESTSPRITE_REQUEST_TIMEOUT_MS, then 120 seconds. Thanks @Herbicide7665 (#626).

  • A setup key check that times out is reported as a timeout, not a rejected key. A client-side timeout while verifying the key printed API key rejected by <url> and exited 3 with error.code: "CLI_ERROR", which sent people to rotate a working key. It is now REQUEST_TIMEOUT (exit 7) with the line API key check against <url> timed out, profile NOT updated; a key the server rejects still exits 3. Thanks @Andy00L (#647).

  • TESTSPRITE_API_KEY plus an endpoint override no longer need a readable credentials file. With both TESTSPRITE_API_KEY and TESTSPRITE_API_URL (or --endpoint-url) set, every command still read ~/.testsprite/credentials and failed when the file was unreadable. The file is now read only when the key or the endpoint has to come from it. doctor still inspects the file: for an unreadable file its Credentials check is warn when both overrides are set and fail otherwise, including with only TESTSPRITE_API_KEY set, which used to be warn. Thanks @walter-w-white (#574).

  • A failing command keeps its exit code when the reader of its stdout closes early. With output piped into a reader that exits first (testsprite … | head), the broken-pipe handler exited 0 on the spot, before the command had set its own exit code, so a failed run or an error looked like success. Outside an active wait the command now finishes and exits with its own code (0 for a clean result); a pipe that closes during a wait is handled as before. Thanks @crypticsaiyan (#631).

  • Text tables stay aligned with CJK text, emoji and overlong values. Cells were padded by string length, so a full-width character (two terminal columns) or a value longer than its column pushed every later column out of line in project list, test list, result list and the other list tables. Columns are now measured in terminal display width and widen to fit their header and widest cell. Text output only: the spacing of a table can differ from 0.16.0 where a value or header was wider than its column, and JSON is unchanged. Thanks @OceanRanger (#616).

  • --report junit drops characters XML 1.0 forbids. A control character in a test name or failure message (NUL, 0x01–0x08, 0x0B, 0x0C, 0x0E–0x1F, U+FFFE, U+FFFF) was written into the report as is, and strict parsers then rejected the whole file. These characters are now removed; tab, newline and carriage return are kept. Thanks @tujack39 (#627).

  • --report junit keeps same-name tests distinct. Two list members sharing a human title (a multi-project list's Checkout from each project, or the same title twice within one project) were emitted with identical classname/name, and standard JUnit consumers such as GitLab — which key a testcase on those fields and ignore the extra testId/resultId attributes — collapsed the pair into one row, dropping one verdict and one run's diagnostics. Colliding rows now carry the stable logical test id in the standard name (Checkout [test_a]); unique names render exactly as before. (#515) Thanks @zhengqiuyang (#724).

  • setup --skip-if-configured fails when the saved key is rejected. With a saved key that had since been revoked or lost access, the 401/403 from the identity lookup was swallowed and setup printed an initialized summary with a placeholder env and empty scopes and exit 0, so a CI step passed with credentials that could not authenticate. It now exits 3 with the server's AUTH_INVALID / AUTH_FORBIDDEN error (message, nextAction and request id intact) and prints no summary. A valid saved key still succeeds without a prompt, and any other lookup failure, or a lookup after a freshly entered key, keeps the display-only fallback (#698). Thanks @dianchechihan (#721).

  • A network outage no longer ends a --wait immediately. Execution waits read through transport outages and retryable throttles until the shared --timeout deadline, including an outage on the first read. An outage still present at the deadline is exit 10; a throttle still present is a timeout (exit 7). An initial hard refusal, such as an unknown run or unavailable results route, retains its own error classification. A wait that ends without a terminal verdict reports the known execution IDs to follow with run wait or stop with run cancel; it does not launch the tests again. A command-owned local tunnel closes with the command, so following the execution cannot reopen it: use the printed tunnel recovery guidance (#437). Thanks @Andy00L (#599).

  • A preview behind Vercel Deployment Protection is refused before anything is dispatched. A protected preview answers every visitor without a Vercel session with a redirect to vercel.com/sso-api, and the --target-url reachability check counted that redirect as reachable, so ci run ran and billed the whole suite against Vercel's login page. The check now refuses it with exit 5 and details.reason: 'target-protected', and the message names the two ways past it: turn off Vercel Authentication for previews, or pass a Protection Bypass for Automation secret in the URL. The redirect's Location is only parsed, never fetched; every other redirect is still treated as reachable (#620). Thanks @Andy00L (#638).

  • ci run refuses a loopback target. A localhost or 127.0.0.1 target, from --target-url or from the selected environment, cannot be reached by the cloud runner, and ci run opens no tunnel. It is now refused with exit 5 before anything is dispatched, the step outputs are written as zeros so later workflow steps see a result, and the message points at testsprite run --project <project-id> --env <name> --wait for a local app (#549). Thanks @ayush-that (#653).

  • run --project <project-id> --wait reports a rejected key before rate-deferred tests. When a polled run failed with an auth error and other tests were deferred by the rate limit, the command exited 7 (deferred) and the rejected credential went unreported. It now exits 3 in that case; other failures still yield to the deferred exit (#471). Thanks @yeziR4 (#605).

  • An unreadable credentials file no longer ends every command. The telemetry hook reads the saved key before each command, and a credentials file that could not be read (EACCES / EPERM) ended the process there, before doctor could report the file or an offline command could run. The command now runs with telemetry off for that invocation, and --debug says why (#465). Thanks @Davidson3556 (#641).

  • A --*-file secret saved as UTF-16 is read correctly. Windows PowerShell 5.1 > and Set-Content -Encoding unicode write UTF-16LE with a BOM, and these files were decoded as UTF-8: project env create and project env update stored a garbled --password-file or --credential-file secret, and a valid --header-file or --auto-auth-file was refused as malformed. The file is now decoded by its BOM (UTF-8, UTF-16LE or UTF-16BE; a file without one is still read as UTF-8), and a UTF-16BE file with an odd byte count is VALIDATION_ERROR (exit 5). Thanks @Herbicide7665 (#628).

  • A UTF-16 credentials file is read, and the next save no longer deletes the key. PowerShell 5.1 > / Out-File, and Notepad's "Unicode" / "Unicode big endian", resave ~/.testsprite/credentials as UTF-16 with a BOM. Read as UTF-8, that file parsed as empty, so doctor reported no API key and the next setup or auth write replaced every saved profile. The shared BOM reader decodes it, and the next save writes UTF-8 without a BOM. Thanks @asosa933 (#680).

  • Run cancellation and tunnel cleanup retry a dropped connection. run cancel, the cancel and tunnel delete a --local or automatic-tunnel run sends when it stops, and the tunnel delete when tunnel start ends or tunnel stop runs were sent once and failed on a connection reset or a bare 502/504 from a proxy, which could leave the run going or the tunnel binding behind. They now retry those failures the way reads do (up to 4 attempts), since sending them twice is harmless. Thanks @kieugiathinhphat (#588).

  • --dry-run no longer announces a sample before a validation error. The [dry-run] sample response — not from the server banner printed as soon as the API client was built, so a command that then failed its own checks, such as test list --dry-run without --project, printed the banner and then the error (exit 5). The banner now prints with the first sample response, and previews that print without a request, such as run --dry-run and ci run --dry-run, still print it once. Thanks @Vinayak19112003 (#575).

  • Corporate-network failures name the real cause. An untrusted, expired or wrong-host TLS certificate and a proxy 407 were tried up to four times and reported as "Service is temporarily unavailable"; they now fail on the first attempt (exit 10) with an error that names the fix (NODE_EXTRA_CA_CERTS set to your organisation's CA bundle, or proxy credentials in HTTPS_PROXY) and the cause code in details.cause. A 4xx whose body is not JSON (a proxy, firewall or captive-portal page) is no longer reported as a rejected API key (exit 3), missing credits (exit 12), a missing resource (exit 4) or an outdated CLI (exit 14): it exits 10 as something between this machine and the API answering, with details.httpStatus and details.contentType. Thanks @OkeyAmy (#634).