Repository navigation
Writing tests now goes through the project's agent: plan asks it for tests and apply saves what it proposes; run executes them and result get shows one test's evidence. If the agent is not yet enabled for your account, the agent commands print The TestSprite agent is not enabled for this account yet. and exit 13; stay on 0.16.x until it is. For a command-by-command migration, see Upgrading from 0.16.
Breaking changes
- One launch entry:
testsprite run. The formertest run,test wait,
test cancelandtestlist runcommand paths are removed without aliases;
old invocations exit 5. Migrate project launches from
test run --all --project Ptorun --project P --wait, and named launches
fromtest run T1 T2torun T1 T2. Resume withrun wait R1 R2; cancel
withrun cancel R1 R2. Replacetestlist run Lwithrun --testlist L,
andtestlist run L --case T1 --case T2withrun T1 T2 --testlist L.
Rootrunrejects--alland--case. Update custom CI scripts and agent
guidance, and reinstall skills reported stale bytestsprite agent status.
Usetest listandtest getto inspect cases. Test-list management and
ci runretain their paths.
Added
-
A run card names its environment, and
apply --env <name>picks another. The conversation shows which environment a proposed run goes to (the one the agent named, else the project's default), andtestsprite apply --env stagingstarts it elsewhere, as the portal's card lets you pick one;applysays where the run started. Needs a backend that carries the card's environment; an older one shows the card without it. -
apply --answeranswers the agent's question card. When the agent asks one to four questions, each with options, the conversation shows them numbered with lettered options, andtestsprite apply --answer 1=b --answer 2=a,canswers them (your own words in quotes). An apply short of an answer is refused and the card keeps waiting. Needs a backend that carries the card's questions; an older one still shows the card as an action. -
The project's agent writes tests:
plan,apply,reply,reject,status,stopanddiscard.testsprite plan "<what to test>"asks the project's agent for tests (it explores the app, or reads--filedocuments and your--diff);applysaves what it proposed andrejectturns it down with what to change;replyanswers it in the same conversation,statusshows where it is,stopstops it anddiscarddrops its task. They replace the test-writing commands (see Removed). -
A backend environment's API sign-in is set on
project env.project env createandupdatetake it the way the project's agent writes it inenvironments/<name>.md:--auth none|bearer|basic|apikeywith--credential-file(or--credential) for a static credential,--header "Name: value"(repeatable) and--header-filefor the extra headers every request carries, and--auto-auth-filefor a sign-in that refreshes the credential. They work on any environment, not only the default, and are saved through the same server path as the Portal's environment card; the auto-auth file is sent as it is and the server refuses a field it does not know.project env get/listshow a backend environment as the agent reads it: its auth, whether each secret is set, and its headers with secret values hidden. Saving a sign-in needs a plan with advanced environment settings, as in the Portal. Which endpoint needs which header stays a Portal decision. -
setupkeeps.testsprite/out of Git. Inside a Git repository,testsprite setupappends.testsprite/(where failure artifacts are written) to the project's.gitignorewhen it is not already ignored; skip it with--no-gitignore, and--dry-runonly reports it. The entry is appended rather than rewritten, a symlinked.gitignoreis not written through, and a failure here is a warning that never fails setup (it also runs when the agent skill install fails after credentials were saved).testsprite doctorgainsGit repositoryandGitignore safetychecks: inside a repository it warns (never fails) when no.gitignorebetween the project directory and the repository root ignores.testsprite/, and outside a repository both are informational. Thanks to @sandman-sh (#257, closes #258). -
result listandresult getread test results. They replacetest result,test steps,test failure get,test failure summaryandtest artifact get(see Removed).testsprite result list --run <run-id>lists one run's results;testsprite result list --test <test-id>lists one test's results across runs, newest first (--status,--env,--limit 1-100; cancelled results only when--statusnames them).testsprite result get <result-id>reads one result: the verdict, the error, its cause and fix, the failed step with its screenshot, and every step in--output json.--out <dir>downloads it instead:result.json, the video,steps/001.png/steps/001.html…, andsign-in/for a result blocked at sign-in;--failed-onlykeeps the failed steps and one on each side, and every step when none failed (text mode says so on stderr). In an existing<dir>,result.json,video.*,steps/andsign-in/are replaced and nothing else is touched, so use one directory per result (.testsprite/results/<result-id>). A file that cannot be downloaded leaves the rest written and exits 10; when<dir>already holds a download of the same result, that file keeps its earlier copy (kept: trueinfailed, andresult.jsonnames it). Both commands exit 0 whatever the verdict. -
result getshows a backend test's output and traceback. Text prints the last 20 lines of its Python traceback (trace) and the first 20 of its captured output (apiOutput), each cut to 2,000 characters, with a line saying how many lines and characters were left out (above the traceback, below the output) and that--output jsonandresult.jsonhold them;--output jsonand--out <dir>'sresult.jsoncarry both whole. Both arenullfor a frontend test, and an older backend sends neither. -
ci rungates pull requests whose platform posts only a commit status — shape C. Some platforms create no GitHub deployment for a pull request (Vercel with itsdeployment_statusevents switched off, builds skipped by an Ignored Build Step, Cloudflare Pages) and leave only a status or check on the head commit, whose link is the platform's dashboard rather than the site; apull_requestworkflow then has no event carrying the preview URL and starts before the build finishes. Two newci runflags cover it.--wait-for-status <name>holds the run until the status context or check run of that name succeeds on the head commit — both GitHub APIs are polled and matched exactly, so which one the platform writes does not matter — for up to--status-timeout(default 900s, its own budget); a failed, errored, skipped or cancelled check, or a success whose description says Ignored / Canceled / Skipped, is a::notice::plus a one-line job summary, zero step outputs and exit 0 (nothing was tested), and a check still pending at the deadline is exit 7 naming every status and check seen on the commit.--target-url-template <pattern>derives the preview address from the pull request with the GitHub App's own placeholders ({pr},{pr-number},{branch},{branch-slug},{sha},{short-sha}; samebranch-slugrule, so a Portal trigger's pattern renders the same URL); a pattern the run cannot fill is refused with exit 5 naming the placeholder, never guessed, andci runnever falls back to the environment's URL or production; a placeholder in the host part must render to letters, digits and-(a raw{branch}namedattacker.example/xwould move the host, so it is refused the same way). The wait is bounded as a whole: each GitHub read gets the remaining budget as its own timeout, both lists are read to their last page, a round in which one endpoint did not answer decides nothing, and a rate-limited answer's cooldown is honored inside the deadline. Theci init githubtemplate's placeholder comment and thetestsprite-ciskill describe shape C and how to read the name and the pattern off a recent pull request withgh.
Removed
-
run --no-auto-heal. Whether a run heals drifted generated code is the server's decision for each run, and code a person wrote is never re-authored, so the opt-out is gone:runsends nothing about healing and prints no advisory. A script that still passes the flag exits 5 (unknown option). The launch'sautoHealin--output jsonstill says what the server decided. A 0.16 CLI that still passes the flag gets the strict replay it asked for, and its--output jsonlaunch carries adeprecationnotice saying the flag is gone. -
test code getandschedule run list. Inspect generated test code and a schedule's execution history in the Portal. Both commands, including their now-emptytest codeandschedule rungroups, are deleted without aliases or a deprecation period; old invocations exit 5 as unknown commands. Test reads, run/wait/cancel, and schedule CRUD remain available. -
test wait --max-concurrency. The former option is removed and is not supported byrun wait. Waits start one independent results-listing loop per distinct execution under one shared deadline. Set--max-concurrencyonrunlaunches only; it controls server dispatch per execution. -
Tests are written by the project's agent; the CLI no longer writes them. A coding agent runs tests, reads their results and fixes product code; new, changed and removed tests go through the agent (
testsprite plan "<what to test>", thentestsprite apply;testsprite reply/testsprite rejectto iterate). These commands are deleted, with no deprecation period; stay on 0.16.x if you still need one:Removed Use instead test create,test create-batch,test scaffold,test lint,test plan generate,test plan accepttestsprite plan "<what to test>", thentestsprite applytest update,test delete,test delete-batch,test code put,test plan puttestsprite plan "<test> should …"(orreplyin an open conversation), thentestsprite applyproject docs uploadattach the document to the request: testsprite plan --file <path>test rerunrun <id...>, orrun --project <id>for--all;runtakes no--statusor--skip-terminaltest flakyno replacement; run run <id>againtest openthe Dashboard:link thatrunprintsproject sign-in get,project sign-in setan existing environment's sign-in is set in the Portal; project env createstill takes the sign-in flagsproject credentialproject env create/updatewith--authand--credential-fileproject auto-authproject env update --auto-auth-file <json>;--authswitches it offproject create --url,--local,--username,--password,--password-file,--skip-preflightproject env create <id> --name default --url …(or--local <port>), which takes the sign-in flagsproject create --descriptionnothing: a project has no description project update --url,--local,--username,--password,--password-file,--skip-preflightproject env update <id> <name> …test list --status,--env,--created-fromnothing on test list: a case carries no result;result list --test <test-id> --limit 1reads the latest onetest run --all --statusrun <id…>with thecaseKeyof each resultresult list --run <run-id> --status failed,blockedlistsci doctor,ci connectnone; read deployment history with gh api repos/<owner>/<repo>/deploymentsand link the platform with its CLIinit,auth configure,auth whoami,auth logout(hidden aliases)setup,setup,auth status,auth removeThe first-project hints after
setup,project create, an emptytest listand an emptyrun --project <project-id>now point attestsprite plan, thentestsprite apply. Theschemas/plan.schema.jsonplan-file schema is no longer shipped.auth statusanddoctorno longer print the routing advisory (its one open gap wastest rerun).Skills installed by an earlier CLI name removed commands: run
testsprite agent status, thentestsprite agent install <target> --forcefor each target it reportsstale(own-file targets keep a.bak; the codex section is replaced by a plain re-install).testsprite doctorwarns while the verify skill is stale. -
result listandresult getreplace the six read commands. They are deleted with no alias; an old invocation is an unknown command (exit 5). Stay on 0.16.x if you still need one:Removed Use instead test result <test-id>result list --test <test-id> --limit 1, thenresult get <result-id>test result <test-id> --historyresult list --test <test-id>(--status,--env,--limit)test steps <test-id>(--run-id)result get <result-id>(itssteps)test failure get <test-id>result get <result-id>;--out <dir>downloads the evidence,--failed-onlynarrows ittest failure summary <test-id>result get <result-id>(cause,fix)test artifact get <run-id>result get <result-id> --out <dir>(the idtest artifact gettook is the result id)test diff <run-a> <run-b>none Against
test artifact get,result get --out <dir>has no default directory (it was./.testsprite/runs/<run-id>/), names step filessteps/001.png/steps/001.html(wassteps/01-screenshot.png/steps/01-snapshot.html), no longer writesmeta.json,failure.json,code.<ext>(inspect generated test code in the Portal),.partialor the per-stepNN-evidence.json, and keeps going when one file fails.Not carried over: the
--source,--sinceand--rerunfilters and the cursor paging oftest result --history(result list --testtakes--status,--envand--limit1–100 instead), and, fromtest resultand the failure bundle, the--include-analysisflag (result getalways includes the cause and fix, ascause/fix),failureKind,codeVersionandsnapshotId.The JSON fields, from the removed commands to
result get/result list:Before ( test result,test result --history,test failure get)Now status/verdict(lower case)testStatus, upper case:PASSED,FAILED,BLOCKED,RUNNING,CANCELLEDexecutionStatusprocessStatusrunIdIfAvailable; historyrunIdresultIdtestIdcaseKeytest failure get'sresult.*,failure.apiOutput,failure.tracethe top level of result getanalysis.rootCauseHypothesis,failure.rootCauseHypothesiscauseanalysis.recommendedFixTarget,failure.recommendedFixTarget{ kind, reference, rationale }fix, plain text (there is nofix.kind)failure.evidence[];steps[].screenshotUrl,steps[].htmlSnapshotUrleach step's screenshotSignedUrl/htmlSignedUrl(steps[],failedStep)failure get'scodeinspect generated test code in the Portal failedStepIndex(a number)failedStep.stepIndex(a string)history createdAtcreatedenvironment { id, name }envId(result get),environmentName(result list)startedAt,finishedAt,targetUrl,targetUrlSource,summary,failureAnalysisUrl; historysource,isRerun,createdFromgone; created/modifiedare the result's own record times, not the run's start and endhistory { runs[], nextCursor }{ results[], hiddenRunCount, retentionDays }, with no cursorexit 4 from test failure getwhen the test had no failing result, and fromtest artifact getwhen the result was not ready, not failing or cancelledresult getexits 0 whatever the verdict: readtestStatus -
Per-account platform routing. Every account now runs on the same platform, so
auth statusdrops itsrouting:line anddoctorits Routing check, and the org and workspace lines render whenever the backend reports the org wallet.run --wait,run waitandrun --project <project-id> --waitno longer fall back to the test's latest result (GET /tests/{id}/result) for a backend run whose row never finished, because backend run rows now finish; a row that does not finish times out (exit 7). Therun --project <project-id>advisories and help about frontend tests skipped by the old backend-only engine are gone. -
A removed command or option says where it went. A removed command is still an unknown command (exit 5), and its error adds one line naming what to use instead, or that there is none (in
--output json, the envelope'snextAction), for example`test create` was removed in 0.17.0; tests are written by the project's agent: testsprite plan "<what to test>", then testsprite apply. The removed options ofproject create,project updateandtest list, and thetest run,test rerunandtestlist runoptions thatrundoes not take (--all,--case,--status,--skip-terminal), do the same on theirunknown optionerror. The line names the option, never the value given with it, and takes the place of the(Did you mean …?)guess, which could name a similarly spelled command that is not the replacement (test lint→list).
Changed
-
test listandtest getreturn the test case, not its latest result. The object is the case as it is written:id,slug,title(wasname),description,type,category,priority,version { hash, label },stepCountand, onget,steps(wasplanSteps).status,createdFrom,statusByEnvironment,headlineEnvironmentanddetailsare gone, and with themtest list --status,--env,--created-fromandtest run --all --status, which selected by a result. Text mode lists ID / SLUG / TITLE / TYPE / CATEGORY / PRIORITY / VERSION / STEPS / UPDATED. The latest result of a case isresult list --test <test-id> --limit 1(its status istestStatus, upper case), andresult get <result-id>reads it. Needs a backend that answers CLI 0.17.0 with this shape. -
A project is created without an environment.
project createno longer takes a URL or a test account and no longer requires one for a frontend project; add where the app runs withproject env create, and the first environment becomes the project's default.project get/listno longer reporttargetUrlororiginMode(read the environment withproject env get), the hint afterproject createnames the environment step first, and--instructionis accepted on backend projects too. A secret handed to one of the removed flags is never echoed back in the unknown-option error, and neither is a URL that carries credentials (--url=https://user:password@host). -
usageshows the organization wallet only. The per-usercredits & planblock, its per-run cost lines and its~N runsestimate are gone: the backend's top-levelcredits/subPlan/creditsPerRunare a flattened copy of the same wallet (still passed through in--output json). -
Dashboard links are only the ones the server sends. The CLI no longer builds
https://www.testsprite.com/dashboard/tests/…links itself when a run, batch or project response carries nodashboardUrl; those old routes only redirect to the dashboard home. -
Portal links use the current routes. API-key, billing and credit-refusal hints point at
/dashboard-v3/settings/apikeyand/dashboard-v3/settings/billinginstead of the old/dashboard/settings/…pages. -
A run already in flight is attached only when its environment is verified. On a 409,
run --waitandrun --project <project-id> --waitno longer attach to an in-flight run whose read carries noenvironment(an older run shape); like anullenvironment, it is reported as a conflict (exit 6) withrun wait <run-id>to follow it. -
Update scripts that consume launch or wait JSON. A
runIdidentifies one execution. AresultIdidentifies one test's result in it. Use the two top-level arrays on everyrunselector and onrun wait:Read Use runs[].runIdResume or cancel that execution. A selection spanning projects can return multiple runs. results[].resultIdRead one test's verdict and evidence. Results can be empty immediately after launch. summaryon a waitCounts for this command. A completed failure exits 1. An incomplete timeout or interruption includes partial: true.runFailures[], when presentExecution-level diagnostics, including a cancellation or a timeout without a matching test result. --summary-file/ JUnitThe CI sidecar keeps per-test rows in results[]. JUnit testcases carryresultIdwhere available. -
run waitandrun canceltake execution IDs. Waiting follows the whole execution, including integration results added after the first tests finish. Cancellation stops all its queued and running work. A finished or already-cancelled execution can be cancelled again safely. Read the per-resultrefundfield for the outcome. A failed refund can be retried by cancelling the same run again. -
Project, named-test and list launches return any known test outcomes in
results[]. A list that refuses members reports their IDs and reasons. Project launches that lose the acknowledgement of a request report it underunconfirmedProjects. Use the returnedretryIdempotencyKeyto retry the same selection. -
A request's retry wait ends at its deadline. A 429
Retry-Afteror backoff wait inside the HTTP client now ends when the request's own deadline passes (--timeoutonrun waitand on each poll), as well as on Ctrl-C. -
Several
--localtests share one tunnel and start one run per project.run <id> <id> … --local <port>andrun --project <project-id> --local <port>start one run per request (per project, up to 5,000 named tests) through one tunnel, instead of one run per test.For project and named-test launches,
--max-concurrencycontrols dispatch per execution (default 5 with a tunnel, maximum 15). The default wait through that tunnel is 1,200 seconds. Saved test lists use their own limits. If the launch fails partway through, keep its run receipts and inspect any unconfirmed projects before retrying. -
Text output never carries a terminal escape from the server. In text mode, everything a command prints on stdout and every line it writes to stderr drop the escape sequences and C0/C1/bidi control characters that server text (titles, names, errors, URLs, the agent's replies) can carry; line breaks and tabs stay, and a carriage return becomes a line break. An escape sequence ends on its own line, so it never removes the lines the CLI wrote between two server values. An id, a name, a title or a URL inside a line or a table cell stays on that line, and so do an error's message and request id, a
--verboseline, the tunnel client's log lines and a prompt that names a server value (the password prompt ofapply). Thecompletionscript is printed as it is, on a terminal and on a pipe.--output jsonon stdout and stderr keeps the server's text and writes DEL, C1 and the bidi controls as\uXXXX: the same JSON value, with no 8-bit CSI or OSC byte and no hidden reordering; a file written with--outis plain JSON. -
testsprite test --helpdescribes the group as "List and inspect tests", and the line printed on Ctrl-C points attestsprite run wait <run-id>ortestsprite result list --run <run-id>, or, with no run id printed yet,testsprite result list --test <test-id>, instead oftestsprite test list. -
run cancel --helpno longer promises a refund, nor does the line a--localrun prints when the tunnel it borrowed is no longer registered and its run was cancelled. For a frontend run cancelled before a terminal verdict the server attempts one, and each result'srefundsays what happened. -
A batch
--waitreports its runs in the order they were dispatched, not the order their polls finished: text rows, JSON receipts, the JUnit report,--summary-fileand the Resume/Cancel line, so two runs of the same batch print in the same order.
Fixed
-
Interrupted runs and result history retain the latest observed verdicts.
result list --test <id> --status <list> --limit <n>requests matching results within the accessible history window. -
Help for an unknown subcommand of a group is an error.
testsprite test result --help,testsprite help test resultandtestsprite test help resultprinted thetestgroup's help and exited 0, so a removed command looked like it ran. They exit 5 (VALIDATION_ERROR) and name the unknown command, at every level, astestsprite <typo> --helpalready did. So dotestsprite test --help result,testsprite test -h resultandtestsprite --help config: a help flag before the name does not hide it. -
ci run's skip notice keeps a third-party check's text on one::noticeline. A%, carriage return or line feed in that text is encoded, so GitHub shows it as sent and a line break in it can no longer start another workflow command. -
A
--localrun whose tunnel is lost sends its cancel once. When the run had already finished (409), the cancel was sent again a second later, spending half the grace a second Ctrl-C allows. -
A report gives each test that was left out its own reason. A positional test ID selected with
--testlist <list-id>that is not a member of the list readsnot a member of this list (not dispatched)also when arun --testlist <list-id>through a tunnel stops part-way, nottest not found. A backend test thatrun --project <project-id> --localskips keepsbackend-testas its reason in--summary-fileand the JUnit report when no test was sent, instead of the no-runnable-tests reason. -
A JUnit report that cannot be written no longer replaces the exit a waited batch owes. Under
run --project <project-id> --wait,run <id> <id> … --waitandrun --testlist <list-id> --wait, when a test failed, timed out or was not dispatched, a report that cannot be written (its directory removed during the run, say) is a line on stderr and the exit stands. With nothing else owed it is still the error, also on a batch with--allow-emptythat dispatched nothing. -
A batch whose first request is refused names the tests it never sent. When
run <id> <id> …stops because its first request did not confirm a field it was sent (--target-url,--skip-dependencies), it prints[run] Dispatch stopped (…). N tests were not dispatched: …on stderr, stdout carries the partial result with their ids innotRunTestIds(--output json; in text mode stdout stays empty, as nothing was accepted), and the error's next action names them, as--localdoes; under--wait,--summary-fileand the JUnit report list them as skipped. The stop line no longer starts its hints with an empty sentence when nothing had been accepted. -
A
.or..id or name can no longer redirect a request.project env delete <project-id> .. --confirmresolved toDELETE /projects/<project-id>/and deleted the whole project. Any path segment that is.or..(also as%2E%2E) is now refused withVALIDATION_ERROR(exit 5) before a request is sent;project envalso refuses.and..as an environment name (create, get, update,--rename, delete, set-default, dry runs included) before any request. Names that merely contain dots (v1.2,..staging) are unaffected. Thanks @RK-625 (#716). -
A write sent without an idempotency key is no longer replayed after a 5xx. The server may have applied the write before it failed, so the automatic retry could repeat it: the tunnel mint behind
tunnel startand--localcould create a second tunnel. Such a request is now sent once and its error is reported; reads, requests carrying anIdempotency-Key, run cancel and tunnel delete still retry, and 429 and 409 answers are handled as before. A 408, 502 or 504 with a JSON body that is not a TestSprite error envelope (an API gateway's own timeout message) is now treated the same way. Thanks @crypticsaiyan (#624). -
A connection that drops while a response is being read is retried. A reset or close after a successful response's headers had arrived was reported as a non-JSON response (
INTERNAL, exit 1) and never retried. It is now a transport failure, retried like a failed connection (reads and requests carrying anIdempotency-Key); when the retries run out, or the request may not be retried, the error isUNAVAILABLE(exit 10) instead ofINTERNAL(exit 1). Thanks @Davidson3556 (#608). -
--waitpaces its status reads when the server answers them at once. A long-poll read is re-issued immediately because the server is expected to hold it for up to 25 seconds; a server or proxy that answered without holding turned the wait into back-to-back requests, which could trip the rate limit and end a healthy wait withRATE_LIMITED. Reads that come back in under 2 seconds are now spaced 2 seconds apart, within the--timeoutdeadline. Thanks @cipoklean (#581). -
An auth failure during a batch
--waitcarries its real error code. When polling a run inrun --project <project-id>,run <id> <id> …,run --testlist <list-id>orci runfailed with an auth error, the JSON error envelope'serror.code(and telemetry) saidCLI_ERROR. It is now the run's ownAUTH_REQUIRED,AUTH_INVALIDorAUTH_FORBIDDEN; the exit code (3) and the message are unchanged. Thanks @tujack39 (#592). -
An empty
--filteris refused instead of selecting every test.ci run --filter ""andrun --project <project-id> --filter ""(or--filter=, for example from an unset shell variable) were read as no filter and ran every test in the project. An explicitly empty--filteris nowVALIDATION_ERROR(exit 5) before any request onci runandrun, also under--dry-runand onrun <id> --filter "", which used to run the id; omit the flag for an unfiltered project launch. Onrun,--filteris project-only: userun --project <project-id> --filter <substring>without named test IDs or--testlist. Thanks @WUWTest (#660). -
run --project <project-id> --waitinterrupted while it checks in-flight runs prints the runs it already started. Ctrl-C during the step that reads the in-flight runs the batch might attach to (also onrun <id> <id> …andci run) exited with nothing on stdout, so the accepted ids were lost. stdout now includes the acknowledged run IDs and any results seen so far, and stderr names therun waitandrun cancelcommands for them; the signal's exit code (130/143) is unchanged. Thanks @walter-w-white (#565). -
The
--target-urlreachability check can no longer wait indefinitely on DNS. The DNS lookup that precedes the HTTP probe had no time limit of its own, so a resolver that did not answer held the command before anything was dispatched. The lookup and the probe now share one 8-second budget: a lookup that does not finish in time is an[advisory], not a refusal, and an interrupt during the lookup stops the command without dispatching a run. Thanks @lxcario (#625). -
The re-trigger hint quotes the environment name. When
run <id> --waitattaches to a run already in flight, its[advisory]on stderr suggeststestsprite run <id> --env <name>for a later run; a name with spaces was printed bare and could not be pasted. The name is now always single-quoted (--env 'My Staging'). Thanks @bagusardin25 (#713). -
Every text run card carries the run's
Dashboard:link.run,run --waitandrun waitprintDashboard: <url>under each run on stdout when the server sent that run a link; it opens the run's page in the Portal, and JSON carries it asruns[].dashboardUrl. A saved list launched without--wait(run --testlist <list-id>) is sent no link for its runs, so that receipt prints none;run wait <run-id>prints it. 0.16 printed a singleDashboard:line on stderr holding the test-case link or, when the server sent none, the run's execution link under that label. A launch now returns one link per run, so there is no separateExecution:line and nodashboard/executioncard line. Thanks @bagusardin25 (#712). -
A failed backend run gets the same hint under
--output jsonas in text. In JSON mode the stderr hint after a failed or blockedrun --waitoffered a download that has nothing for a backend run. It now matches text mode (testsprite result get <result-id>, without the--out <dir>offer for a backend test); when the test's type is not already known it comes from one extraGET /tests/{id}(5 seconds, no retries), made only for a failed or blocked run. Stdout and exit codes are unchanged. Thanks @Davidson3556 (#576). -
run cancelexits 3 on a rejected key and 14 on a too-old CLI. These errors were recorded per test and the command exited 1 (4 when another id was not found). The summary is still printed, then the command exits with that error, and the JSON error envelope carries its code instead ofCLI_ERROR. 3 outranks 14, and both outrank a result not found (4) and a failed request (1). Thanks @protonjls (#595). -
--out <dir>creates missing parent directories. An explicit--outneeded an existing parent and exited 5 otherwise, so the--out ./.testsprite/…commands in the verify skill, the README and DOCUMENTATION.md failed in a fresh project.result get --outnow creates them once the result has been read, so a failed read still leaves nothing on disk; an--outat an existing file, or one that runs through a file, still exits 5 (#729, closes #654). Thanks @naufalfx805-source (#729). -
run --testlist <list-id> --waitinterrupted with Ctrl-C reports the runs it dispatched. An interrupt during the wait exited with nothing on stdout, so the ids were lost. stdout now carries the usual result object, with execution receipts inruns[]and the latest observed test outcomes inresults[], and stderr names therun waitandrun cancelcommands for the unfinished runs; the signal's exit code (130/143) is unchanged. A list run through a tunnel keeps its existing interrupt handling. Thanks @luckyzd (#564). -
schedule listchecks--columnsbefore the request. In text mode an unknown column was only noticed while rendering rows, so on an empty list the command printedNo schedules.and exited 0. An unknown column is nowVALIDATION_ERROR(exit 5) before any request, whatever the list holds;--output jsondoes not use--columnsand is unchanged. Thanks @grzdev (#580). -
schedule update --env <name> --dry-runexits 0 instead of 7. The dry-run sample did not echo the requested environment, so the command's check that the environment pin was confirmed failed withUNSUPPORTED(exit 7). The sample now echoes--envasenvironmentwithenvironmentMode: "pinned", and--start/--endasstartAt/endAt. The sample schedule thatschedule list,schedule getandschedule updateprint under--dry-runalso gainsenvironment: nullandenvironmentMode: "inherit". Thanks @grzdev (#584). -
A credentials file can no longer change objects the CLI shares. A
[__proto__]section in~/.testsprite/credentialswrote itsapi_key/api_urlonto every object in the process, so a profile without its own key picked that one up, and[constructor],[toString]and similar sections wrote onto built-ins. Such sections are now ignored, profile names that spell an inherited member (toString,hasOwnProperty) work as ordinary profiles, and--profile __proto__or--profile constructorreads as no profile and is refused with exit 5 whensetupwould write it. -
API requests no longer follow a redirect to another origin. A redirect from the configured API endpoint to a different scheme, host or port used to be followed with the
x-api-keyheader attached. It is now refused before anything is sent there and ends as exit 10 (UNAVAILABLE) without a retry; same-origin redirects still work, up to 20 hops (#446). Thanks @Ololadestephen (#629). -
run --testlist <list-id>now enforces the same report-flag rules asrun. Orphan--report-fileand--report-suite-nameflags could dispatch every member of the list (billed), poll, and exit 0 without writing any JUnit report — the CI upload step then found no file or uploaded a stale one while the gate was green. These combinations are now rejected with exit 5 before anything is dispatched, through the shared validator. (#569) Thanks @zhengqiuyang (#723). -
Several tests in one project share an idempotency key. Named tests and whole-project launches use one project request instead of one request per test. The request uses the supplied
--idempotency-key, or the auto-minted key printed to stderr under--output json,--verboseor--debug; a selection spanning several projects derives a deterministic key per project. Keep the same key and request, including any borrowed--tunnel-client, when retrying. This key mapping does not guarantee that a changed request or a newly created tunnel can replay the original receipt (#492). Thanks @Shubhdeep12 (#494). -
An empty or whitespace-only positional argument is refused on every command that takes one.
testsprite project get "",run " ", a blank element in arun cancel,run waitortestlist addlist and the like used to reach the API or the dry-run preview. They now exit 5 (VALIDATION_ERROR) before the command's action runs, live and under--dry-run, and the message names the argument, for exampleArgument `<test-id>` is invalid: cannot be empty or whitespace-only.fortest get ""(a list argument is shown as<run-id...>or[test-id...]). Commands that already refused a blank positional keep exit 5 with the new message.agent install ""now exits 5 instead of dropping the blank and installing for the detected agents (or prompting on a terminal). The project's agent commands follow the same rule:plan "",reply ""andreject ""exit 5;planandrejectwith no argument work as before (#368). Thanks @grzdev (#585). -
runandrun --testlist <list-id>check--report-fileand--summary-filepaths before a run is dispatched. A path with a missing parent directory, a directory as its target, or a trailing separator used to surface only after the runs had finished and been billed: a bad--report-filefailed the command with exit 5 and the report was lost (a relativereports/could instead be written to a file namedreports), and a bad--summary-fileonly printed acould not write --summary-filewarning and left no file. Both are now refused with exit 5 before anything is dispatched.ci runis unchanged (#455). Thanks @ronkenx9 (#594). -
A stalled environment preflight read is reported as a timeout. When the 5-second preflight read before a
--localrun timed out, the CLI exited 1 withUNCAUGHT_EXCEPTION. It now exits 7 (REQUEST_TIMEOUT) and the message names the preflight read (#397). Thanks @tujack39 (#659). -
--debugshows two errorssetupused to swallow silently: a failed profile read while resolving the endpoint, and agent-install output that is not JSON. Each writes one[debug]line to stderr; without--debugnothing changes (part of #186). Thanks @naufalfx805-source (#354). -
project get/project listvalidate the response shape. A malformed response now exits 1 with anINTERNAL"Response shape mismatch" error (request id and the first mismatched paths, never the body) instead of a rawTypeError, or a bad body printed as a success. Additive server fields and new project type values still pass through unchanged, and an omitted projectnextTokencounts as the last page. Thanks @jerrybate-sketch (#340). -
ci runondeployment_statusgates only an open pull request whose head is the deployed commit (#550). A redeploy of a merged or closed pull request's head commit, or an open pull request that merely contains the commit, is now skipped with a::notice::and exit 0 (no_pull_request) instead of cloning an environment and gating it. Thanks @PugarHuda (#562). -
The
deployment_statuspull-request lookup honors--request-timeoutand survives a non-JSON reply (#551). A GitHub API that never answers is exit 10 within the request budget instead of hanging the job, and a 200 with an HTML body (proxy, captive portal) is exit 10 with a clear message instead of an uncaught exception. 401/403 keep exit 5. Thanks @PugarHuda (#562). -
Two workflows generated by
ci init githubno longer cancel each other. Every generated workflow used the concurrency grouptestsprite-<pull request number or deployment sha>, and groups are repository-wide, so two generated workflows in one repository (a second one written with--path, for another project) cancelled each other's jobs. The group now includes the workflow's file name, URI-escaped with uppercase letters escaped too (testsprite-testsprite.yml-…). Existing workflow files are not changed; edit them by hand or regenerate withci init github --force(the old file is kept as.bak). Thanks @Ololadestephen (#655). -
The workflow from
ci init githubskips Dependabot pull requests. Dependabot pull requests run without the repository's Actions secrets, so the gate started with noTESTSPRITE_API_KEYand failed. The generatedif:now also requiresgithub.actor != 'dependabot[bot]', next to the existing fork and draft checks. Existing workflow files are not changed. Thanks @Smile-Catpy (#619). -
ci init githubwrites action versions that run on Node 24. The generated workflow now usesactions/checkout@v7,actions/setup-node@v7andactions/upload-artifact@v7in place of the@v4releases, which still declare the Node 20 runtime that GitHub Actions flags with a deprecation warning on every job; the job's ownnode-versionstays'22'.setup-nodealso getspackage-manager-cache: false: the job installs only the CLI, and without it the step fails in an npm repository whosepackage.jsonnames apackageManagerbut has no lockfile. Existing workflow files are not changed. Thanks @lidxgoesgiri (#715). -
ci init githubrefuses to write the workflow through a symlink that leaves the repository. The workflow path was written without being inspected, so a symlinked.github,workflowsdirectory or workflow file could put the file (and with--forceits.bak) outside the directory the command runs in. A symlink on the path that resolves outside that directory, or cannot be resolved, is now exit 5 before any request; a symlink that stays inside is still followed. Thanks @ayush-that (#623). -
Generated files converted to CRLF are no longer treated as edited. When Git's line-ending conversion turned a generated skill file or workflow into CRLF,
agent install(andsetup) reported itblocked,agent statusreportedmodified, andci init githubexited 5 asking for--force. The comparison now ignores CRLF versus LF:agent installreportsskipped,agent statusreportsok, andci init githubtreats the file as already written. Thanks @yeziR4 (#593). -
Agent detection ignores the CLI's own numbered backups.
agent install --forcekeeps earlier versions as.bak,.bak.1….bak.99, but only.bakwas recognized as the CLI's own file, so after a second forced reinstall the numbered backups counted as evidence of that agent insetupandagent install. All of them are now ignored, like the skill files themselves. Thanks @Satianurag (#557). -
The
testsprite-ciskill's frontmatter is valid YAML. Its description containson: deployment_statusandon: pull_request, and every own-file target wrote it unquoted, so YAML readers took the:for a nested key and rejected the frontmatter: Kiro skipped the skill, and Cursor, Windsurf and Copilot could drop the whole block, includingtriggerandapplyTo, whileagent statusstill reportedok. A description that is not safe as a plain YAML value is now written double-quoted; thetestsprite-verifyandtestsprite-onboardfrontmatter is unchanged. A skill installed by an earlier CLI readsstaleand is refreshed withagent install <target> --force, as described above. Thanks @Smile-Catpy (#617). -
Arrow keys and pasted escape sequences no longer end up in a prompt answer. At an interactive prompt (for example the API key in
setup), an arrow, Home, End or Delete key or a bracketed-paste marker lost only its ESC byte and left the rest ([D,[200~) in the answer, which silently corrupted the key. Whole escape sequences are now dropped, including one split across two input chunks. Thanks @cipoklean (#615). -
setup's key check honorsTESTSPRITE_REQUEST_TIMEOUT_MS. The request that verifies the API key before it is saved read only--request-timeoutand otherwise waited the 120-second default. It now uses the same precedence as every other command:--request-timeout, thenTESTSPRITE_REQUEST_TIMEOUT_MS, then 120 seconds. Thanks @Herbicide7665 (#626). -
A
setupkey check that times out is reported as a timeout, not a rejected key. A client-side timeout while verifying the key printedAPI key rejected by <url>and exited 3 witherror.code: "CLI_ERROR", which sent people to rotate a working key. It is nowREQUEST_TIMEOUT(exit 7) with the lineAPI key check against <url> timed out, profile NOT updated; a key the server rejects still exits 3. Thanks @Andy00L (#647). -
TESTSPRITE_API_KEYplus an endpoint override no longer need a readable credentials file. With bothTESTSPRITE_API_KEYandTESTSPRITE_API_URL(or--endpoint-url) set, every command still read~/.testsprite/credentialsand failed when the file was unreadable. The file is now read only when the key or the endpoint has to come from it.doctorstill inspects the file: for an unreadable file itsCredentialscheck iswarnwhen both overrides are set andfailotherwise, including with onlyTESTSPRITE_API_KEYset, which used to bewarn. Thanks @walter-w-white (#574). -
A failing command keeps its exit code when the reader of its stdout closes early. With output piped into a reader that exits first (
testsprite … | head), the broken-pipe handler exited 0 on the spot, before the command had set its own exit code, so a failed run or an error looked like success. Outside an active wait the command now finishes and exits with its own code (0 for a clean result); a pipe that closes during a wait is handled as before. Thanks @crypticsaiyan (#631). -
Text tables stay aligned with CJK text, emoji and overlong values. Cells were padded by string length, so a full-width character (two terminal columns) or a value longer than its column pushed every later column out of line in
project list,test list,result listand the other list tables. Columns are now measured in terminal display width and widen to fit their header and widest cell. Text output only: the spacing of a table can differ from 0.16.0 where a value or header was wider than its column, and JSON is unchanged. Thanks @OceanRanger (#616). -
--report junitdrops characters XML 1.0 forbids. A control character in a test name or failure message (NUL,0x01–0x08,0x0B,0x0C,0x0E–0x1F,U+FFFE,U+FFFF) was written into the report as is, and strict parsers then rejected the whole file. These characters are now removed; tab, newline and carriage return are kept. Thanks @tujack39 (#627). -
--report junitkeeps same-name tests distinct. Two list members sharing a human title (a multi-project list'sCheckoutfrom each project, or the same title twice within one project) were emitted with identicalclassname/name, and standard JUnit consumers such as GitLab — which key a testcase on those fields and ignore the extratestId/resultIdattributes — collapsed the pair into one row, dropping one verdict and one run's diagnostics. Colliding rows now carry the stable logical test id in the standardname(Checkout [test_a]); unique names render exactly as before. (#515) Thanks @zhengqiuyang (#724). -
setup --skip-if-configuredfails when the saved key is rejected. With a saved key that had since been revoked or lost access, the 401/403 from the identity lookup was swallowed and setup printed aninitializedsummary with a placeholderenvand emptyscopesand exit 0, so a CI step passed with credentials that could not authenticate. It now exits 3 with the server'sAUTH_INVALID/AUTH_FORBIDDENerror (message,nextActionand request id intact) and prints no summary. A valid saved key still succeeds without a prompt, and any other lookup failure, or a lookup after a freshly entered key, keeps the display-only fallback (#698). Thanks @dianchechihan (#721). -
A network outage no longer ends a
--waitimmediately. Execution waits read through transport outages and retryable throttles until the shared--timeoutdeadline, including an outage on the first read. An outage still present at the deadline is exit 10; a throttle still present is a timeout (exit 7). An initial hard refusal, such as an unknown run or unavailable results route, retains its own error classification. A wait that ends without a terminal verdict reports the known execution IDs to follow withrun waitor stop withrun cancel; it does not launch the tests again. A command-owned local tunnel closes with the command, so following the execution cannot reopen it: use the printed tunnel recovery guidance (#437). Thanks @Andy00L (#599). -
A preview behind Vercel Deployment Protection is refused before anything is dispatched. A protected preview answers every visitor without a Vercel session with a redirect to
vercel.com/sso-api, and the--target-urlreachability check counted that redirect as reachable, soci runran and billed the whole suite against Vercel's login page. The check now refuses it with exit 5 anddetails.reason: 'target-protected', and the message names the two ways past it: turn off Vercel Authentication for previews, or pass a Protection Bypass for Automation secret in the URL. The redirect'sLocationis only parsed, never fetched; every other redirect is still treated as reachable (#620). Thanks @Andy00L (#638). -
ci runrefuses a loopback target. Alocalhostor127.0.0.1target, from--target-urlor from the selected environment, cannot be reached by the cloud runner, andci runopens no tunnel. It is now refused with exit 5 before anything is dispatched, the step outputs are written as zeros so later workflow steps see a result, and the message points attestsprite run --project <project-id> --env <name> --waitfor a local app (#549). Thanks @ayush-that (#653). -
run --project <project-id> --waitreports a rejected key before rate-deferred tests. When a polled run failed with an auth error and other tests were deferred by the rate limit, the command exited 7 (deferred) and the rejected credential went unreported. It now exits 3 in that case; other failures still yield to the deferred exit (#471). Thanks @yeziR4 (#605). -
An unreadable credentials file no longer ends every command. The telemetry hook reads the saved key before each command, and a credentials file that could not be read (
EACCES/EPERM) ended the process there, beforedoctorcould report the file or an offline command could run. The command now runs with telemetry off for that invocation, and--debugsays why (#465). Thanks @Davidson3556 (#641). -
A
--*-filesecret saved as UTF-16 is read correctly. Windows PowerShell 5.1>andSet-Content -Encoding unicodewrite UTF-16LE with a BOM, and these files were decoded as UTF-8:project env createandproject env updatestored a garbled--password-fileor--credential-filesecret, and a valid--header-fileor--auto-auth-filewas refused as malformed. The file is now decoded by its BOM (UTF-8, UTF-16LE or UTF-16BE; a file without one is still read as UTF-8), and a UTF-16BE file with an odd byte count isVALIDATION_ERROR(exit 5). Thanks @Herbicide7665 (#628). -
A UTF-16 credentials file is read, and the next save no longer deletes the key. PowerShell 5.1
>/Out-File, and Notepad's "Unicode" / "Unicode big endian", resave~/.testsprite/credentialsas UTF-16 with a BOM. Read as UTF-8, that file parsed as empty, sodoctorreported no API key and the nextsetuporauthwrite replaced every saved profile. The shared BOM reader decodes it, and the next save writes UTF-8 without a BOM. Thanks @asosa933 (#680). -
Run cancellation and tunnel cleanup retry a dropped connection.
run cancel, the cancel and tunnel delete a--localor automatic-tunnel run sends when it stops, and the tunnel delete whentunnel startends ortunnel stopruns were sent once and failed on a connection reset or a bare 502/504 from a proxy, which could leave the run going or the tunnel binding behind. They now retry those failures the way reads do (up to 4 attempts), since sending them twice is harmless. Thanks @kieugiathinhphat (#588). -
--dry-runno longer announces a sample before a validation error. The[dry-run] sample response — not from the serverbanner printed as soon as the API client was built, so a command that then failed its own checks, such astest list --dry-runwithout--project, printed the banner and then the error (exit 5). The banner now prints with the first sample response, and previews that print without a request, such asrun --dry-runandci run --dry-run, still print it once. Thanks @Vinayak19112003 (#575). -
Corporate-network failures name the real cause. An untrusted, expired or wrong-host TLS certificate and a proxy
407were tried up to four times and reported as "Service is temporarily unavailable"; they now fail on the first attempt (exit 10) with an error that names the fix (NODE_EXTRA_CA_CERTSset to your organisation's CA bundle, or proxy credentials inHTTPS_PROXY) and the cause code indetails.cause. A 4xx whose body is not JSON (a proxy, firewall or captive-portal page) is no longer reported as a rejected API key (exit 3), missing credits (exit 12), a missing resource (exit 4) or an outdated CLI (exit 14): it exits 10 as something between this machine and the API answering, withdetails.httpStatusanddetails.contentType. Thanks @OkeyAmy (#634).