Skip to content

v0.26.1

Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 27 Oct 04:15
· 3 commits to main since this release
v0.26.1
26716d5
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Security

  • Secret text is masked more thoroughly in warnings, run output and recordings. See the repository security advisories for details.

Added

  • MCP: start_tests, test_status, test_result and cancel_tests run tests in the background. run_tests blocks until probe test exits, so a multi-device run (--parallel, composite_devices, several devices) could outlast the MCP client's tool timeout and be reported as failed while it kept running. start_tests takes the same arguments, returns a job_id at once and runs the command in its own process group; test_status returns the state, elapsed time and the last lines of output; test_result returns the retained output (the last 1 MB) and the exit code; cancel_tests sends SIGTERM to the run and kills it after 5 seconds. Jobs live in the probe-mcp process and are killed when it exits. The hosted-upload flags are refused here exactly as in run_tests. run_tests is unchanged. The MCP server now has 40 tools.
  • probe lint checks the references a run would resolve. Lint reports each step and recipe call that matches no built-in step and no recipe in the file, its use statements or the recipes folder (the check probe test --dry-run makes), each selector that can never match, and each <name> placeholder that no example column, recipe parameter or stored value defines. It runs without a device, gives the line of each finding, and checks hook bodies and the device blocks of composite tests too.
  • Recordings show the element each step acted on. A step that taps, double taps, long presses, types, clears, toggles, or swipes or scrolls with a selector records its selector, and its bounds and tap point when the element is on screen as the step starts. The viewer shows them under "Element". Text typed into a field is never recorded, and a failed lookup is an advisory warning that does not change the step.
  • probe test --record writes a step-by-step recording of each test. Each test attempt includes source files and lines, screenshots before and after steps, widget trees after steps, and the app's dart:io HTTP exchanges with credential redaction, plus an offline viewer. Bundles stay under reports_folder/recordings or --record-dir; nothing uploads, and recording warnings do not change test results or exit codes. Native SDK and WebView traffic and screenshot redaction rules are not yet recorded; screenshots are not redacted and widget trees are not captured before steps. Keep recordings out of version control because they can contain screens and request or response bodies.
  • Recording manifests name the platform and OS version. manifest.json in each recording bundle now carries the run's platform (ios or android) and os_version, the same values the report shows, instead of leaving both empty.
  • Studio's AI chat can send the last run and the device's widget tree, but only when the workspace turns that on (off by default). Workspace settings have an AI chat context section with two switches, stored in probe.yaml under studio.ai_context. When either is on, a line above the chat input says what is sent. The widget tree is redacted first: obscured fields and credential-looking text are removed, and each part is cut to a size limit. A new backend call, SuggestSelectorFix, asks the model for one ProbeScript line that replaces the last failed step. Nothing is applied, and the pane button is a follow-up.
  • Probe Cloud (preview): local runs, probe test --upload-local. A run from a local simulator, emulator or device can be sent as a local run. It is opt-in for each run and never implied by --upload-report. The envelope carries the repository (GitHub remotes only), the branch or HEAD, the commit, a dirty flag, and a tree hash of the state that was tested: HEAD^{tree} for a clean tree, and the tracked and untracked, non-ignored files for a dirty one. It also carries a random per-installation signer and a sequence number that is never reused for a signer. No device serial, UDID, name, e-mail address or path is sent. The service keeps local runs as unverified evidence, shown beside the CI result; they never count toward a pull request's verdict. The team must enable local uploads and register or allow the repository. Limits are stricter than for CI: 256 KB and 2,000 results per upload. A retry resends the same bytes. --upload-dry-run sends nothing and reserves no sequence number. The hosted service is not live yet, so a real local upload gives a warning today. There is no signing yet.
  • Studio's ProbeScript editor offers completions, hover help and snippets for every command in the ProbeScript dictionary and every reserved word in the grammar. Both come from the docs through a generator (studio/frontend/scripts/gen-probescript-dictionary.mjs), so the editor and the documentation do not drift apart. npm run check:probescript fails when the generated module is out of date.
  • VS Code: tests in the native Testing view. Each test "<name>" block in a .probe file appears in VS Code's Testing view, with a run button next to it and in the editor gutter. A run executes the file with probe test --yes --format json and reports each selected test's pass or fail from the CLI's report. The CLI has no per-test filter yet, so a run executes the whole file. The Tests sidebar is unchanged.
  • Studio: a Settings panel in the device pane header changes dark mode, font size, app language, orientation (Android) and opens a deep link on the connected emulator or simulator, through the same code as probe device set. It acts only on the connected device and says why it is unavailable for WiFi connections and physical iPhones. Browser tests 26, 27 and 37 cover it (37 on a named simulator, reading each setting back).
  • probe device set and the set_device_setting MCP tool change an OS setting of one named emulator or simulator: appearance (dark mode), font scale, app language, orientation (Android) and opening a deep link, through adb and simctl. The device must be named (a serial, UDID or unique name, never booted), every value is checked before a command runs, and physical iOS devices are refused. Checked against an Android 15 emulator and an iOS simulator. The MCP server now has 36 tools.

Changed

  • The disclosure guard reports positions only, and can check a path tree. A match is reported by rule number and position: a changed file's number, the line, or a tree file's number. The matched text, the file path and the pattern are never printed. scripts/disclosure-scan.sh takes --tree PATH to check every file tracked under a path at the end of the range. A deleted file is not matched. Binary files, and file names that git quotes, are scanned too.

  • The MCP tools keep to the local tool surface. run_tests and run_script refuse the upload flags (--upload-report, --upload-local, --upload-dry-run, and any other --upload- flag) with the message "hosted upload is not available through MCP". The MCP tool text no longer describes the upload flags. Use the CLI from a terminal for uploads.

  • Building from source needs Go 1.26.10 or newer (was 1.26.1). The go line in go.mod, studio/go.mod, tools/probe-convert/go.mod and go.work is now 1.26.10, and the install docs say so. With the default GOTOOLCHAIN=auto, an older Go 1.26 downloads 1.26.10 on its own; with GOTOOLCHAIN=local the build stops with a version error until Go is upgraded.

  • Older evidence files removed. The disclosure guard also scans docs/evidence.

  • probe lint fails on references a run cannot resolve. A step or recipe call that resolves to nothing, or a selector that can never match, now makes the command exit with status 1, where it used to pass. A placeholder no value defines and a use file that does not exist are warnings and do not change the exit status. Lint also checks hook bodies and the device blocks of composite tests. probe test --dry-run checks only test bodies, so a bad step in those places fails at run time today, and a repository whose CI runs probe lint may see new failures there. Environment variables (${NAME}) are not checked, and the other limits are listed in the CLI reference. Within a file, errors print before warnings. Example output for a file with an error (colour codes omitted; the last line goes to stderr):

      ✗  x.probe
     line 7: unknown step "send reset code": not a built-in step and no recipe with that name is defined (check the use statements and the recipes folder)
     line 8: <code> is not defined: no example column, recipe parameter or stored value has that name

Error: 1 file(s) have errors

    A file with only warnings, such as a placeholder no value defines or a missing use file, prints them under a warning mark and still exits with status 0. The run then ends with All files valid.:

      ⚠  y.probe
     line 3: <code> is not defined: no example column, recipe parameter or stored value has that name

  All files valid.

  • Recording manifests list what is redacted and what is not. manifest.json replaces the redaction boolean with redacted and not_redacted lists, and the bundle schema is now version 2. Each entry says what its rule covers, and the not_redacted list names the kinds of content the rules do not reach, such as screenshots, free text, unusual encodings and values the run did not resolve from the environment. The viewer says that recordings may contain on-screen data, so review them before sharing. The recording feature is not released yet, so no released reader is affected.

  • probe record refuses an agent whose version does not match the CLI, as probe test already does.

  • Recording redaction covers credential names by their words. A name is treated as a credential when one of its words (split at separators, digits and camel-case boundaries) names a secret, token, password, key, card or session value, or when it is a known compound spelling. The value is replaced whatever its type, in JSON, forms, XML, multipart parts, URLs (query, fragment, path parameters and userinfo), text bodies and headers, and the agent masks custom credential headers by the same rule before it sends them. Resolved environment values are masked in their JSON-escaped and URL-encoded forms. A name that contains a credential word is masked whatever else it says, counters included. This is a stated trade-off.

Docs

  • Recording evidence write-ups no longer name private apps. The prose and commands in the evidence notes use neutral labels and placeholders. The raw run output that the notes cite is unchanged.
  • Run recording groundwork, internal only. The CLI's new internal/record package can write a recording bundle, follow the agent's HTTP log without skipping requests still in flight, redact credentials from headers, URLs and bodies, and render a bundle as an offline viewer. Step events now carry their wall-clock time. The CLI now exposes this through probe test --record; the design is in docs/prd/run-recording.md.

Fixed

  • Studio: the frontend builds with bundler module resolution. TypeScript 7 removed moduleResolution: "Node", so the Studio build failed on the TypeScript 7 and Monaco 0.57 upgrades. The Studio tsconfig now uses bundler.
  • CI: the website checks and the docs deploy run on Node 22.23.4. Astro 7.4.2 needs Node 22.23.4 or newer, and the runner cache held 22.23.3. The Dependabot compatibility jobs and docs.yml now pin 22.23.4.
  • VS Code: the extension compiles again with TypeScript 7 and @types/node 26. The tsconfig names the Node types, which TypeScript 7 no longer loads by default. Without it, npm test failed to compile after those two upgrades.
  • CI: the Dependabot auto-merge job adds approved pull requests to the merge queue again. It passed --squash with --auto, so the queue ignored the request, nothing was queued, and the run still reported success. The job now passes no merge method and logs a warning for each pull request it cannot queue.
  • CI: the Studio Go tests and the frontend and VS Code test suites run on every change to studio/ or vscode/. They ran in no workflow before, and only the browser suite covered Studio. The frontend suites run on Node 26.
  • probe-convert writes Robot keyword recipes in lower case, and its calls match them. The recipe kept its Robot spelling (Login As User), while the lexer writes every call word in lower case, so no converted call matched its recipe. The checkout golden is regenerated, and a test checks the names. A recipe name that keeps capital letters still cannot be called from a test; that is tracked separately.
  • CI: the CLI test package runs in seconds, not minutes. The hosted-upload failure test waited out the 5-minute retry window; it now uses a short window and checks that the failure is still only a warning.
  • probe test --device finds the Android application ID for a camelCase project.app. When the bundle ID as written is not installed on an Android device and its snake_case form is (com.example.myApp → com.example.my_app), the single-device run uses the snake_case ID, as --devices and parallel runs already did. Before, the restart relaunched the ID as written and failed. A run keeps the ID as written when that ID is installed, and when neither form is, so a project that runs today is unchanged. --devices and parallel runs still convert without this check.
  • Composite runs write the platform and OS version into each recording manifest. probe test --composite-device now passes each device's platform and OS version to its recorder, as single-device runs already do, so manifest.json is no longer empty for those devices. Devices the CLI cannot read a platform for (a WiFi composite device) still write empty values. The recording ships with probe test --record, which is not released yet.
  • The recording viewer fits a 390 px phone screen. Long step names, URLs and error text no longer push the page sideways: the single-column layout shrinks to the screen, long words wrap, and the 16 px side gutters stay. The desktop layout is unchanged. The viewer ships with probe test --record, which is not released yet.
  • VS Code: the Run Test commands no longer pass --name to probe test. This version of the CLI defines no such flag, so those runs failed with an unknown-flag error. "FlutterProbe: Run Test File at Cursor", the CodeLens buttons and the Tests view now run the file that contains the test, and their titles and tooltips say so. Until the CLI has a per-test filter, the extension runs whole files.
  • Studio: the top bar is a window drag handle on macOS. It used -webkit-app-region, which Wails does not read, so the full-size content view had no drag handle. It now sets --wails-draggable: drag; its buttons and device picker opt out so their clicks still work.
  • Studio: the macOS window has its native title bar again. The title bar was transparent, and Wails then makes the web view cover it, so the window had no native drag area and some toolbar controls did not show. The title bar is now opaque and sits above the top bar, so the window can be dragged from it and every control shows.
  • Dev tooling: @azure/msal-node is pinned to 7.1.0 in the VS Code extension's development tree, which closes GHSA-xcx6-jx8p-x3h5. The package reaches that tree through @azure/identity, and the packaged extension does not include it.
  • Dev tooling: the Studio frontend's npm tree resolves dompurify 3.4.17. monaco-editor pins dompurify exactly to 3.4.15, so an override sets the npm package to 3.4.17. The editor bundles its own copy of DOMPurify, which is unchanged at 3.4.15, so the code Studio ships does not change. The advisories need options that the bundled sanitizer does not set.
  • Output masking covers more paths. Resolved values are masked before a timeout hint is cut for display, and before a with ai assertion or a read ... with ai query is sent to the vision provider.
  • Short environment values are masked only when they are credential-shaped. A value of eight characters or more is always masked. A shorter value is masked only when its variable's name has a secret word as a whole token: PASS, PASSWORD, PASSWD, PWD, PW, SECRET, SECRETS, TOKEN, TOKENS, OTP, CRED, CREDENTIAL, CREDENTIALS, PIN, KEY, PASSPHRASE, PASSCODE, CSRF, SID, SIGNING or ENCRYPTION. It is then masked from three characters, and a plain word such as true is never masked under eight characters. So API_KEY=abc and PIN=1234 are masked, while SPINNER, MAPPING, BYPASS_MODE and MONKEY_COUNT are not, because the secret-looking part is not a whole token. A short value under any other name is printed as written, so ${COUNT} set to 1 no longer hides that digit in unrelated output. A short password under an ordinary name, such as a six-character DB_LOGIN, is not masked; name the variable with one of those words to mask it. Values typed into a system dialog from an environment variable, and the sandbox sign-in credentials, follow the same rule.
  • A password field stays secret when the app swaps it for a plain field under the same key. The agent remembers the key of a field that was seen obscured when the key is a ValueKey on a text field, so a field the app rebuilds under that key is not listed or read as text. Fields without such a key are tracked per controller only, and the agent keeps a bounded number of keys; past that bound every keyed field counts as secret. The recording's password-field map holds its fields weakly and starts fresh for each recording.
  • Connection error text is checked before it is printed. HTTP connection errors and relay connect errors no longer include the agent's address query.
  • The recording agent no longer errors on a text field that leaves the tree or is rebuilt. A change to a field that had left the tree was judged on its missing widget, which raised an error for each change, and a rebuilt field stayed bound to its old element, so its later text was lost. Text still pending in a field that leaves the tree is now recorded as obscured, with no value, and a rebuilt field is recorded through its new element. A pending text timer is also cancelled when recording stops, so it cannot emit into a later recording. Text typed into an ordinary field that leaves the tree before its 300 ms debounce fires is recorded as a type event with an obscured, empty value, because the field can no longer be read; this is a documented trade-off. The recording ships with probe test --record, which is not released yet.

Security

  • Studio no longer returns the Anthropic API key to the window. The key panel shows whether a key is saved and starts with an empty field. Saving an empty field no longer clears a saved key; use Remove key to delete it.
  • Studio keeps each WiFi token in the OS keychain, under its device, and never returns it to the window. A device with a saved token shows a 🔑 saved tag, and connecting to it needs no paste. Its ✕ control forgets the token. Studio deletes the plain-text copy that an earlier version kept in local storage on first start.
  • Studio's Run File accepts only .probe files inside the open workspace or an automation folder. Other paths are refused before anything runs, as the automation endpoint already refused them.
  • Studio opens only .probe files inside the open workspace or an automation folder. The editor's open action uses the same check as Run File, and a path with .. is refused.
  • Studio saves .probe files and lists folders only inside the open workspace or an automation folder. Save and the file tree use the same rule as Read File and Run File, including the .. and symlink checks. With no workspace open, the file tree is empty.
Assets 18
Loading