Skip to content

shuttle v0.1.13

Choose a tag to compare

@github-actions github-actions released this 21 Oct 07:26
· 79 commits to main since this release
  • Bundled: cua-driver v0.37.0-loomhand.5 (was v0.37.0-loomhand.4): an element click can ask for verify_hit, which hit-tests the element's centre after fronting its window and a 150 ms (Windows) / 40 ms (macOS) hover settle and clicks only when the element is there, refusing with hit_mismatch otherwise; on macOS background text to kitty goes in as an accessibility value write, which kitty takes without being the key window, and a blank kitty read-back reports unverified instead of zero delivered; a macOS window on another Space is found through its app's focused or main window, and an off-Space window of an app with its own renderer (WeChat, Electron, Chromium, Qt) carries a note that its pixels can be stale.
  • Off-Space note. The unverified note on background input to a macOS window on another Space now says that background input cannot read that window back; after typed text, not to type the text again, and to ask the user to confirm the text before submitting it with Return or Send; after a key, click, scroll or drag, not to send the input again, and to ask the user to confirm its effect. Before, "read it back on its Space before sending the input again" read to an agent as a stop before the Return too.
  • Renamed to shuttle, under a loomhand namespace. The command is shuttle and the macOS app Shuttle.app (bundle id io.github.loomhand.shuttle); release archives are shuttle-vX.Y.Z-<target>. Everything else lives under loomhand, so another tool named shuttle does not collide: the app in ~/Applications/Loomhand, the config in ~/.config/loomhand/shuttle (%APPDATA%\loomhand\shuttle), the daemon socket in ~/Library/Caches/loomhand/shuttle or ~/.cache/loomhand/shuttle (Windows: \\.\pipe\loomhand-shuttle-desk), the Linux install in ~/.local/lib/loomhand/shuttle, the Windows install in %LOCALAPPDATA%\Programs\loomhand\shuttle, and the process-wide class names LoomhandShuttleControls and LoomhandShuttleConfirmWatch. There is no fallback to the old names. To upgrade: quit the old daemon (pkill -f 'cocoon-shuttle serve'), install, move ~/.config/cocoon-shuttle to ~/.config/loomhand/shuttle (Windows: %APPDATA%\cocoon-shuttle to %APPDATA%\loomhand\shuttle), point your agent's MCP entry at shuttle mcp, grant Accessibility and Screen Recording to ~/Applications/Loomhand/Shuttle.app, and remove ~/Applications/Cocoon Shuttle.app, ~/.local/bin/cocoon-shuttle and the old grants. Other tools also install a shuttle command (cargo-shuttle, Homebrew shuttle-cli), so check which shuttle; the daemon starts its app by full path, never by name, since another macOS app is named Shuttle.
  • Linux front window. windows marks as active the one window that _NET_ACTIVE_WINDOW names, by its pid and title, when AT-SPI still calls another window of the session active, so after a background type_text into a back GTK window front names the window in front and not the back one; GTK keeps a window's AT-SPI active state after the X focus moved on. Two windows of that pid with that title leave no window active, so front names none rather than maybe the wrong one; an X window whose pid has no AT-SPI window, or whose title is longer than 4 KiB, keeps AT-SPI's state. X window titles are read whole up to 4 KiB, where they were cut at 256 bytes. The check reads the active window, its pid and its title on the MIT-SHM connection, three X round trips per windows call for a title of up to 256 bytes and one more per further 256 bytes; without MIT-SHM the AT-SPI state stands.
  • computer_act clicks on text fields on macOS and Windows. A left click on a text field element (AXTextField, AXTextArea, AXComboBox, Edit, ComboBox) asks the bundled cua-driver for a verify_hit click, which v0.37.0-loomhand.5 delivers: the driver brings the field's window to the front and posts a click at the field's centre only when the accessibility hit test there finds that same element or one inside it, so typing next goes into the field. Before, the click ran the field's press action, which does not focus it on macOS, or a background click on Windows, and the text went to whatever window was in front. When the driver refuses (another element on top, the field gone or replaced, no foreground), the field gets its press action, and if it has none the refusal is returned rather than a click at a point; a click that failed after it was sent is not repeated. A cua-driver without verify_hit, such as an older one at its default socket that a build without a bundled driver uses, presses the field as before.
  • Grounded clicks need an element and a clear yes. When a gui_plan click finds no element and grounds the target on the screen, it clicks only where the accessibility tree has an element under the point that is not a root (a window, frame, application, dialog or document, which macOS AXWebArea and UIA Document map to) and lies inside the plan window or a window it owns, judged by the element's ancestors when they name a window (a Windows UWP app shows its content from another process than its frame's) and by its process when they name none; otherwise it refuses and asks no verification model, with grounding pointed at no accessible element, …, a window or document root, …, which belongs to another process, …, whose process is unknown or, when the element under the point could not be read, … a point whose element could not be read: <reason>. desk.at now returns the pid of the element it finds. So gui_plan no longer clicks by vision alone in an app that exposes no accessibility tree, such as desktop Chrome on a cocoonbox desktop: use computer_act there. The first check of the 640x400 crop now takes the click alone only at p ≥ 0.9 (was 0.5); below that the zoom check (0.65) and the screen check with a ring (0.95) decide. A control that was not on a Chrome page got a grounded click in about 1 run of 7 before, at crop answers of 0.43–0.58 for a point with no element under it. The step's trace entry now carries ground with the check path, p, p_zoom, p_full and the element under the point (hit), whether the step clicked or refused.