shuttle v0.1.19

• macOS: foreground click, type and key with a window bring that window to the front first and refuse when it cannot, instead of going to the frontmost app.

shuttle v0.1.18

• Bundled: cua-driver v0.38.0-loomhand.2 (was v0.38.0-loomhand.1). macOS: background typing into a native search field now triggers the app's search. A background accessibility write into a single-line native field outside web content, whose window is the app's key window, is followed by a net-zero key edit (space, backspace), so search-as-you-type fields such as WeChat's search box run their search. When that edit changes the field's text, as an accepted inline completion does, the result carries a warning.

shuttle v0.1.17

• Embedded mode. An app that ships shuttle inside its own bundle and starts shuttle serve itself sets SHUTTLE_EMBEDDED=1 for serve and mcp, so the macOS permissions belong to that app (install.md). serve then starts the bundled cua-driver with CUA_DRIVER_EMBEDDED=1, which raises no permission prompts of its own. mcp only connects: when no daemon answers it fails at once with no shuttle daemon answers at <endpoint>, and never starts one. Before, mcp from a copy of shuttle inside another app's bundle found that app as its own bundle and ran open -a <that app> --args serve.
• Stale driver socket. Before each start of the bundled cua-driver, the first one and every restart, the daemon removes a socket file at the driver's endpoint that refuses a connect, as one a killed driver leaves; a socket that answers or any other error leaves the file alone. A cua-driver in embedded mode refuses to start over an existing socket file. Windows pipes are unchanged.
• A click, scroll or drag that names a window takes pixels of that window's shot. With window, x, y (and to_x, to_y for a drag) are pixels of the image screenshot {window} (desk shot with window) returns, in the foreground and in the background, on any display and at any scale, so the point an agent picks in that image is the point it clicks. A cropped shot returns origin, its top left in the whole window image, and its pixel (px, py) is (origin[0] + px, origin[1] + py). A title that matches two listed windows alike, both equal to it or both containing it, is refused for such points, since a shot by that title may show the other window, and a window shot waits for a click, scroll or drag in flight, so the driver maps the click by the read it was made for. Before, they were desk coordinates, the main display's pixels, which an agent had to work out from a window origin that desk windows does not give, and which on a Retina main display are not the pixels of a window on a 1x display. Without window they stay desk coordinates; at keeps desk coordinates with window too. A script that passes window with desk coordinates now clicks elsewhere: drop window or convert the point.
• macOS: a click after a window shot lands where it is aimed. screenshot {window} reads the window at full size, and cua-driver maps the next pixel click on that window by that read, while shuttle still scaled the click by the smaller image of its own earlier read. On a window wider than 1280 image pixels (on a Retina display, wider than 640 points) a background click, or a foreground click in the window, after a shot of it landed short, at about 0.6 of its distance from the window's top left on a 2113-point TextEdit window. Shuttle now keeps the scale of the shot's read for the next click.
• at without a window reads the window at the point. desk at with neither window nor pid hit-tests the topmost app window the driver lists at the point, and the allowlist judges that window's app; with no app window there it fails with no app window at the point. Before, it hit-tested in the front window, so with an allowlist a point on another window failed with the front app's name (Warp is not in the app allowlist for a point on a window of an allowed app on a second display).

shuttle v0.1.16

• Bundled: cua-driver v0.38.0-loomhand.1 (was v0.37.0-loomhand.5), the fork's patches rebased onto upstream cua-driver 0.38.0 and later fixes: Windows list_windows leaves out the driver's own overlay windows, keys for a popup a Linux MPX click opened go through the master keyboard of the session that clicked, and the macOS background gate and window-change detector enumerate windows without the Space snapshot, now upstream (#5566, #5652, #5712); upstream also tracks the agent cursor's placement explicitly, so negative display coordinates no longer read as an unplaced cursor, keeps a Windows foreground drag off the window that covers its target, and keeps GNOME element actions inside the requested window. verify_hit, the kitty AXValue route, the off-Space window lookup, the macOS overlay tile and the X11 core-focus bounce carry over unchanged.

• macOS: a snapshot or find by title picks the on-screen window over its hidden twin. Among windows of the same title, none of them active, snapshot, find and the other verbs that pick a window by the accessibility tree take the newest one on a display before a minimized or hidden one, so snapshot of WeChat's "微信" reads its main window and not the hidden 280×380 one beside it. A window shows when its accessibility frame is the bounds of an on-screen window of its process, read from the window list each window listing already reads and from its position and size in the existing batched read. An app's only window of a title, such as a minimized one, can still be picked; two windows with the same frame both count as shown.

• shot of a window on any display. desk shot and the screenshot tool take window and return that window alone as cua-driver captures it, so a window on a second display, which the main-display capture does not hold, can be shot; x, y, w, h then crop the window's image in its own pixels. The window is picked as focus picks it, within the app allowlist. A rectangle off the main display still fails, and the error now says to name the window. A device shot refuses window.

• macOS: a title picks the on-screen window over its hidden twin. Background input, focus and the element click by title or pid no longer pick an app's hidden window that is off screen on the current Space when a window of the same app and title is on screen: WeChat keeps a hidden 280×380 "微信" beside its main window, and a click by title went to the hidden one and failed with "this window takes no pointer input without the foreground". An app's only window with a title, such as a minimized one, can still be picked, and twins across Spaces are still refused.

• gui_plan names a front app that stopped answering. On Linux, when no window is in front because the app of the X active window lists no window in AT-SPI (LibreOffice while it saves or reloads, an app whose accessibility peer hangs), a step that names no window stops with front app "<app>" (pid N) is not answering accessibility in place of no focused window, and an escalation without a front window shows that line as its outline. The new desk verb unanswered gives that window. It is read only when a plan stops.

shuttle v0.1.15

• gui_plan with no window in front. When no window is in front, as on GNOME while _NET_ACTIVE_WINDOW names none and no listed window is active, a wait step takes the one listed window that carries its title and brings it to the front, and a step that names no window acts in the plan's last window when one listed window has its title, bringing it to the front first. Before, wait failed with expected window "…" did not appear while its own window list held that title, and the next step failed with no focused window. Two listed windows with the awaited title still fail the step, with no window in front, an AT-SPI alert is not taken for the awaited window, and a plan with no window of its own yet still stops at no focused window. Background plans are unchanged.

shuttle v0.1.14

• Faster gui_plan wait for a window already in front. A wait step whose window is already the front one no longer reads that window's tree first. A dialog that has just opened, such as Save As, is still being built then, and that first read cost 67–70 ms (about 5 ms once built), so ctrl+shift+s then wait: "Save As" took 75–113 ms and now takes about 20 ms on an Ubuntu 24.04 XFCE desktop. A front window whose role is alert still stops the plan, as before, also when its tree reads empty. Since 0.1.13 the front window on Linux follows the X active window, which made the dialog front by the time the step started; 0.1.12 had skipped the read only because its stale front still named the editor.
• computer_act on an element without a press action (macOS). A left click or double click on an element whose app refuses its press action with -25206 (a window, sheet, toolbar, list or native text field) is sent once more as a verify_hit click: cua-driver brings the element's window to the front and clicks the element's centre only when the accessibility hit test there finds that element or one inside it. When the driver refuses, the result carries the press error and the refusal, and nothing is clicked. Before, shuttle clicked the element's centre on the desktop whenever its window was on top there, so it clicked whatever covered that centre inside the window (a toolbar's centre over a heading below it), and it refused when another window covered the element. A press that succeeds costs nothing more.
• desk.at names a window on top. On Linux and macOS at hit-tests inside the window it is asked about, so it named an element of that window even where another window covered the point (30 of 30 covered points on an Ubuntu 24.04 XFCE desktop, 56 of 56 on macOS). It now also returns covered with that window's title and pid: on Linux from the window manager's stacking order (_NET_CLIENT_LIST_STACKING, top down, past the panel and desktop windows; a popup menu or tooltip, which the window manager does not manage, is not checked), the top window counting as the target's own when it is its process's only window or that process's one window with the target's title, so two same-titled windows of one app count as covering, on macOS from the system-wide accessibility hit test. A gui_plan grounded click refuses such a point with grounding pointed at …, under the window "<title>" before any model check. Windows is unchanged: its hit test already returns the top window's element.

shuttle v0.1.13

• Bundled: cua-driver v0.37.0-loomhand.5 (was v0.37.0-loomhand.4): an element click can ask for verify_hit, which hit-tests the element's centre after fronting its window and a 150 ms (Windows) / 40 ms (macOS) hover settle and clicks only when the element is there, refusing with hit_mismatch otherwise; on macOS background text to kitty goes in as an accessibility value write, which kitty takes without being the key window, and a blank kitty read-back reports unverified instead of zero delivered; a macOS window on another Space is found through its app's focused or main window, and an off-Space window of an app with its own renderer (WeChat, Electron, Chromium, Qt) carries a note that its pixels can be stale.
• Off-Space note. The unverified note on background input to a macOS window on another Space now says that background input cannot read that window back; after typed text, not to type the text again, and to ask the user to confirm the text before submitting it with Return or Send; after a key, click, scroll or drag, not to send the input again, and to ask the user to confirm its effect. Before, "read it back on its Space before sending the input again" read to an agent as a stop before the Return too.
• Renamed to shuttle, under a loomhand namespace. The command is shuttle and the macOS app Shuttle.app (bundle id io.github.loomhand.shuttle); release archives are shuttle-vX.Y.Z-<target>. Everything else lives under loomhand, so another tool named shuttle does not collide: the app in ~/Applications/Loomhand, the config in ~/.config/loomhand/shuttle (%APPDATA%\loomhand\shuttle), the daemon socket in ~/Library/Caches/loomhand/shuttle or ~/.cache/loomhand/shuttle (Windows: \\.\pipe\loomhand-shuttle-desk), the Linux install in ~/.local/lib/loomhand/shuttle, the Windows install in %LOCALAPPDATA%\Programs\loomhand\shuttle, and the process-wide class names LoomhandShuttleControls and LoomhandShuttleConfirmWatch. There is no fallback to the old names. To upgrade: quit the old daemon (pkill -f 'cocoon-shuttle serve'), install, move ~/.config/cocoon-shuttle to ~/.config/loomhand/shuttle (Windows: %APPDATA%\cocoon-shuttle to %APPDATA%\loomhand\shuttle), point your agent's MCP entry at shuttle mcp, grant Accessibility and Screen Recording to ~/Applications/Loomhand/Shuttle.app, and remove ~/Applications/Cocoon Shuttle.app, ~/.local/bin/cocoon-shuttle and the old grants. Other tools also install a shuttle command (cargo-shuttle, Homebrew shuttle-cli), so check which shuttle; the daemon starts its app by full path, never by name, since another macOS app is named Shuttle.
• Linux front window. windows marks as active the one window that _NET_ACTIVE_WINDOW names, by its pid and title, when AT-SPI still calls another window of the session active, so after a background type_text into a back GTK window front names the window in front and not the back one; GTK keeps a window's AT-SPI active state after the X focus moved on. Two windows of that pid with that title leave no window active, so front names none rather than maybe the wrong one; an X window whose pid has no AT-SPI window, or whose title is longer than 4 KiB, keeps AT-SPI's state. X window titles are read whole up to 4 KiB, where they were cut at 256 bytes. The check reads the active window, its pid and its title on the MIT-SHM connection, three X round trips per windows call for a title of up to 256 bytes and one more per further 256 bytes; without MIT-SHM the AT-SPI state stands.
• computer_act clicks on text fields on macOS and Windows. A left click on a text field element (AXTextField, AXTextArea, AXComboBox, Edit, ComboBox) asks the bundled cua-driver for a verify_hit click, which v0.37.0-loomhand.5 delivers: the driver brings the field's window to the front and posts a click at the field's centre only when the accessibility hit test there finds that same element or one inside it, so typing next goes into the field. Before, the click ran the field's press action, which does not focus it on macOS, or a background click on Windows, and the text went to whatever window was in front. When the driver refuses (another element on top, the field gone or replaced, no foreground), the field gets its press action, and if it has none the refusal is returned rather than a click at a point; a click that failed after it was sent is not repeated. A cua-driver without verify_hit, such as an older one at its default socket that a build without a bundled driver uses, presses the field as before.
• Grounded clicks need an element and a clear yes. When a gui_plan click finds no element and grounds the target on the screen, it clicks only where the accessibility tree has an element under the point that is not a root (a window, frame, application, dialog or document, which macOS AXWebArea and UIA Document map to) and lies inside the plan window or a window it owns, judged by the element's ancestors when they name a window (a Windows UWP app shows its content from another process than its frame's) and by its process when they name none; otherwise it refuses and asks no verification model, with grounding pointed at no accessible element, …, a window or document root, …, which belongs to another process, …, whose process is unknown or, when the element under the point could not be read, … a point whose element could not be read: <reason>. desk.at now returns the pid of the element it finds. So gui_plan no longer clicks by vision alone in an app that exposes no accessibility tree, such as desktop Chrome on a cocoonbox desktop: use computer_act there. The first check of the 640x400 crop now takes the click alone only at p ≥ 0.9 (was 0.5); below that the zoom check (0.65) and the screen check with a ring (0.95) decide. A control that was not on a Chrome page got a grounded click in about 1 run of 7 before, at crop answers of 0.43–0.58 for a point with no element under it. The step's trace entry now carries ground with the check path, p, p_zoom, p_full and the element under the point (hit), whether the step clicked or refused.

shuttle v0.1.12

• Quieter serve log off Linux. On macOS and Windows the computer verbs no longer try the Linux MIT-SHM grabber on each call, so the serve log no longer repeats desk: MIT-SHM capture unavailable, using cua-driver for every computer call.
• Bundled: cua-driver v0.37.0-loomhand.4 (was v0.37.0-loomhand.2): on macOS the agent cursor overlay repaints only the region around its cursors instead of the whole screen each animation frame, so a background press_key, type_text or scroll takes about 0.5–0.6 s in place of 1.3–2.3 s while the cursor animates on a GPU-less VM; on Linux under xfwm4, a background type_text into a window behind another no longer leaves the front window looking inactive: the focus guard puts back the window manager's active window by moving the core focus through None and back, where it used to report not_restored after its full restore budget (about 1.1 s → 0.64 s per call on cross-app pairs).

shuttle v0.1.11

• Auto input. [desk] input = "auto" sends desk input in the background and takes the foreground only for an action that background input refused before anything was sent: shuttle brings that action's window to the front, runs it once in the foreground, and its result says "input": "foreground". A Windows drag runs in the foreground from the start. Input the driver could not confirm comes back with unverified, as in background, and is never sent twice. gui_plan types a step again in the foreground only when its read-back shows that nothing landed and the front held, and a step background refuses before sending anything (a macOS menu-bar entry, grounding, a pixel state read, a vision expect) fronts the plan window and runs, with the rest of the plan, in the foreground. On Linux and with an [android] section, auto acts as foreground (safety.md).
• macOS menu-bar icon. The indicator is an 18 pt template icon in place of the Shuttle text: outlined when idle, filled while acting, struck through when stopped. Its tooltip and the first line of its menu name the version and the input mode. The icons are vector PDFs built into the binary, and the item redraws only when its state changes. A menu-bar manager such as Ice puts a new item in its hidden section until you move it (safety.md).
• macOS windows on another Space. With [desk] input = "background", windows on macOS also lists the windows on another Space, a desktop or a full-screen app's, marked space: "other" and never active, so background input can name them; before, a window on another Space was missing from the list even by pid. Background input by title no longer picks an app's hidden window that belongs to no Space over its window on a Space, and refuses a title that two windows of one app share across Spaces, naming both. When the driver cannot confirm input to a window on another Space, unverified adds that its capture may be stale. In background mode snapshot, find, at and close on such a window fail with "is on another Space" in place of "no window". The listing asks WindowServer for the windows of the Spaces no display shows, about 1 ms per windows call in background mode; the private calls it uses are looked up at run time, so a macOS without them only loses the list.

shuttle v0.1.10

• Background gui_plan. With [desk] input = "background", gui_plan drives the window its app or wait step names, and the dialogs that window owns, without fronting it or moving your pointer, on macOS and Windows; it refuses only on Linux. It reads that window by its id, and refuses input when another window of the app has the same title, a step that needs the screen, a macOS menu-bar entry, a close of a dialog it cannot tie to the window, input the driver could not confirm that its own read or the next wait, verify or expect step does not show, and input after its window is gone; an app that brings itself to the front fails the step. Known limits: the typing check counts the text in any element of the window, and a macOS dialog with no owner is taken by its app and its arrival after input. desk windows names the window that owns a macOS sheet or a Windows owned window as owner (safety.md).
• Computer verbs on macOS and Windows. desk computer_state, computer_act, computer_end, computer_scale and look answer on macOS and Windows as on Linux, refused under an app allowlist. Screenshots and the settle come from cua-driver's captures; screenshot pixels map to the screen by the driver's screen and screenshot widths when it reports no frame_scale; keys and typing go to the frontmost window; input at a screen point now goes in foreground delivery, so it drives the real pointer and may bring the app it clicks to the front, and on a Retina Mac it goes in capture pixels; window bounds may be fractional; and an element whose macOS app refuses its press action gets one click at its centre when the accessibility hit test there lands in its own window. look with a region cuts it from a full-size driver capture wherever MIT-SHM is not available.
• macOS launch grace. A macOS app in its first 10 s on screen is no longer skipped from windows for 10 s when its accessibility reads keep timing out, so a launch that keeps its app busy no longer hides its window past a gui_plan app step's 8 s wait (the 0.1.9 known issue). An app that was on screen before keeps the 2 s / 10 s rule.
• macOS typing pace. macOS foreground typing of basic-plane text no longer waits 30 ms per character in the driver: 20 characters take about 385 ms in place of 866. Astral characters, such as most emoji, and background typing keep the driver default.
• Windows pipe hang-up. On Windows a client connection dropped with a write still pending cancels its I/O, so the other end sees the hang-up at once: a service splice whose first pump ended and a call an MCP cancel dropped mid-write left the server waiting before, since the handle stayed open until a write it never read completed.
• Bundled: cua-driver v0.37.0-loomhand.2 (was v0.37.0-loomhand.1): on macOS get_desktop_state with no overlay to leave out captures through ScreenCaptureKit instead of a screencapture process, about 170 ms in place of 325 ms at a 1024 cap on a GPU-less VM; on Linux an ended session keeps its master pointer while its menus are open by upstream's own code (trycua/cua#5254), and a text may use keycode 248 as a spare, so 18 of a desktop box's 19 free keycodes.
• Known issue: on a Retina Mac guest whose apps answer accessibility slowly, a gui_plan step can still fail with no accessibility answer within 3000 ms, or find a just-launched window late; this is the app's own latency, not the skip.