Desktop Tools 0.17.1

Fixed

• desktop_select selects text in a web page that isn't a field, such as a paragraph or a link, through WebKit's text markers. An image pasted into a web field is refused with PASTE_UNVERIFIABLE_RECEIVER before the focus check, since the page's handling can't be read back. (#88, #99)
• Text selected in a Safari page with desktop_select can be copied: selecting focuses the page, as a click would, so Edit > Copy in the activated Safari copies the selection instead of the page's address. Safari copies only while active, so desktop_activate it and choose Edit > Copy with desktop_menu; the tool description and docs said to press Command-C, which Safari ignores in the background. (#188)

Desktop Tools 0.17.0

Changed

• The app runs a pure Swift host in place of the Peekaboo Bridge runtime, and bundles its own MCP server, Desktop Tools.app/Contents/MacOS/desktop-tools-mcp. Set agents up with that path; Node is no longer needed. Calls are faster, for example desktop_windows at a p50 of 6 ms instead of 115 ms and an inspection of a Ghostty window at 42 ms instead of 551 ms, and results have the same text. The app's binaries are about 6 MB in place of 84 MB. (#143)
• An npx setup of desktop-tools-mcp at v0.17.0 or later hands over to the app's server. A setup pinned to an earlier tag no longer works with this app: its calls are refused with CLIENT_OUTDATED, naming the app's server to set up instead. (#143)
• desktop-tools-build --native2 builds the Swift host, client, MCP server and app without Peekaboo; embedders switch with it and createDesktop({ wire: 2 }). The default build is unchanged. (#171)

Known gaps

• desktop_switch_space is checked only on a Mac with one Space.
• Where MenuBarAgent doesn't draw the menu bar, desktop_menubar_items lists and clicks status items through their WindowServer windows. This path is unit-tested but not yet checked live.

Desktop Tools 0.16.1

Fixed

• The window outline no longer draws over a sheet or Open panel that opens on top of its window, such as an Open panel on System Settings' Search Privacy sheet. The window server keeps a window's sheets with it, so the outline is cut away under them, and it follows them as they open and close. (#149)
• desktop_dialog and desktop_dialog_answer read and answer an Open panel stacked on a sheet, such as the one + opens on System Settings' Search Privacy sheet. With the app or the parent window as the target, they picked neither and refused the target as ambiguous; the innermost dialog, which the others wait on, is now the one chosen. (#148)
• A point click on a control below a list that grows as it's read, such as Search Privacy… in System Settings > Spotlight, presses it. Finding the control read the whole pane, which made the app list longer and moved the button off the point, so the click was refused as finding no pressable element. (#147)

Desktop Tools 0.16.0

Added

• desktop_select selects text on a PDF page in Preview. It was refused because the page's text reports itself disabled and has no selection until one is set. To copy it, activate Preview and choose Edit > Copy, since Preview disables Copy in the background. (#130)

Changed

• Results are compact, so an agent's context fills far more slowly: the restore, focus, inspect and click loop on a TextEdit window went from 34,721 characters to 5,543. Elements list their fields once, with bounds as [x, y, width, height], real booleans and state only when set; window and app rows keep their target and drop the fields it repeats; detection metadata, timings, diagnostics, coordinate context and checked receipts are left out. A large window lists more of its 200 elements, since the limit applies to the compact result. See Result shape. DESKTOP_TOOLS_VERBOSE=1, or verbose in the client config, returns native results whole. (#144)
• Window actions (focus, restore, minimize, maximize, full screen, move, resize) return the window's refreshed row instead of the app's whole window inventory and an inspection with a screenshot; inspect before input. (#144)
• desktop_windows counts the untitled off-screen windows that most apps keep, such as menu bar strips, in hidden_windows instead of listing them; all: true lists them. (#144)
• The agent's name label under the window outline is a point smaller and tighter, and edged in a darker shade of the outline color instead of white.
• Peekaboo is vendored in native/vendor/Peekaboo, with our changes applied, instead of fetched and patched at build time. Native builds no longer fetch Peekaboo, and npm run peekaboo-test runs the vendored tests without --scratch. git diff peekaboo-upstream-4d43dc9 -- native/vendor/Peekaboo shows our changes to upstream.

Fixed

• A point click on a sheet that Accessibility only lists inside its parent window, such as the + in System Settings > Spotlight > Search Privacy, or Cancel in the Open panel it opens, reaches the sheet. It was refused as occluded by the parent, although nothing covered the sheet. (#141)
• A point click that finds no pressable element, or an element in a window covering the target, is refused instead of unknown, since nothing was sent, and the message no longer suggests a --foreground option the tools don't have. The occlusion refusal names the window it found. (#141)

Desktop Tools 0.15.0

Added

• desktop-client status and desktop-client repair, run from a shell through the link in the state folder, show whether the app serves its socket and fix it when it doesn't, by opening the app and waiting for it to serve. See Keeping the socket served. (#140)
• desktop_tools_check() in the runtime's C ABI checks the socket now, for an embedding app to call when it is opened again. (#140)
• The Desktop Tools app has an icon, a placeholder for now, in Finder, its disk image and its prompts. (#135)
• A recipe for picking from a <select> in Chrome and other Chromium browsers: one desktop_key call with Down to open the menu, Down to the option and Enter, in the background. Clicking or AXPick on its menu items is refused there, because the menu closes between tool calls. (#139)

Changed

• The Desktop Tools menu is shorter. Agent activity moved from Recent Activity and All Activity to a log, activity.log in the state folder, which Open Log opens. The log keeps every operation, not just the last 20, and also notes when the app starts, with its socket, and when you pause input or allow or refuse an agent. The socket path is no longer in the menu, Serve agents is gone (Pause All Input, Quit or Approved Agents do what it did), granted permissions are no longer listed (a Grant item shows only while one is missing), and Pause All Input sits just above Quit. Embedding apps can write the same log with desktop_tools_set_log. (#132)

Fixed

• Desktop Tools keeps retrying when it can't start serving, such as when it reopens while the copy that is quitting still holds the socket. It used to stop at the first failure and refuse every call until it was quit and reopened. The menu's status line shows the error and when it retries next. (#140)
• Desktop Tools no longer quits when an agent closes its connection before the reply. A write to the closed connection raised SIGPIPE, which killed the app and left its socket behind, so every later call failed until it was opened again. (#140)
• Desktop Tools publishes its socket again when something removes or replaces it while it serves. Before, the app kept running with no socket, and every call failed with "Open Desktop Tools" until it was quit and reopened. Opening the app while it runs now checks the socket too. (#140)
• When a call finds no app serving the socket, the client opens Desktop Tools in the background, waits up to 10 seconds for it to serve, and then runs the call. Before, an app that wasn't running or wasn't serving needed the user to notice and restart it. When that doesn't help, calls fail with HOST_NOT_RUNNING or HOST_NOT_SERVING, saying what the client found, instead of Connection refused or "Open Desktop Tools". (#140)
• Only one Desktop Tools app runs for each state directory. A second one, such as the app binary run from a shell, exits with a message instead of taking over the first one's socket. desktop-tools-app --version prints the version, and other arguments are refused. (#140)

Desktop Tools 0.14.1

Fixed

• The agent cursor sits directly above the window the agent is using instead of above every window, so a window covering that app covers the cursor too. Input outside any one window, such as on the Dock or the menu bar, still shows it on top. (#133)
• The agent window outline and its label stay on the display when the window fills it; they were drawn just outside the window and ran off the screen's edges. (#133)
• desktop_close on a Finder window is completed again instead of unknown: Accessibility lists Finder's desktop as a window without an ID, which made every close unverifiable on macOS 27, even after the window was gone. (#133)
• desktop_wait and expect app conditions with bundle_id find the app when many apps are running; the app list cut to fit a result could drop it, and a cut list no longer counts as proof that an app quit. (#133)
• desktop_read with table: true finds the column titles of web tables in Safari 27, which reports <th> cells as plain cells. (#133)
• desktop_select in Safari 27 web text no longer crashes the Desktop Tools app, and finds text bounds without the text marker attributes Safari dropped. (#133)
• Pasting files with desktop_insert into a Finder window on macOS 27 finds the window's folder from its path bar, since Finder no longer reports it otherwise; without the path bar, the refusal says to turn on View > Show Path Bar. (#133)
• desktop_set_value on a Safari 27 date input is refused before input, with a hint to type the date, instead of unknown; it accepts an ISO 8601 date where a date can be set. (#133)
• Management failures, such as an unverified desktop_close, include the native cause, as action failures do. (#133)

Desktop Tools 0.14.0

Added

• The Desktop Tools app asks once per agent, such as Claude Code or a terminal app, "Allow name to control this Mac?", and refuses every request from an agent you haven't allowed with CALLER_NOT_APPROVED, listings and captures included, without making it wait. Agents are recognized by their validated signature, or by their path when unsigned, and Approved Agents in the menu revokes, allows or forgets each one. After updating, each agent you use is asked about once. Recent Activity names the agent, such as claude-code, instead of node. Embedding apps opt in with desktop_tools_set_caller_handler. (#120)
• desktop_fill_password fills the focused password field. It reads a Keychain item the agent names, by service or by server (an internet password fills only a web page on that server), so macOS asks you before Desktop Tools reads it and the agent never sees the password. It can also take text from the agent, which you approve fill by fill in a Desktop Tools prompt that names the agent, the app and the site, unless you turn on Always Allow Agent-Supplied Passwords in the menu (off by default). It is completed when the field shows the right number of characters. Results and Recent Activity never include the password or its length, and system password prompts, such as administrator authentication, are refused. Embedding apps ask the user through desktop_tools_set_confirmation_handlers and set the always-allow choice with desktop_tools_set_password_autofill. (#124)

Changed

• desktop_inspect keeps containers an agent can scope into, such as a Finder window's file list after its long sidebar, when its result is cut to size, dropping other elements first. (#126)
• Agent input aimed at any Desktop Tools process, including another build of it, or sent while one of its prompts or menus is open, is refused with SELF_TARGET_REFUSED, so only the user can answer its prompts or choose Pause All Input. Agents can still observe them. (#120)
• Behavior change: while a password field has keyboard focus, desktop_key refuses keys that type characters and Command-V, and desktop_insert refuses all its routes, and desktop_menu refuses Paste commands (found by their Command-V shortcut, whatever the title), with SECURE_FIELD_REQUIRES_FILL. Nothing is sent. Return, Tab, Escape, arrows, Delete and other shortcuts still work. Passwords go in only through desktop_fill_password and its approval. (#124)

Fixed

• desktop_click with modifiers, such as Command-click or Shift-click, no longer fails its cleanup every time with DESKTOP_ACTION_FAILED and "cursor restoration … could not be verified": the host read the pointer back before its restoring move had landed. It reports cursor_restoration again and stays unknown with DESKTOP_ACTION_UNVERIFIED, since a modifier click's effect can't be confirmed. (#128)
• desktop_clipboard_read with save_path saves images again; the host never enabled image export, so every such read was refused. The build now fails if the host would drop an operation the app allows. (#125)
• A clipboard paste whose receiver never took it, such as an image pasted into a TextEdit .rtf document, no longer blocks every clipboard write and paste until that app quits: closing the receiving window releases it too, and the refusal says so. (#125)
• window_appears no longer counts a window macOS keeps listed but ordered out on the current Space as open, so expect with window_appears for Spotlight works on macOS 27, where Siri AI (com.apple.campo) hosts the panel. (#125)
• Command-A in a text field is completed again instead of unknown with a receipt mismatch: the client checked the host's receipt for a layout-resolved key without the keyboard layout, which only the main thread can read. (#126)
• Finder windows with a long sidebar keep their focused file list in observations, so desktop_insert with format: "files" pastes into them instead of being refused for having no focused control. (#126)
• Renaming a file through Finder's Name column with desktop_set_cell, or its name field with desktop_type or desktop_set_value, used to report completed while Finder only showed the new name and never renamed the file. It is now refused before input with CELL_NOT_EDITABLE. (#126)
• A desktop_key sequence that stops partway reports KEY_SEQUENCE_STOPPED whatever stopped it, as when Command-N moves focus to a new window, instead of a bare DESKTOP_ACTION_FAILED, and says when the key that stopped it may have been sent. (#126)
• desktop_read with table: true no longer reports a role description such as "cell", "column" or "image" as cell text or a column title, or a folder's disclosure triangle as 0 in its Name cell, and names a sorted column by its title rather than its sort direction. (#126)

Desktop Tools 0.13.1

Fixed

• Secure text fields, such as password fields, show attributes.subrole: "AXSecureTextField" and no value in desktop_inspect, desktop_read and desktop_wait, not even their mask characters; desktop_type, desktop_set_value and desktop_select refuse them before dispatch instead of returning unknown. (#118)
• desktop_inspect, desktop_read and desktop_wait see Chrome 155's web content again: Chrome now rejects AXManualAccessibility, so the host falls back to AXEnhancedUserInterface for apps that ship Chromium, and results name the one used in web_accessibility_via. An app that accepts neither now reports unsupported instead of unknown. (#119)

Desktop Tools 0.13.0

Added

• desktop_insert with format: "files" copies files into a Finder window's folder, refusing names already there, and restores the clipboard once every name appears. (#114)

Desktop Tools 0.12.0

Added

• npm run peekaboo-test runs Peekaboo's Swift tests without AI provider credentials in the environment, which its agent tests refuse, and stops only that run's test helpers. (#112)

Changed

• desktop_inspect and desktop_read retry an incomplete Accessibility read once, and say retried when it fails again. (#111)
• desktop_wait also turns on Chromium and Electron web accessibility before its element and text polls, so it finds web content as inspect and read do. (#110)