Agent Screen 0.1.18

Features

• get_window_state and act_on_element take max_depth, which leaves out elements nested deeper and says how many. act_on_element takes result: changes returns only the elements that appeared, changed or went away since the window's last tree, with the window image, and none returns only "Done.". The default still returns the whole window state. On Calculator, a press with changes returned 166 bytes of text in place of 1,859.
• place_window takes x, y, width and height to move or resize a window on the agent's display, in screenshot pixels from the display's top-left corner. It returns the frame the window ended with, since apps keep to their own minimum sizes, and says when that isn't the frame asked for. An agent no longer needs System Events to resize a window, where a position meant for the display moved the window onto the user's screen.
• Recordings… in the menu bar icon's menu opens a local video library. Select a recording to play, pause or seek in the app. Show Pointer toggles a visual overlay without changing the video, sound or playhead. New recordings store timed pointer coordinates; older videos still play when their timelines cannot accurately replay a pointer. The overlay hides during Takeover because user pointer movements were not recorded. New recordings request HEVC/MP4 when macOS advertises it and use H.264/MP4 when HEVC is unavailable. The recording result reports the selected codec. Hardware encoder use and same-quality size or CPU improvements have not been verified for ScreenCaptureKit. Sound still joins through passthrough export, and the pointer stays outside the encoded video.
• request_control lets an agent ask for its display back while you are operating it. You get a notification and a card under the menu bar icon with its reason, and answer Let It Continue, which hands the display back, or Keep Control, which tells the agent you kept it. Nothing else ends your Takeover: an unanswered request stays open, and the agent can withdraw it. When the agent already has control, the call returns at once.
• Agent Screen now updates itself. It checks for a new version once a day, downloads it in the background and installs it once no Agent Display is open, so an update never ends an agent's work. The menu bar icon's menu has Check for Updates…, and Restart to Update… once an update is waiting; with Agent Displays open, it asks before the restart ends them. This version is the last to install by hand.
• Drags in web pages work in Chromium and Electron apps an agent starts with launch_app: splitters, sliders, sortable lists and HTML5 drag and drop. Before, every move reached the page with no button held, so the page took the drag for hover. launch_app now starts these apps itself with a private DevTools pipe that only Agent Screen holds. No port opens. A drag goes through the pipe inside the app, so your mouse button and pointer stay yours. For a browser, the agent starts a new instance on a throwaway profile. A drag in a web page of an app without the pipe, such as Safari or an app you opened, now fails with a message that suggests launch_app or request_help. Before, it seemed to succeed and did nothing.

Fixes

• The Apple Account sign-in alert ("Your Apple Account is being used to sign in to a new device", with Don't Allow and Allow) and its verification code window now move to your main display when macOS opens them on an Agent Display. Before, they stayed on the agent's display, where you didn't see them and an agent could answer them.
• Collect build information for a bug with the MCP tool get_diagnostics in the failing session, or AgentScreen diagnostics --json for the CLI and responding app. Reports use compiled package versions and source revisions, distinguish stale or unknown builds, and leave the app stopped if it is not running. The bug report form and reporting guide explain older-build fallbacks and what to attach.
• The Help Card and a Window Request's card end a long reason after two lines with "…". Before, the rest of the text ran on in the second line past the card's edge and was cut mid-letter. The Viewer is at least 640 pt wide, so its header's buttons fit for a tall, narrow display. A 640×1600 display made a 365 pt Viewer that pushed Give, Take Back and the close and collapse buttons past its edge.
• Agent Screen retries an AeroSpace claim-file write after it fails, even when the next request repeats the same claim. It keeps the intended displays and windows separate from the last published file, so a failed display registration does not prevent later window claims.
• The display log in ~/Library/Logs/Agent Screen/displays.log records when the app starts and quits, and a released … quit line for each display still connected at quit. Before, most displays left with the process and the log never said so, so a slow or frozen Mac couldn't be matched to those display changes.
• When you bring up an app whose windows are all on Agent Displays, from the Dock or a launcher, Peek drops down from the menu bar icon and shows where they are: that display's tile glows orange with the app's windows outlined. Click the tile to open it in the Viewer, or Take Back to bring the windows to your screen. Peek goes after 5 seconds unless your pointer is on it. Clicking the app's Dock icon does this too, also when the app is already the front app. Before, the app seemed to open with no window. It shows at most once a minute for an app, and not when an app the agent started brings itself forward.
• Pressing a menu item of an Electron app with act_on_element now comes back with a note. Electron tells the app that the press was the item's keyboard shortcut, and some apps ignore shortcuts while a text field has focus. The note, the tool's description and the agent instructions all say to click outside the field first. Agent Screen can't make the press look like a click: macOS runs an accessibility press with no current event.
• An Agent Display keeps the size it was made with. When macOS switches it to another mode by itself, as it did under heavy load from 1600×1000 to 800×600, Agent Screen switches it back as soon as macOS reports the change, and notes it in the display log (~/Library/Logs/Agent Screen/displays.log).
• launch_app with arguments or an environment takes the instance it started. Every checkout of an Electron project runs the same Electron.app, and a copy another tool started from a shell during the launch could be taken for it: that copy's windows went to the agent's display, and the agent's own window was lost. Agent Screen now picks the new process launchd started with the arguments it asked for.
• launch_app with arguments or an environment, on an app that allows only one instance and already runs, answers with the running instance's window on your display and says the app was already running. It used to wait about 10 seconds and answer that the app was still starting, or fail with "did not start". When the running instance has no window on the display, the error says the app is already running.
• When you open an app from Spotlight, Raycast or Alfred while you work in a Viewer, its window now leaves your screen within about 70 ms. Before, it showed there for 150 to 430 ms, and AeroSpace re-tiled your windows around it meanwhile. Agent Screen now checks for the new window every frame instead of every 150 ms, and moves it before it asks AeroSpace anything.
• macOS closes an app's open menu, such as a web page's select, at your next click or key anywhere, on your own screen too. An agent that opened one on its display and then clicked or typed to choose an item hit the page under the closed menu instead, or changed the field without seeing it. Now that click or key fails, says the menu closed, and nothing is sent.
• Clicking the menu bar icon after hovering now pins Peek instead of closing it. A later click closes the pin. Hover opening uses a 300 ms dwell, and an unpinned Peek uses a 500 ms grace period after the pointer leaves both the icon and Peek. A left-button press keeps its original intent while hover timers run, so its release cannot turn a close into a reopen.
• place_window answers placed only once the window has stayed on the display for half a second after its last move. When the app keeps moving it back to the user's screen, the call fails and says so. A window of the agent's that leaves the display later, as a UTM VM window did, is listed by list_windows with strays: true and place_window places it again. Before, place_window could answer placed for a window that was already on its way back, and then refused it, or asked the user for it as if it were theirs.
• Clicks to remote panels drawn by an app's background service now use the target window's current frame when macOS omits it from a targeted window lookup. Before, the event could carry global coordinates in place of window-local coordinates. The fallback checks both the process and window id; it does not use the visible host's frame.
• A new window that an app the agent started opens by itself, long after the agent's last action, goes to the agent's display when none of the app's other windows is on your screen. Before, it opened on your screen until the agent called place_window. UTM did this on every guest reboot, opening a new VM window each time.
• The Agent Pointer in the Viewer and Peek uses a new original cubic arrow with a rounded cap, shoulder and heel. Uniform light and dark outlines keep the theme accent visible on different backgrounds. Existing movement and click feedback stay shared, and the image boundary clips the pointer and ripple without moving the hotspot. The Mac's system cursor is unchanged.
• An agent's screenshot no longer shows your menu bar's status items, such as the clock, battery and CPU, right after an app on its display quits or a window there changes. macOS moves the same status item windows from your screen onto the Agent Display at those moments, and Agent Screen now checks where they are at each capture.
• An app on an Agent Display that brings itself to the front, as Electron apps do when they open a window, no longer keeps your keyboard. Your app is back in front about 10 ms later. Before, the agent's app stayed in front for 2 to 4 seconds. A click in the Dock or a launcher that opens the app still brings it forward.
• An explicit place_window frame now stays below the reserved menu strip, so an agent can move or shrink a zoomed window whose toolbar became unreachable. The result reports the actual frame and explains when the constraint shrinks a dimension. For a window already on the display, placement without frame arguments keeps its frame. Shared-browser key refusals now describe foreground/focused-window evidence instead of claiming the user is active. Browser drag guidance explains that a fresh profile has separate sign-ins and cannot add a pipe to an existing window.
• Agent Screen no longer quits when you click a field in the Viewer and paste into it during a Takeover. If the Viewer gets redrawn while it is still handling your click or key, it now finishes handling the input first and then applies the new layout.
• computer type checks the field after typing. A page that drops keys arriving faster than its script handles them, as an address autocomplete did, used to lose characters while the call reported success. Now, when the focused field's text lacks some of the typed characters, typing stops, and the call fails with how far it got and what the field holds. Typing into a busy page waits until the page shows the text.
• Viewer clicks can be traced with AGENTSCREEN_VIEWER_CLICK_DIAGNOSTICS=1 at startup. The opt-in stderr records link a press and release to their coordinate mapping, window target, focus handshake and event stamp. They contain no text, window titles, keys or screenshots. Diagnostics are off by default and do not confirm that the target app executed an action.

Install or update with Homebrew: brew tap fanzzzd/agent-screen && brew trust fanzzzd/agent-screen && brew install --cask agent-screen, or brew upgrade --cask agent-screen. Or download the DMG below. It's signed with a Developer ID and notarized by Apple.