Repository navigation
v0.7.0
Reading apps faster, OCR, and working in minimized windows
Ideas from arc-cua (MIT), ported to macOS and Windows; Linux gets the reading and recheck parts.
Reading (get_app_state)
- Only what is on screen is read. Rows and content scrolled out of view are skipped, and the outline says how many;
offscreen: truelists everything. Each element's attributes come back in one batched call. Finder with 2,000 files: 2.2 s → 0.17 s. A 3,000-row Windows list: 2.2 s → 0.84 s. On Linux, controls on tabs that aren't shown are left out too. - Dialogs scope the outline. While a sheet, alert or modal dialog is open, only its controls are listed, with a note, so the agent can't act on the window it blocks.
- Electron apps (VS Code, Slack, Discord, Spotify…) are asked for their full accessibility tree on macOS, with a note while they're still building it.
- On-device OCR for apps that expose little. Apple Vision on macOS, Windows OCR on Windows: the window's text is listed with ids that
click,hover,right_clickanddragaccept, andtype_texton one clicks it first.ocr: auto(default, only when an app shows no labelled controls),alwaysornever. Text that an element already shows isn't repeated. The image never leaves the machine. On a Mac, the first OCR use after installing a version prepares Vision's model for a minute or so; the result says so.
Acting
- Ids are rechecked before acting. If the element is gone, or its role or name changed (
Count 0→Count 1), the action is refused with "changed since get_app_state" instead of hitting the wrong thing. Typing into a field doesn't invalidate its id. return_statewaits for the app, not a fixed 300 ms. It follows the app's accessibility notifications (WinEvents on Windows) and returns once the app goes quiet, waiting up to 2 s for a sheet sliding in or a page loading. Elements that were already updating on their own (a clock, a spinner, a video) don't hold it up. A typical click returns in ~0.3 s; one with no visible effect waits 0.6 s for a reaction. Linux keeps the fixed pause.- Cmd shortcuts reach background apps on macOS through the app's menu bar, so
cmd+S,cmd+Wand the like work without bringing the app forward. - Minimized windows and hidden apps on macOS are moved, for the moment they need pixels or input, onto an invisible display where the agent can click, type, screenshot and OCR without anything appearing on your screen or taking focus. They go back after 20 idle seconds, when the server exits, or on screen if you bring the app up yourself.
COMPUTER_USE_PARK_WINDOWS=0turns this off. - Minimized windows on Windows: press, toggle and set_value work; pointer and keyboard actions return a clear "restore it first" error.
- Windows safety: a click on a window covered by another app's window is refused rather than landing on the other window, and a Windows Security prompt that blocks all clicks is named in the error.
Tested on macOS (Retina and 1x displays), Windows 11 and Linux (GTK under Xvfb). 32 tools; no extension update needed.