Skip to content

v0.7.2

Choose a tag to compare

@sheehanmunim sheehanmunim released this 18 Oct 23:13
c2ad457

Browser errors that say what to do, one coordinate space at any display scale, and choosing the window

Windows: screenshots and clicks agree at every display scale (#31)
The server was not DPI aware, so at 125% scaling a point seen at screenshot pixel (600, 550) had to be clicked at (480, 440). It now runs per-monitor DPI aware: screenshots, element bounds, clicks and window lookups all use physical pixels, and the screenshot mapping line can be used as is. Background clicks into older apps that are not DPI aware are scaled into their own coordinates. Clicks on OCR text had the same offset and are fixed too.

Windows: no clicks on controls that are off screen (#32)
A control can be listed with a position far outside its window. Google Docs' hidden "Edit document" input sits near y = -9850. Clicking one sent the real cursor there, Windows clamped it to the screen edge, and the result still said "clicked". Such a click is now refused: the error says nothing was clicked and keyboard focus did not move, and suggests type_text or set_value with element_id, which focus the control through accessibility. Coordinates that are off every display are refused the same way.

Windows: activate one specific window (#33)

  • get_app_state window lines show a window_id, which stays the same as long as the window exists.
  • activate_app takes window_id to bring that one window forward, for an app such as Chrome where several windows share one process.
  • activate_app, type_text and press_key say which window is in front or received the keys.
  • activate_app no longer un-maximizes a maximized window.

"the extension disconnected" now says which link broke (#28)
When several MCP servers run, the first one owns the connection to Chrome and the others send their browser calls through it. If that owner had no connection from Chrome, the others got a bare the extension disconnected with nothing to act on. A browser call that cannot reach Chrome now says:

  • which link is down: this server has no native host, the native host dropped during the call, the owning server exited, or the owning server itself cannot reach Chrome;
  • the pid of the server that owns the connection (and its version, when it differs), and how long Chrome's native host has been away;
  • that a session_id is never the cause: sessions only keep their own tabs and share the one extension connection;
  • the next step: retry once (checking with browser_snapshot before repeating anything unsafe), end a stale or sandboxed owner, reload the extension at chrome://extensions, or run munim-computer-use check-native-host.

Servers of different versions keep working together.

Two instances of one app (#29)
Two Chrome profiles, for example, share a name, so get_app_state({app: "google-chrome"}) cannot know which one is meant.

  • list_apps shows each instance's main window title when several share a name, and ends with a note listing their pids.
  • On Windows and Linux the "matches several apps" error names every candidate with its pid, window count, window title and whether it is in front, so it is enough to choose from even if windows changed since list_apps. On macOS a name still picks the frontmost instance that has windows, and the note says so.

query searches the whole outline (#30)
get_app_state with a query used to read only max_elements elements and then filter them, so a small max_elements hid matches that were on screen and reported none.

  • A query now reads up to 5000 elements; max_elements caps only the matches printed ("N matching elements, showing the first M").
  • A read that runs out of budget is reported as an incomplete search, never as a definitive empty result.
  • An empty on-screen search says off-screen elements were not searched and points at offscreen=true.

Tested on Windows 11 and macOS; cargo test passes on Windows and Linux. 32 tools (activate_app gains window_id on Windows); no extension update needed.