Skip to content

v0.12.0

Choose a tag to compare

@sheehanmunim sheehanmunim released this 06 Nov 03:55
· 22 commits to main since this release

Browser: edit pages fully in a background tab (#61)

An agent tab can now be edited end to end while it stays in the background. The browser tools never select the tab, raise Chrome's window, or send OS keystrokes.

  • Real input in background tabs. While the agent holds a tab, its DevTools session emulates focus, so Chrome keeps rendering the tab and real input reaches it. browser_click presses with trusted mouse events, which frameworks that ignore synthetic clicks (Angular Material buttons, for example) respond to. Page events remain only as a fallback, and the reply says which route ran.
  • browser_press_key takes any key plus modifiers (cmd, ctrl, alt, shift): Delete, arrows, Home/End, PageUp/PageDown, F1–F12, characters, and shortcuts such as key: "a", modifiers: ["cmd"] to select all. Editing shortcuts (select all, copy, cut, paste, undo) mean the same with cmd or ctrl on every platform. Browser shortcuts such as close-tab never reach Chrome itself. The reply says where focus is and how many characters are selected.
  • New browser_fill replaces a field's whole value by its snapshot index: input, textarea or contenteditable, including inside web components (open shadow roots). It selects everything and inserts the new text through Chrome's text input, so the page sees trusted beforeinput/input events. An empty string clears the field. The reply says whether the field now holds exactly the text, and what it holds when the page limited or refused it.
  • Snapshots list table and grid rows by their cell text (row "Cara | Owner"), plus cells and other elements that are click targets of their own (a custom-element row with a pointer cursor). Tabs, checkboxes, radios, switches and tree items are listed too.
  • The server instructions and tool descriptions now say never to use the desktop click, press_key or type_text on a Chrome agent tab: those go to whichever Chrome window is in front, which can be the user's.

While the agent holds a tab, that page sees itself as visible and focused, and Chrome renders it like a shown tab. That ends when the tab is released or closed.

Clicks never land on the wrong thing (#62)

In Google Play Console's permissions dialog, a click on a checkbox scrolled out of view changed the Admin (all permissions) checkbox instead. Every checkbox had been listed as "on", and a covered target still got a synthetic press.

  • browser_click scrolls the element into view and presses only where the element itself, or its own label, is on top. When a banner, dialog or another control covers it, nothing is pressed and the call fails, naming what is in front.
  • Checkboxes, radios and switches are named by their label (aria-label, aria-labelledby, <label>, or the text beside a custom checkbox), never by their value. Snapshots show their state: input[checkbox] "Admin (all permissions)" (unchecked), (checked, disabled). A checkbox click says what the box is now. A wrapper around a listed control is no longer listed beside it, so indices stay put.
  • Desktop click by element_id checks the point first. When the click has to go to a position (web content, say), the app's own hit test checks it first, on macOS and Windows, after scrolling the element into view if needed. If something else is there, the call fails instead of clicking a stale point. On macOS, a checkbox or radio button that did not change after the click is reported as an error.
  • Type alongside the agent on macOS. To deliver keys or a click to a background app, the server lends that app keyboard focus for the length of the action. Keys you press in that moment no longer end up in the agent's window: they are held and replayed, in order, to your own window once it has focus again. Windows already worked this way. Set MUNIM_COMPUTER_USE_HOLD_USER_KEYS=0 to turn it off.

34 tools on every platform. Update the Chrome extension too: reload the unpacked extension or install the new munim-computer-use-chrome-extension.zip. The background-input changes live in the extension.