Repository navigation
Replies: 2 comments 2 replies
|
A useful negative-control test for this contract would be a SPA that keeps the same For iframe targets, could the URL check's scope be explicit? An expected top-level URL and an expected target-frame URL can express different requirements. A fixture with an approved top-level page and a different frame URL would make the intended comparison clear, particularly for the focused-element keyboard commands. I maintain a Windows browser SDK. This is design feedback prepared with AI assistance, not a reproduction or test result for agent-browser. |
|
+1 for point 1 (exact refs). A concrete case where the fallback crosses a document boundary on v0.43.0 is in #1922: a ref issued for page A resolves to the same-named button of page B after a click-triggered navigation. #1923 would close that case; exact refs would also cover the same-document one, where a caller that gates each action cannot tell that the node was replaced. From outside the daemon the only check we found is a second |
Uh oh!
There was an error while loading. Please reload this page.
Summary
When agent-browser runs inside another tool that has its own approval step ("the user approved clicking Send on this page"), that tool must check, right before the action, that the page and element are still the ones approved. From outside the daemon that check is always a separate call, so the page can change in between. Three optional features would let the check happen inside agent-browser, right before it dispatches:
1. Exact refs (no lookalike fallback)
Today, when a ref's element has been replaced,
click @e5falls back to the first element with the same role, name and position (RefEntry::allow_role_fallback). That is convenient for agents. For an approved action, though, it can act on a different "Delete" in a re-rendered list. Proposal: a session setting such as--exact-refs(orAGENT_BROWSER_EXACT_REFS=1) that makes a stale ref fail withElement ref eN is stale, as it already does foradd_exact_backend_noderefs.2. Preconditions on an action
Optional checks that
click,fill,type,press,select,checkanduncheckrun inside the daemon, just before the input is dispatched. The action fails before dispatch if any doesn't hold:--expect-url: the page's URL (fragment ignored), or that of the frame the element is in.--expect-role/--expect-name: the element's accessibility role and name, read from the AX tree at that moment.--expect-focus-role/--expect-focus-name: for keyboard input without a target, the element that has the focus, in any frame agent-browser can see.--refuse-password: fail if the target, or the focused element for keyboard input, is an<input type=password>.A failed precondition would return a stable error code (
precondition_failed, with which one), so callers can tell it from an action that may have reached the page.3. Password fields reported from the browser's own data
get attr @e typeruns in the page's JavaScript, so a page can overridegetAttributeor thetypegetter. Proposal: mark password inputs in the snapshot itself from the DOM as the browser keeps it (DOM.describeNodeattributes, as--refuse-passwordwould use). For exampletextbox "Password" [password, ref=e7], and"password": truein the--jsonrefs map.Why in agent-browser
All three are read in the same place agent-browser already resolves refs and checks for covering elements (
resolve_element_center,check_node_interception), so the window between check and dispatch stays as small as it already is for those checks. A caller outside can only narrow that window, not close it. The new behaviour is opt-in, so nothing changes for anyone who doesn't ask for it.Context
We drive agent-browser from a desktop app that asks a person to approve some actions before they go out. We hold these rules today through a second CDP connection of our own, alongside agent-browser's, which we'd be glad to drop.
Related: #2574 / #2579 (allowlist and frames).
All reactions