Skip to content

browser actions

Dr K edited this page Nov 14, 2026 · 4 revisions

Browser actions

Every call an agent makes to a page takes one road, core.recipe.Recipe.run, and ends with the same report of where the page is. This area holds what the actions share: selectors, waits, the optional url, clicks and what received them, the pointer, keys, frames, dialogs, single-page apps, scripts, uploads and the wording of a failure. Each tool's arguments and result fields are generated from its docstring into the wiki's docs/<tool>.md page and are not restated here.

Use cases and neighbours

  • An agent clicks a link that exists twice, once hidden: the visible one is clicked and the result says which. A script waits with assert, and a flow uses the same action as a step.

ACT-1. The browser a call reaches, its turn and the reopen are in workspaces-and-sessions.md; the capability table, failure statuses and the screenshot's MCP shape in surfaces.md; refusals under a person's control in live-view-and-control.md; typing a secret and the scrub in secrets-and-scope.md. open_session, save_site_data, print's file and network_body are in session-settings.md, site-data.md, workspace-files.md and capture.md.

The recipe

ACT-2. navigate, interact, drag, write, press_key, extract, execute_script, assert, outline, screenshot, frame, resize, dialog, upload_file, print and save_site_data run Recipe.run. In order it checks the selector and the url, takes the session's turn (WS-39), reattaches, goes to url when one was named and the browser is elsewhere, waits for the element, runs the action's body, and reads where the page is.

ACT-3. What a caller can get wrong is refused before the browser is touched, so a refusal costs no page load:

  • a bad selector or url scheme;
  • an unknown mouse, frame or dialog action, print format or key;
  • send_text without text;
  • a drag with no destination, with both a destination and an offset, or with an offset of zero;
  • an upload with no source or several;
  • an assert whose stable_for is not shorter than wait_timeout.

ACT-4. The result is the body's fields, then matched and index (ACT-11), then the page state (ACT-12), later keys replacing earlier. The page is read once.

ACT-5. An element that goes stale between the wait and the act makes the wait and the body run once more, only once, for interact, write and press_key with a selector. A cancelled call, or one whose browser is ended, stops at its next poll and answers as ended (WS-40).

Selectors

ACT-6. A selector is one object with exactly one of xpath or css, a non-empty string. A JSON string is parsed, which a multipart form needs. Id, name, tag, class and link text are reachable through those two.

One naming both, neither or another key is refused with a sentence that names what was given and shows an example, and so is a value that is not an object. There is never a fallback from one to the other.

Which actions take one:

  • interact, write, extract and upload_file require one.
  • drag requires one for the source and takes the destination as to or by_x/by_y.
  • press_key, outline and screenshot take an optional one.
  • frame takes one for switch as an alternative to index.
  • The rest take none.

Waiting and which match

ACT-7. An action waits for its element in one of three ways, polling every 0.2 s.

Wait Satisfied by Used by
Clickable The first match that is displayed and enabled click, double_click, right_click; drag's source; write; press_key with a selector
Shown The first displayed match, else the first hover, scroll_to (a clickable wait would refuse the off-screen element it is for); drag's to
Presence The first match, displayed or not upload_file, extract, frame, outline, screenshot of an element

ACT-8. wait_timeout is seconds, 30 by default (10 for dialog); a value that is not a whole number is the default. A clickable or shown wait asks about the first 50 matches per poll, since each is_displayed is a round trip inside the turn, and skips a match that goes stale. When none is usable a clickable wait keeps waiting and a shown wait settles on the first match.

ACT-9. A timeout reads no clickable element matched '<value>' within <n>s (no element matched for the other waits), then The browser is at '<url>'; if that is not the page you expected, the wait is not the problem.

Then, for a clickable wait only, what the page says:

  • with no match: Nothing on the page matches it.;
  • with one: that element's reason (ACT-10);
  • with several: <n> elements match it, and none could be used: and the first five as [i] <reason>, and <n> more, Only the first 50 are tried; narrow the selector. past 50, and the move of the first listed match that has one.

The page's answer never replaces the timeout.

ACT-10. The reasons an element cannot be used, tried in this order. One page script gives them to the timeout, a click's intercept and outline. What is on top is read from the whole stack at the click point (elementsFromPoint):

ACT-11. When more than one element matched, interact, drag, write and press_key return matched (how many) and index (which, from 0), and never when one matched. A selector matching several never fails a call by itself and is never resolved blindly.

Where a call ends

ACT-12. Every page action returns the top page's url and title, from inside a frame too. An open dialog does not fail a call that otherwise succeeded: url and title are null, dialog holds the message and hint says to answer it with dialog.

ACT-13. write(submit=true) and press_key with Enter or Return wait for a navigation the keystroke may have started, since sending keys carries no promise to wait: up to 2 s for the focused element to go stale, then for the new document to stop loading. Expiring is ordinary, because a form handled in JavaScript never navigates. interact and navigate wait through WebDriver.

ACT-14. A single-page app sets its title after the URL changes. When a call moved the page to another screen within the same document and the title is still the one the call began with, it is waited for up to 1 s. A full load, a frame, a dialog, a failed read, a title already changed and a move that left the screen as it was never wait. A screen is the path and the hash route with its own query; a change to the URL's query, to an anchor or to a non-route fragment is the same screen, so a search box mirroring into ?q= stays fast.

The optional url

ACT-15. interact, drag, write, press_key, extract, execute_script, assert, outline, screenshot, print, upload_file and save_site_data take an optional url, which is not an assertion: when the browser is elsewhere it navigates there first, and on that page it does nothing. navigate's url is required and always loads. frame, resize and dialog take none.

ACT-16. Two URLs are one page when their host, port, path, hash route, query, route query and userinfo are equal (urls.same_page). The host compares without case, a default port written out equals none, and a trailing / on the path does not count. A hash route is a fragment starting / or !/, cut at its own ?, without a trailing /, the ! dropped; any other fragment (#top, #access_token=...) is not part of a page. The root route counts as no route unless it carries a query, so /#/ and / are one page. A URL with no page key (about:blank) compares without its fragment or trailing slash.

ACT-17. A caller may open http, https and about:blank. Any other scheme (data:, file:, javascript:, view-source:) is refused naming the scheme and never the URL, which may carry credentials.

ACT-18. Every page a caller names loads through browser.go. A host that does not resolve and a certificate the browser does not trust are 400s worded as ERR-17 says, in place of the browser's own text, which quotes the URL.

Clicks, the pointer and what received them

ACT-19. interact takes click, double_click, right_click, hover or scroll_to. click is WebDriver's element click, which refuses a covered target; double_click and right_click are pointer actions that press whatever is at the centre. All but scroll_to move the pointer onto the element first and act second, so afterwards it is where a person's would be.

ACT-20. For the three that press, a script runs at the click point after the move and before the gesture. The point is the in-view centre of the target's first client rect, which is the point WebDriver clicks.

When the topmost element there is the target or inside it, the result has received_by {element, depth, painted}, depth 0 being the target. It also has a warning when all of these hold:

  • the receiver is at depth 1 or more;
  • it is unpainted or deeper than 2;
  • neither it, the target nor anything between is a control.

A selected link or button never warns.

ACT-21. A double_click or right_click whose receiver is outside the target is reported, never refused: received_by has depth null and a warning says what took it and why, with its checked selector.

  • A click whose receiver is outside is WebDriver's to refuse, and if it lands nothing is reported.
  • hover and scroll_to report no receiver.
  • A failed probe leaves its fields out and never fails the gesture.

An intercepted click raises the driver's first line, then, when several matched, The selector matched <n>; this was [<i>], the first that could be used., then the reason (ACT-10) of the element that was clicked, not of the selector's first match.

ACT-22. An element is painted when no ancestor has opacity 0, it is not visibility:hidden, and one of these holds:

  • it is a replaced element (img, video, canvas, iframe, embed, object, input, textarea, select);
  • it has a background colour with alpha above 0, a background image, a non-blank text node of its own, or a ::before or ::after that draws;
  • it is an SVG element other than <svg> with a fill or stroke that is not none or transparent.

Borders and shadows do not count.

ACT-23. The pointer report has glided when a move was sent, nudged when the pointer was already inside the target and glide_note when a requested glide was a jump. A move onto a target the pointer is already inside first steps out to a point near it, so mouseover fires. glide is false for interact and true for drag: a straight eased path of 6 to 40 steps before WebDriver's own element move, a jump when the start is not known.

ACT-24. The pointer's position is kept per Grid session id, in memory or in Redis to match the workspace store.

  • It is forgotten when a browser opens, when end_browser runs and when a move failed part way.
  • A store that fails means an unknown position and fails nothing.
  • A move WebDriver will not make (an element taller than the viewport) costs only the position, and the gesture runs as WebDriver does it.
  • A gone browser or an unreachable Grid still fails the call.

ACT-25. drag presses at the source's centre, holds 120 ms, travels, holds again and releases; the approach is a jump. The destination is the first displayed match of to, or the landing point moved by by_x and by_y, read after the approach because scrolling moves every rect. One outside the window is clamped to its edge and clamped says where. The result has from, to and glided.

Typing and keys

ACT-26. write waits for a clickable element, clears it unless clear is false, types text and returns value, the field's value read back.

  • A password field is never read back, nor is a bound secret's field: with the read turned off it is not made (SEC-22).
  • A bound write into an element that hosts another document is refused (SEC-20).
  • submit=true reads the value first, because submitting makes the element stale, then sends Enter and settles (ACT-13).

ACT-27. press_key with a selector waits for a clickable element and sends the key there. With none it sends to the document's active element, <body> when nothing has focus, because sending to <body> itself takes focus off the element that had it.

ACT-28. A key is one of:

  • one character;
  • a name in any spelling (ArrowLeft, arrow_left and arrow-left are one key);
  • a combination joined with + such as Control+a, where a + after a + is the plus key. A combination is one sequence, each modifier held for the keys after it.

An unknown name is refused naming it and listing every key name.

Frames, dialogs and the window

ACT-29. frame takes switch, parent or default; switch needs a selector or an index, and a selector wins when both are given. The frame the session is in is state on the Grid: it lasts across calls until something switches back, in_frame reports it, and every locator looks in that frame only. in_frame is window !== window.top, never window.self, which a page can forge with self = top.

ACT-30. dialog takes accept (the default), dismiss, read or send_text. It waits up to wait_timeout for a dialog, reads its message before answering, and returns action and message. read leaves the dialog open. send_text types text and accepts. resize sets the window from width and height, keeping one left out, and returns the size the window has; it is the one action that changes what the workspace stores, so a reopened browser has that size.

Reading the page

ACT-31. extract waits for presence and returns text, what a person sees, and html, the element's innerHTML. No password input's value attribute is in html, in a <template> too (SEC-65).

ACT-32. outline returns elements, count and total.

  • By default the elements are the actionable ones: links, buttons, form controls, summary, label, iframe, and anything with an onclick, a tabindex, contenteditable, an ARIA state or a widget role. interactive=false lists every element.
  • text keeps those whose name contains it, ignoring case.
  • limit (default 50, never negative) cuts the list while total still counts what matched.

ACT-33. Content comes before navigation, banner, footer and sidebar chrome, each entry carrying its region. An entry has role, name, visible and a selector checked to match exactly that element. The selector is the first of these that fits:

  1. #id;
  2. data-testid, data-test, data-qa, name, aria-label, placeholder, title or href, when unique;
  3. normalize-space() XPath on its own text;
  4. a full CSS path.

A not-visible entry has reason (ACT-10), blocked_by when the reason names something, and for a hidden one with a found trigger revealed_by and open_with. expanded appears with aria-expanded.

ACT-34. A name comes from aria-label, aria-labelledby, an image's alt, a form field's label, placeholder, button value or name, and, for the roles ARIA names from content (button, link, tab, option and the like), its text. Anything else is named only from text it owns, where at most one child at each level carries any, so siblings are never run together. A name is cut to 80 characters ending in …, after the scrub.

ACT-35. screenshot returns image (base64 PNG), width, height and bytes.

  • width and height resize the window first and leave it so; full_page resizes to the document and restores. An element is found after the window is sized.
  • Unless save is false the PNG is kept in the workspace as file, and a failed save says why in file_error and never costs the image (FILE-16, FILE-17).

print makes a PDF (with landscape and background, off by default) or an HTML file (the document as it stands), keeps it in the workspace's Files and returns file, format and bytes (FILE-18).

Scripts and assertions

ACT-36. execute_script runs the script with no arguments and returns result. One that runs past the session's script timeout is a 400 worded as ERR-17 says, since Selenium's bare timeout reads as a failed element wait.

ACT-37. assert runs a script that must return a boolean; any other value is refused naming its shape (null, a string of 3 characters) and never its content. It asks again every 0.2 s until the answer is true or wait_timeout passes, wait_timeout=0 asking once, and an answer arriving after the deadline does not pass.

ACT-38. stable_for (seconds, 0 by default) requires the answer to stay true that long, and the clock restarts each time it turns false. A value that is not a number is refused, never read as zero, which would remove the guard. The result has asserted, script, the page as it was when the answer held, and stable_for when one was asked.

ACT-39. A failed assertion says the caller's message, or assertion failed after <n>s on '<url>'. Give the step a message to say what should have been true., or became true on '<url>' but did not hold for <n>s when it had been true. The script is never echoed, since it can carry a literal.

ACT-40. upload_file takes exactly one source: text, content (base64) or file, a workspace://files or session://downloads address. FILE-20 and FILE-21 say how the source is checked and how the bytes reach the browser, and FILE-12 to FILE-14 how the name the page sees is made.

Failures

ACT-41. A failure is one line a model can act on (ERR-14). It names the fix when there is one: the selector shape, the key names, dialog for an open dialog, insecure=true for an untrusted certificate. A selector that never matched, an intercepted click, a refused argument, a failed assertion, a script timeout, an unresolved host and an untrusted certificate are the caller's, a 400 (SURF-41). A Grid node that stops answering mid-call is NodeLost (ERR-17).

One table

ACT-42. Each action is one row of core.capabilities.CAPABILITIES (SURF-2) and one method of core.actions.Actions, named alike except assert (assert_) and print (print_). A tool, a route and a flow step end in that method through Workspaces.act, so all three get the same answer. The page scripts are files in js/, concatenated into one string because execute_script has no module loader. A click's receiver, a timeout's reason and outline share one helper file, so the error and the map cannot disagree about one element.

Proposed: several sessions in a workspace

Not built. The reasoning, and the questions still open, are in the multiplayer design.

ACT-43. (proposed) Every session can switch between its tabs and follow the popup windows its pages open, whether or not another agent shares it.

ACT-44. (proposed) An action reaches the browser over one of two transports, classic WebDriver or BiDi, chosen as auto, classic or bidi. Classic stays for every action, and BiDi is used where it is needed. Where the choice is set, beyond run_flow (FLOW-50), and what auto picks for each action are open; the BiDi action layer has a design of its own.

Clone this wiki locally