Skip to content

Foxhand helper v0.5.4

Choose a tag to compare

@github-actions github-actions released this 25 Oct 10:04
8e74df1

The tools are reworked so agents can read and act on real pages on the first
try, without working around them. Every change below was checked on live
sites (Spiegel, GitHub, YouTube, Wikipedia, TodoMVC, react-select, MUI,
Lexical, ProseMirror, Slate, CKEditor) as well as in the tests. The names now
follow one pattern, <thing>_<verb>. Agents and prompts that call tools by
their old names need updating.

Buttons the snapshot used to miss

  • Buttons built from plain elements. A <div> the page made clickable
    with script shows as role clickable, from a pointer cursor, a click
    attribute, Google's jsaction, or a React or Vue click handler.
  • Icon-only buttons have names, from their tooltip, SVG title, the icon's
    own name (fa-trash becomes "icon: trash"), a test id or a class.
  • Closed shadow roots are read and acted on, not only open ones.
  • Controls that only show on hover, like the ✕ on a to-do item, are listed
    (marked [on hover]) and can be clicked, although an add-on can't produce
    CSS :hover.
  • Styled checkboxes and radios (a hidden input under a styled label) show
    as the checkbox, with its state.
  • Unlabeled fields are named by the text beside them or, for a custom
    dropdown, the value it shows.

iframes from other sites

  • Cookie banners, sign-in buttons, payment fields and embeds in cross-origin
    iframes
    are part of the snapshot. Their refs work like any other; a name or
    selector that isn't on the page is looked for in its frames; a click at a
    point over a frame lands inside it. page_text and page_find include their
    text.
  • A consent dialog that closes itself after "Accept" is reported as done,
    not as a failure the agent would retry.

Snapshots with a shape

  • Controls are nested under what they belong to: the landmark, dialog,
    form, list item, table row or card. The agent can tell which "Delete" goes
    with which row. Lines that add nothing are folded away.
  • An open modal, or an overlay covering the screen, is listed first and
    flagged in the header, so a long page can't push it out of the snapshot.
  • Repeated buttons say which card they're in: each "Add to cart" on a shop
    page is labeled with its product, each "More actions" with its video, even
    when the cards are plain <div>s.

Typing like a person

  • Every character is a key press: keydown, keypress, beforeinput, the edit,
    input and keyup. Autocomplete boxes open, input masks (phone numbers, cards)
    format, and one-box-per-digit code fields follow the focus the page moves.
  • Rich editors keep the text: Lexical, Slate, ProseMirror, CKEditor and
    editors in same-origin frames. If an editor rejects the text, the result says
    so instead of forcing text in and corrupting it.
  • Clicks land on the element under the pointer, such as the icon inside a
    button, which is often where the page listens.
  • Submitting a form presses its real submit button. A "Cancel" or "Back"
    button placed first in the form is skipped (before, a checkout form could be
    cancelled instead of continued).

Renamed and merged

Before Now
tabs_list, tabs_query tab_query (query: "*" lists every tab)
active_tab tab_check_active
nav_back, nav_forward, nav_reload tab_navigate with go: "back", "forward" or "reload"
tab_duplicate tab_create with duplicateOf
tab_pin, tab_unpin tab_pin with pinned: false to unpin
focus_tab tab_focus
window_list window_query
snapshot_ax page_snapshot
screenshot page_screenshot
act_click, act_type, act_select, act_hover, act_scroll, act_key, act_wait page_click, page_type, page_select, page_hover, page_scroll, page_key, page_wait
act_fill_form page_fill
act_find page_find
bookmarks_search, bookmarks_create, bookmarks_remove bookmark_query, bookmark_create, bookmark_remove
history_search history_query
downloads_list download_query
cookies_for_tab cookie_query
sessions_recently_closed, sessions_restore session_query, session_restore
wait_for_tab_event event_wait
cursor_note thought

tab_move is removed. The thought argument on every tool is gone too: the
thought tool shows a note, and it stays on the next page action within
30 seconds. Otherwise the cursor describes each action itself.

Acting on pages

  • Find elements by what they say. Every page tool takes name (visible
    text or label), with optional role and nth, as well as ref and
    selector. When several elements match, the error lists them.
  • Refs last longer. A ref keeps pointing at its element until the page
    navigates, so one snapshot serves several actions.
  • Results say what changed. Every action reports the elements that
    appeared, text that appeared or went away (an error under a field, "Added to
    cart"), what opened, closed, got checked or selected, new alert and status
    messages, and the new URL or title. The agent rarely needs a new snapshot
    after each step. After typing into a search box it waits briefly for the
    suggestions.
  • A name that isn't found lists the closest ones, including common
    synonyms ("Sign in" finds "Log in"). The name written exactly as given wins
    over looser matches.
  • Clicks at a point. page_click takes x,y, for maps, editors and games
    drawn on a canvas.
  • page_type finds fields by label or placeholder and handles rich editors,
    checkboxes and selects.
  • page_select opens custom dropdowns (comboboxes and listboxes) as well as
    <select>.
  • page_key performs the default action when the page doesn't: Tab moves
    focus, Enter submits, and text keys edit.
  • page_scroll finds the inner scroller of chat apps and can scroll an
    element into view.
  • page_wait waits for things to go (gone: true, for spinners), for the
    page to settle, or for the address to change (urlContains, after a login
    or checkout redirect).
  • Pages that never finish loading (long-polling, endless trackers) no
    longer cost the full wait: tab_create and tab_navigate return once the
    new page is up, with ready: true.
  • Modifier keys in any spelling: ctrl, Control, cmd, Option.
  • New: page_drag for sortable lists, sliders and map panning, and
    page_upload to attach files the agent supplies.

Reading pages

  • page_text comes in chunks. Long pages no longer flood the agent's
    context: follow nextOffset for more. within reads one region.
  • page_text with format: "markdown" keeps headings, lists, tables,
    emphasis and link URLs, and shows form fields.
  • page_snapshot can read one region (within) or only what's on screen
    (viewportOnly), and includes shadow DOM.
  • page_screenshot captures one element or the whole page, as PNG or
    smaller JPEG, at any scale.
  • page_find returns the ref to click for each match and whether it's off
    screen, and finds phrases split across elements ("Add to cart").

Browser data

  • download_query says where files landed: path on disk, size, progress,
    whether the file still exists and why a download failed.
  • New: download_start downloads a URL with the browser's cookies and
    returns the path.
  • Cookies can be written. New cookie_set and cookie_remove (needs
    confirm:true).
  • New: a notebook for the agent. storage_get, storage_set and
    storage_remove keep JSON values in the add-on across pages and restarts.
    With area: "local" or "session" they read and write the page's own
    storage.

Install

Helper app for the Foxhand add-on (formerly WebMCP Controller).

Windows: open PowerShell and paste:

irm https://hamb1y.github.io/foxhand/ps | iex

macOS / Linux: open Terminal and paste:

curl -fsSL https://hamb1y.github.io/foxhand/sh | sh

Already installed? Run the same command again to update: Firefox switches to the
new helper by itself within a few seconds.

Then click Retry in the add-on popup. Manual: download the file for your
system below and run it with install (Windows: double-click the .exe; macOS: run
xattr -c on it first). The webmcp-host-* files are the same programs under their old names.