Skip to content

fix(cua-driver): scope the Linux AT-SPI walk to the requested window - #2895

Merged
f-trycua merged 2 commits into
mainfrom
fix/atspi-window-scoped-walk
Aug 5, 2026
Merged

fix(cua-driver): scope the Linux AT-SPI walk to the requested window#2895
f-trycua merged 2 commits into
mainfrom
fix/atspi-window-scoped-walk

Conversation

@f-trycua

@f-trycua f-trycua commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

The bug

The Linux AT-SPI walker took an xid and never used it to scope anything.

atspi/mod.rs::walk_tree(pid, xid, …)native::walk_tree_bounded(pid, xid, …)walk_tree_bounded_with_timeout(pid, xid, …), whose body called collect_visited_bounded(conn, pid, max_elements, max_depth). The xid reached only the geometry pass and the degraded X11-property fallback. The traversal itself seeded from every child of the application accessible.

AT-SPI publishes one tree per process, so this meant every "window-scoped" snapshot on Linux was really application-scoped. For a single-window app the two coincide and nothing looks wrong. For a browser — three windows under one PID is an ordinary Tuesday — get_window_state(pid, window_id) returned all of them, and matching a control by label could find one in a window the caller never named.

Why it surfaced now

Existing-profile browser setup drives Chromium's internal chrome://inspect/#remote-debugging page through accessibility: find the checkbox labelled "Allow remote debugging for this browser instance", toggle it, prove the new loopback endpoint, close the temp tab. With an application-wide tree and several windows open, that label match is not guaranteed to be the window the caller approved — and there can be one such checkbox per open setup page.

setup_existing_profile_endpoint guarded against that with a cardinality check:

if self.is_only_exact_native_window(request.pid, request.window_id).await? != Some(true) {
    return Err(refusal(
        BrowserRefusalCode::BrowserBindingAmbiguous,
        "existing-profile setup on Linux requires exactly one PID-owned native window; \
         generic Wayland and multi-window processes are refused",
    ));
}

Sound, but it made the route unreachable for most real sessions: a second Chrome window of any kind fails it. macOS carries no equivalent precondition (platform-macos/src/browser/platform.rs) because AXUIElement is genuinely window-scoped there — a good sign this was a workaround for the missing primitive rather than a principle.

The fix

Resolve the native window to exactly one application top-level, then scope the walk to it.

  • resolve_window_frame correlates each frame-role top-level's Component.GetExtents(Screen) against the X11 geometry the caller already named. It runs against the same get_children() list the walk is about to seed from — re-reading it could observe a different window set, and an ordinal resolved against one list but applied to another names the wrong window.
  • correlate_frame_to_window scores by geometry distance and requires the winner to be within a tolerance and to beat the runner-up by a margin. Decoration offsets resolve; two same-geometry windows refuse rather than coin-flip.
  • Every Visited node carries the frame_ordinal it descends from, inherited by children.
  • browser_setup_ui now requires that proof before it matches or actuates any control, and the cardinality precondition is gone. Where window identity genuinely cannot be established — a generic Wayland session with no compositor window list — the same proof fails and setup still refuses, with a message that names the cause and points at --remote-debugging-port.

Element indices stay application-wide

This is the load-bearing detail. Indices are the contract between a snapshot and every actuator that later consumes one — perform_action, focus_element, set_value, scroll_element, type_into_editable_at — and each of those resolves an index against the whole application via collect_visited(conn, pid). Renumbering per window would make a window-scoped snapshot's indices name different elements at actuation time, silently actuating the wrong control.

So render gained an only_frame filter that still consumes the index for skipped nodes and only suppresses emission. A window-scoped snapshot shows one window's nodes carrying their application-wide indices, and actuation keeps resolving them correctly with no changes to any actuator.

Verification

Unit — 7 correlation cases (sibling match, decoration offset, ambiguous tie, no match, no candidates, unmapped zero-area extents, sole-candidate-still-has-to-match). Full platform-linux lib suite: 227 passed.

Live, X11, three-window Chrome under one PID:

window before after
52428859 "Sign in – Google Accounts" all 3 frames [0] frame "…Your Chrome"
52429004 "hello – Google Search" all 3 frames [1] frame "…Francesco 2"
52429010 "Cua: Scale computer fleets…" all 3 frames [2] frame "…Francesco 1"

Each window returns only its own frame, with application-wide indices preserved. browser_prepare with strategy=existing_profile no longer refuses on cardinality and proceeds to drive the setup UI.

Live, two-window GTK process (one Mousepad PID, editor + dialog):

  • dialog window → 2 elements, indices [0] "No", [1] "Yes"
  • editor window → 257 elements, starting at index [2]

Before this change both calls returned all 259 nodes. Then click by element_token minted from the window-scoped dialog snapshot dismissed the dialog — confirming actuation still lands on the right element through the app-wide index space.

Not fixed here

Existing-profile setup still cannot complete against this Chrome instance, for a separate and pre-existing reason: Chrome publishes no renderer content to AT-SPI at all (every window walks to a bare frame with zero children), so the setup page's controls never appear. That is the condition setup_not_ready_message already documents — Chromium needs --force-renderer-accessibility, or an AT client that enables full renderer accessibility. Unrelated to window scoping, and it reproduces identically on windows the setup flow never touched.

Cross-platform e2e coverage for these browser paths, including non-active tabs and multi-window PIDs, is tracked in #2892.

Refs #2892

🤖 Generated with Claude Code

https://claude.ai/code/session_017zpeGjLTLSALtHsuxKLkNp

AT-SPI publishes one accessibility tree per process, so a multi-window
application exposes every window's controls together. The Linux walker
accepted an `xid` and threaded it as far as `walk_tree_bounded_with_timeout`,
but `collect_visited_bounded` seeded the traversal from every child of the
application and never used it — the xid only reached the geometry pass and
the degraded X11-property fallback. Every "window-scoped" Linux snapshot was
really application-scoped, and matching a control by label could find one in
a window the caller never named.

Existing-profile browser setup papered over this by refusing whenever the
browser process owned more than one native window, since single-window
processes are the case where application scope and window scope coincide.
That made the whole route unreachable for ordinary sessions: any second
Chrome window — a search window, an OAuth popup, a second profile — was
enough to fail it.

Resolve the native window to exactly one application top-level instead, by
correlating `Component.GetExtents` against the X11 geometry the caller
already named. The correlation refuses ties rather than guessing, so callers
that act on behalf of an exact window get a proof or a refusal. Setup now
demands that proof and drops the cardinality precondition; where window
identity genuinely cannot be established (a generic Wayland session with no
compositor window list) the same proof fails and setup still refuses, now
with a reason that names the cause.

Element indices deliberately stay application-wide. They are the contract
between a snapshot and every actuator that later consumes one
(`perform_action`, `focus_element`, `set_value`, …), and those resolve an
index against the whole application; renumbering per window would make a
window-scoped snapshot's indices name different elements at actuation time.
`render` therefore filters what it emits while still consuming the index.

Verified on X11 against a three-window Chrome process (each window returns
its own frame, indices 0/1/2 preserved) and a two-window GTK process (2-node
dialog and 257-node editor returned separately, and a click by token from a
window-scoped snapshot lands on the correct element).

Refs #2892

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017zpeGjLTLSALtHsuxKLkNp
…#2899)

Existing-profile setup reached its `chrome://inspect` page by synthesizing
the URL one keystroke at a time: split the fixed URL at `/#`, send the base
through XTEST, press Enter, then hunt for a "Remote debugging" control to
cover the fragment. XTEST keysym lookup is keyboard-layout dependent and
wlroots virtual-keyboard seats drop punctuation, so `chrome://inspect` could
arrive at the omnibox as `inspect` — which Chrome submits as a search query.
Nothing downstream distinguished that from a slow-loading setup page, so the
run ended in a readiness timeout while leaving the user on a search-results
page for "inspect".

Adopt the macOS and Windows strategy: transfer the whole URL in one
operation instead of per-character. Those adapters write it through the
accessibility API (`AXValue`, `ValuePattern::SetValue`), which has no working
counterpart here — Chromium's AT-SPI bridge refuses EditableText writes on
the omnibox — so the transfer goes through the clipboard, which preserves the
property that matters: the exact string arrives at once, with no keysym
synthesis to mistranslate. The user's clipboard is saved and restored. The
only synthesized keys left are ctrl+t, ctrl+l, ctrl+a, ctrl+v and Enter, all
plain letters carrying no layout dependency.

Verification is against the destination rather than the input field. Chromium
exposes no readable text on its omnibox over AT-SPI — no Value interface, and
no Text content even while the field holds a URL — so the read-back those
adapters perform cannot be ported. Proving the tab reached the fixed setup
page is stronger anyway: it catches a mistyped URL, a hijacked search and a
redirect alike, and it reports what was actually reached instead of timing
out with an unrelated message.

Verified against Chrome 151 on X11 with a real user profile: setup now
reaches chrome://inspect/#remote-debugging, matches the exact per-instance
checkbox, toggles it, and closes its temporary tab — the whole navigation
sequence that previously never got past the omnibox.

Not fixed here: after the checkbox is enabled, Chrome 151 exposes no loopback
endpoint at all (no listener for the browser pid across 30s, no
DevToolsActivePort), so browser_prepare still refuses with
browser_requires_setup and rolls the toggle back. That is downstream of
navigation and looks like it needs a browser restart to take effect.

Refs #2892


Claude-Session: https://claude.ai/code/session_017zpeGjLTLSALtHsuxKLkNp

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant