Skip to content

feat(cua-driver): exact-target macOS background input v1 - #2837

Merged
f-trycua merged 8 commits into
trycua:mainfrom
hami-sh:plan/cua-driver-background-input
Aug 4, 2026
Merged

feat(cua-driver): exact-target macOS background input v1#2837
f-trycua merged 8 commits into
trycua:mainfrom
hami-sh:plan/cua-driver-background-input

Conversation

@hami-sh

@hami-sh hami-sh commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Implements macOS background input v1 per the included plan
(libs/cua-driver/docs/macos-background-input-v1-plan.md).

What this does

Every window-addressed background mutation on macOS now enforces the exact
(pid, CGWindowID) invariant. Immediately before acting, the driver gathers
fresh WindowServer and accessibility facts for the requested window. A pure
decision core either permits the selected route or refuses it with a structured
reason. A same-process sibling window cannot satisfy, receive, or confirm an
action addressed to the target window.

Routes

  1. AxSemantic — exact-element AX actions, allowed only with proven
    ownership, ancestry, and fresh AXWindows membership.
  2. WindowPointer — window-local routed pointer events, additionally
    requiring a visible, non-minimized live frame and an in-frame point.
  3. InsertText / GenericKey — process-scoped keyboard delivery, allowed
    only when the requested window is the process's sole eligible destination.

Window-addressed refusals use explicit codes such as window_not_found,
owner_pid_mismatch, off_space_or_ax_unresolved,
minimized_or_hidden_window, same_pid_keyboard_ambiguity, and
element_outside_target_window. delivery_mode:"foreground" remains the
caller's explicit last resort.

Verification semantics

  • Event-post success is not reported as effect confirmation.
  • Native text readback is bound to the proven target window.
  • Web-content writes remain unverifiable without an independent renderer
    oracle.
  • A narrow per-process coordinator spans fact gathering, dispatch, focus
    restoration, and target-bound verification; independent processes remain
    concurrent.
  • Element reveal is separately gated before AXScrollToVisible, and the
    pointer route is revalidated before wheel dispatch.

get_window_state now reports an additive background_input capability
section from the same fresh facts. Per-process Electron accessibility
enablement keys on (pid, process start time) so reused PIDs cannot inherit
stale enablement state.

Compatibility

  • Gating applies only to window-addressed, non-foreground calls; pid-only and
    foreground behavior is unchanged.
  • press_key and type_text_chars now reject stale element handles instead
    of silently proceeding unaddressed.
  • Argument-shape errors retain their pre-gate precedence.
  • Background window-local pointer calls reject out-of-frame points.

Validation

Local source validation at d19d186c734e59e299477953bbb722c8f296dbff:

  • cargo test -p platform-macos --lib: 308 passed.
  • Focused decision-core and background-mutation regression suites passed.
  • cargo fmt --all -- --check passed.
  • git diff --check passed.

Authoritative exact-head CI at the same SHA:

Native macOS certification used a logged-in Lume VM running macOS 26.5.2 on
arm64. The release binary was built from the exact source tree with embedded
source SHA d19d186c734e59e299477953bbb722c8f296dbff, signed with the established
local Cua Driver test identity, and externally verified against its designated
requirement. Installed binary SHA-256:
aca9547959ba89822395180e90d7b65e98d8f08bdb0379c98807f9c542ae9cf8.
The daemon reported Accessibility and Screen Recording as granted under its
own com.trycua.driver.local identity.

Representative exact-window results while Terminal remained frontmost and the
physical cursor remained fixed at (1033, 225):

  • AppKit background AX click incremented the requested window counter from 2
    to 3.
  • AppKit exact-field background text write was confirmed by AX readback as
    candidate-d19d-actual.
  • AppKit raw background key delivery to a same-PID sibling was refused with
    same_pid_keyboard_ambiguity.
  • Electron background AX click incremented the requested window counter from
    2 to 3.
  • Terminal remained frontmost after each mutation and the cursor position did
    not change.

Known v1 limitations

  • Competing-keyboard-destination count is a WindowServer approximation and
    excludes only AX-proven minimized siblings.
  • Live-stream PiP, broad temporary restore behavior, arbitrary Space movement,
    and game/canvas input synthesis remain out of scope.

@hami-sh
hami-sh requested a review from f-trycua as a code owner August 4, 2026 12:43
@hami-sh hami-sh closed this Aug 4, 2026
@hami-sh hami-sh changed the title Plan exact macOS background input feat(cua-driver): exact-target macOS background input v1 Aug 4, 2026
@hami-sh hami-sh reopened this Aug 4, 2026
@f-trycua
f-trycua force-pushed the plan/cua-driver-background-input branch from f6003b1 to ef637a0 Compare August 4, 2026 19:56
f-trycua pushed a commit to hami-sh/cua that referenced this pull request Aug 4, 2026
Bind press-key verification to the requested window and route the remaining window-addressed semantic and pointer tools through the same exact-target gate. Background double-click chooses one actuator class instead of silently crossing from AXOpen into an ungated pointer fallback.

Salvaged from trycua#2837

Co-authored-by: Hamish Bultitude <18391419+hami-sh@users.noreply.github.com>
@f-trycua

f-trycua commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Maintainer refresh at ef637a057266d41a2e66358aa5182bacb98031a5:

  • Rebases the contribution onto current origin/main and preserves Hamish as author plus the linked Amp coauthor identity and source-SHA provenance.
  • Fixes the press_key compile regression and binds its AX postcondition oracle to the requested window.
  • Routes set_value, dedicated right_click, double_click, and scroll through the exact-target gate. Background double-click now chooses one actuator class and cannot silently fall through from failed AXOpen to an ungated pointer.
  • Local: cua-driver-core 495 passed; platform-macos 300 passed; cua-driver passes except the existing host-state failure in protocol_tools_call_test::type_text_chars_tool because TextEdit owns multiple stale windows and the mainline PID-only ambiguity guard refuses before this change runs; formatting is clean.
  • Local contributor-attribution and release-title validators pass.

Required near-merge Lume E2E matrix (not run yet):

  1. Two-window AppKit, A targeted/B key: AX click and set_value mutate only A; B unchanged; frontmost pid, key window, z-order, and hardware cursor unchanged.
  2. Two-window AppKit, A visible/occluded: routed click, right-click, double-click, and scroll land only in A; B unchanged; focus/z-order/cursor preserved.
  3. Two-window AppKit keyboard: type_text, type_text_chars, press_key, hotkey, and key-scroll refuse same_pid_keyboard_ambiguity; both window journals and values remain unchanged.
  4. AppKit refusal states: minimized/hidden A permits exact semantic AX but refuses pointer/keyboard; off-Space/AX-unresolved, stale ID, and foreign owner refuse before dispatch with no leakage.
  5. Two-BrowserWindow Electron fixture: per-lifetime AX enablement materializes both exact windows; semantic AX and routed pointer affect only A; every PID-keyboard route refuses ambiguity; B/focus/z-order/cursor remain unchanged.
  6. Electron relaunch: repeat rows 3 and 5 after process restart to prove the process-lifetime enablement cache does not suppress re-enablement.

Each row should record the fixture case ID, before/after values and journals, frontmost pid, key-window CGWindowID, ordered window IDs/z-order, cursor position, structured result/refusal, exact driver SHA/version, macOS version/arch, and fixture build identity.

One design item remains for review before merge: the checked-in plan requires a per-PID mutation coordinator, while this v1 still documents gate-to-dispatch TOCTOU as a limitation.

f-trycua pushed a commit to hami-sh/cua that referenced this pull request Aug 4, 2026
Hold a narrow per-process coordinator from fresh exact-target fact gathering through dispatch, focus restoration, and target-bound verification. Nested focus clicks use task-local lease proof so callers cannot bypass gating, while independent processes remain concurrent.\n\nSalvaged from trycua#2837\n\nCo-authored-by: Hamish Bultitude <18391419+hami-sh@users.noreply.github.com>
@f-trycua

f-trycua commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Maintainer update at exact head 53319cf1fb99f467ab4ab2a1d3a622c6f79e9a64:

  • Added the plan-required narrow per-PID background-mutation coordinator. The lease is held from fresh exact-target fact gathering through dispatch, focus restoration, and target-bound verification; mutations for different PIDs remain concurrent. Nested focus clicks use task-local, PID-bound lease proof rather than a caller-supplied marker.
  • Focused concurrency coverage proves same-PID serialization, different-PID independence, and nested proof scoping.
  • Exact-head local verification: platform-macos 303/303; cua-driver-core 495 unit + 2 contract + 3 lifecycle; fmt and diff checks clean. Full cua-driver again reaches only the documented host-state failure in protocol_tools_call_test::type_text_chars_tool because the resident TextEdit PID owns multiple stale windows and main refuses it as ambiguous_window_target before this PR's dispatch path.

Required exact-head Lume evidence matrix (run the source-built driver at the SHA above and record tool journals plus before/after focus, key window, z-order, cursor, and both window values):

  1. Two-window AppKit, same PID: start overlapping mutations for A and B; prove B does not gather or dispatch until A has restored focus and completed target-bound verification. Repeat with order reversed.
  2. Two-window AppKit, different PIDs: synchronize two mutations behind a barrier and prove both can enter their mutation sections concurrently while each changes only its addressed window.
  3. Two Electron BrowserWindows, same PID: repeat the serialized A/B case for semantic and pointer routes; PID keyboard routes must refuse same_pid_keyboard_ambiguity without dispatch.
  4. Electron relaunch: repeat after PID/process-lifetime rotation to prove AX enablement and coordinator ownership do not leak across lifetimes.

For every case, capture exact SHA/version, guest OS/arch, fixture build, monotonic gather/dispatch/restore/verify timestamps, structured result, and unchanged sibling-window state. This extends the previously posted AppKit/Electron two-window matrix with the coordinator-specific concurrency oracle.

hami-sh and others added 8 commits August 4, 2026 16:46
Amp-Thread-ID: https://ampcode.com/threads/T-019fcc3e-0497-718a-bfdc-69ff23c99228
Co-authored-by: Amp <289058786+ampagent@users.noreply.github.com>
(cherry picked from commit 288f5fd)
…sh AX acquisition

Add cua_driver_core::background_input, a pure decision module that enforces
the (pid, CGWindowID) invariant for every macOS background mutation: routes
are AxSemantic, WindowPointer, InsertText, and GenericKey; facts are gathered
fresh (WindowServer ownership, AXWindows membership, minimized/hidden as
Option<bool> failing closed on unknown, competing same-pid keyboard
destinations, addressed-element ancestry); refusals are structured
(code/reason/advice) and never fall through to another window of the same
process. Includes the two-window wrong-target regression test.

platform-macos gains ax::exact_target (fresh fact gathering, element window
ancestry proof, window-bound focused-element reader) and ax::enablement
(per-(pid, process-start-time) Chromium/Electron AXManualAccessibility
enablement with a one-time settle), which tree.rs now reuses instead of its
process-lifetime-unaware pid set.

Amp-Thread-ID: https://ampcode.com/threads/T-019fcccf-9ffb-70fa-913c-76da7f968784
Co-authored-by: Amp <289058786+ampagent@users.noreply.github.com>
(cherry picked from commit b6a3eec)
Every window-addressed background mutation now proves exact delivery to the
requested (pid, CGWindowID) before anything is sent, and refuses with a
structured reason instead of escalating or acting on a same-process sibling:

- click: element branch gates as AxSemantic (middle-button fallback and the
  internal selection pixel fallback at the stricter WindowPointer rung); px
  branch gates as WindowPointer after coordinate translation, refuses
  out-of-frame window-local points, and requires the pid-scoped AX hit-test
  element to prove ancestry in the requested window.
- type_text: background keyboard policy decides once per call — full ladder,
  semantic-AX-only (exact element, no CGEvent fallback), or refusal; the AX
  write and read-back verification are window-bound (focused element
  reacquired inside the target window only), and web-area classification
  fails closed when the window-bound focused element cannot be proven.
- press_key/hotkey: GenericKey gate before the px focus click; PID keyboard
  is only permitted when the target is the process's sole eligible key
  destination.
- type_text_chars: always-background InsertText gate; refusal is final.
- press_key/type_text_chars now error on a stale element_index instead of
  silently downgrading to an unaddressed gate, and argument-shape errors are
  reported before any gating.

delivery_mode:"foreground" remains the caller's explicit last resort and is
not gated.

Amp-Thread-ID: https://ampcode.com/threads/T-019fcccf-9ffb-70fa-913c-76da7f968784
Co-authored-by: Amp <289058786+ampagent@users.noreply.github.com>
(cherry picked from commit e8fca36)
Additive structured["background_input"] section computed from the same fresh
exact-target facts that gate every background mutation, reported per route so
an agent can pick a viable route before acting (advisory only — every action
still revalidates). The AxWindowUnresolved escalation now recommends
observation/foreground instead of background px, which is refused in that
state.

Amp-Thread-ID: https://ampcode.com/threads/T-019fcccf-9ffb-70fa-913c-76da7f968784
Co-authored-by: Amp <289058786+ampagent@users.noreply.github.com>
(cherry picked from commit f6003b1)
Bind press-key verification to the requested window and route the remaining window-addressed semantic and pointer tools through the same exact-target gate. Background double-click chooses one actuator class instead of silently crossing from AXOpen into an ungated pointer fallback.

Salvaged from trycua#2837

Co-authored-by: Hamish Bultitude <18391419+hami-sh@users.noreply.github.com>
Hold a narrow per-process coordinator from fresh exact-target fact gathering through dispatch, focus restoration, and target-bound verification. Nested focus clicks use task-local lease proof so callers cannot bypass gating, while independent processes remain concurrent.\n\nSalvaged from trycua#2837\n\nCo-authored-by: Hamish Bultitude <18391419+hami-sh@users.noreply.github.com>
Require exact-window semantic proof before AXScrollToVisible for element-addressed scrolls in every direction. Retain the mutation lease and independently revalidate the pointer route before wheel delivery.

Co-authored-by: Hamish Bultitude <18391419+hami-sh@users.noreply.github.com>
@f-trycua
f-trycua force-pushed the plan/cua-driver-background-input branch from c0d59c9 to d19d186 Compare August 4, 2026 22:17

@f-trycua f-trycua left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the exact-target decision core and platform dispatch paths. The scroll reveal ordering defect was fixed before dispatch; focused tests, exact-SHA Lume AppKit/Electron proof, cross-platform Rust/contract CI, release metadata, and attribution all pass. Contributor credit is preserved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants