feat(cua-driver): exact-target macOS background input v1 - #2837
Conversation
f6003b1 to
ef637a0
Compare
Bind press-key verification to the requested window and route the remaining window-addressed semantic and pointer tools through the same exact-target gate. Background double-click chooses one actuator class instead of silently crossing from AXOpen into an ungated pointer fallback. Salvaged from trycua#2837 Co-authored-by: Hamish Bultitude <18391419+hami-sh@users.noreply.github.com>
|
Maintainer refresh at
Required near-merge Lume E2E matrix (not run yet):
Each row should record the fixture case ID, before/after values and journals, frontmost pid, key-window CGWindowID, ordered window IDs/z-order, cursor position, structured result/refusal, exact driver SHA/version, macOS version/arch, and fixture build identity. One design item remains for review before merge: the checked-in plan requires a per-PID mutation coordinator, while this v1 still documents gate-to-dispatch TOCTOU as a limitation. |
Hold a narrow per-process coordinator from fresh exact-target fact gathering through dispatch, focus restoration, and target-bound verification. Nested focus clicks use task-local lease proof so callers cannot bypass gating, while independent processes remain concurrent.\n\nSalvaged from trycua#2837\n\nCo-authored-by: Hamish Bultitude <18391419+hami-sh@users.noreply.github.com>
|
Maintainer update at exact head
Required exact-head Lume evidence matrix (run the source-built driver at the SHA above and record tool journals plus before/after focus, key window, z-order, cursor, and both window values):
For every case, capture exact SHA/version, guest OS/arch, fixture build, monotonic gather/dispatch/restore/verify timestamps, structured result, and unchanged sibling-window state. This extends the previously posted AppKit/Electron two-window matrix with the coordinator-specific concurrency oracle. |
Amp-Thread-ID: https://ampcode.com/threads/T-019fcc3e-0497-718a-bfdc-69ff23c99228 Co-authored-by: Amp <289058786+ampagent@users.noreply.github.com> (cherry picked from commit 288f5fd)
…sh AX acquisition Add cua_driver_core::background_input, a pure decision module that enforces the (pid, CGWindowID) invariant for every macOS background mutation: routes are AxSemantic, WindowPointer, InsertText, and GenericKey; facts are gathered fresh (WindowServer ownership, AXWindows membership, minimized/hidden as Option<bool> failing closed on unknown, competing same-pid keyboard destinations, addressed-element ancestry); refusals are structured (code/reason/advice) and never fall through to another window of the same process. Includes the two-window wrong-target regression test. platform-macos gains ax::exact_target (fresh fact gathering, element window ancestry proof, window-bound focused-element reader) and ax::enablement (per-(pid, process-start-time) Chromium/Electron AXManualAccessibility enablement with a one-time settle), which tree.rs now reuses instead of its process-lifetime-unaware pid set. Amp-Thread-ID: https://ampcode.com/threads/T-019fcccf-9ffb-70fa-913c-76da7f968784 Co-authored-by: Amp <289058786+ampagent@users.noreply.github.com> (cherry picked from commit b6a3eec)
Every window-addressed background mutation now proves exact delivery to the requested (pid, CGWindowID) before anything is sent, and refuses with a structured reason instead of escalating or acting on a same-process sibling: - click: element branch gates as AxSemantic (middle-button fallback and the internal selection pixel fallback at the stricter WindowPointer rung); px branch gates as WindowPointer after coordinate translation, refuses out-of-frame window-local points, and requires the pid-scoped AX hit-test element to prove ancestry in the requested window. - type_text: background keyboard policy decides once per call — full ladder, semantic-AX-only (exact element, no CGEvent fallback), or refusal; the AX write and read-back verification are window-bound (focused element reacquired inside the target window only), and web-area classification fails closed when the window-bound focused element cannot be proven. - press_key/hotkey: GenericKey gate before the px focus click; PID keyboard is only permitted when the target is the process's sole eligible key destination. - type_text_chars: always-background InsertText gate; refusal is final. - press_key/type_text_chars now error on a stale element_index instead of silently downgrading to an unaddressed gate, and argument-shape errors are reported before any gating. delivery_mode:"foreground" remains the caller's explicit last resort and is not gated. Amp-Thread-ID: https://ampcode.com/threads/T-019fcccf-9ffb-70fa-913c-76da7f968784 Co-authored-by: Amp <289058786+ampagent@users.noreply.github.com> (cherry picked from commit e8fca36)
Additive structured["background_input"] section computed from the same fresh exact-target facts that gate every background mutation, reported per route so an agent can pick a viable route before acting (advisory only — every action still revalidates). The AxWindowUnresolved escalation now recommends observation/foreground instead of background px, which is refused in that state. Amp-Thread-ID: https://ampcode.com/threads/T-019fcccf-9ffb-70fa-913c-76da7f968784 Co-authored-by: Amp <289058786+ampagent@users.noreply.github.com> (cherry picked from commit f6003b1)
Bind press-key verification to the requested window and route the remaining window-addressed semantic and pointer tools through the same exact-target gate. Background double-click chooses one actuator class instead of silently crossing from AXOpen into an ungated pointer fallback. Salvaged from trycua#2837 Co-authored-by: Hamish Bultitude <18391419+hami-sh@users.noreply.github.com>
Hold a narrow per-process coordinator from fresh exact-target fact gathering through dispatch, focus restoration, and target-bound verification. Nested focus clicks use task-local lease proof so callers cannot bypass gating, while independent processes remain concurrent.\n\nSalvaged from trycua#2837\n\nCo-authored-by: Hamish Bultitude <18391419+hami-sh@users.noreply.github.com>
Require exact-window semantic proof before AXScrollToVisible for element-addressed scrolls in every direction. Retain the mutation lease and independently revalidate the pointer route before wheel delivery. Co-authored-by: Hamish Bultitude <18391419+hami-sh@users.noreply.github.com>
c0d59c9 to
d19d186
Compare
f-trycua
left a comment
There was a problem hiding this comment.
Reviewed the exact-target decision core and platform dispatch paths. The scroll reveal ordering defect was fixed before dispatch; focused tests, exact-SHA Lume AppKit/Electron proof, cross-platform Rust/contract CI, release metadata, and attribution all pass. Contributor credit is preserved.
Implements macOS background input v1 per the included plan
(
libs/cua-driver/docs/macos-background-input-v1-plan.md).What this does
Every window-addressed background mutation on macOS now enforces the exact
(pid, CGWindowID)invariant. Immediately before acting, the driver gathersfresh WindowServer and accessibility facts for the requested window. A pure
decision core either permits the selected route or refuses it with a structured
reason. A same-process sibling window cannot satisfy, receive, or confirm an
action addressed to the target window.
Routes
ownership, ancestry, and fresh
AXWindowsmembership.requiring a visible, non-minimized live frame and an in-frame point.
only when the requested window is the process's sole eligible destination.
Window-addressed refusals use explicit codes such as
window_not_found,owner_pid_mismatch,off_space_or_ax_unresolved,minimized_or_hidden_window,same_pid_keyboard_ambiguity, andelement_outside_target_window.delivery_mode:"foreground"remains thecaller's explicit last resort.
Verification semantics
oracle.
restoration, and target-bound verification; independent processes remain
concurrent.
AXScrollToVisible, and thepointer route is revalidated before wheel dispatch.
get_window_statenow reports an additivebackground_inputcapabilitysection from the same fresh facts. Per-process Electron accessibility
enablement keys on
(pid, process start time)so reused PIDs cannot inheritstale enablement state.
Compatibility
foreground behavior is unchanged.
press_keyandtype_text_charsnow reject stale element handles insteadof silently proceeding unaddressed.
Validation
Local source validation at
d19d186c734e59e299477953bbb722c8f296dbff:cargo test -p platform-macos --lib: 308 passed.cargo fmt --all -- --checkpassed.git diff --checkpassed.Authoritative exact-head CI at the same SHA:
Native macOS certification used a logged-in Lume VM running macOS 26.5.2 on
arm64. The release binary was built from the exact source tree with embedded
source SHA
d19d186c734e59e299477953bbb722c8f296dbff, signed with the establishedlocal Cua Driver test identity, and externally verified against its designated
requirement. Installed binary SHA-256:
aca9547959ba89822395180e90d7b65e98d8f08bdb0379c98807f9c542ae9cf8.The daemon reported Accessibility and Screen Recording as granted under its
own
com.trycua.driver.localidentity.Representative exact-window results while Terminal remained frontmost and the
physical cursor remained fixed at
(1033, 225):to 3.
candidate-d19d-actual.same_pid_keyboard_ambiguity.2 to 3.
not change.
Known v1 limitations
excludes only AX-proven minimized siblings.
and game/canvas input synthesis remain out of scope.