native-cua

Independent native computer-use tools, exposed through a CLI and an MCP server. The macOS backend uses public Apple APIs; no Codex binaries or proprietary runtime are required.

Prototype: macOS is implemented with limited accessibility-based control. Windows and Linux have reserved backend directories, not working implementations. General raw mouse control is not supported.

Platform status and tested coverage

Coverage means a verified example for each published tool family, not percentage of all desktop tasks, code coverage, or reliability across applications. A tool counts once when at least one intended operation has been exercised on a real host and its result checked. There are 10 tool families in the current shared API; foreground actions and read-only tools are included in that denominator.

Platform	Backend status	Verified tool-family coverage	Verification environment
macOS	Swift prototype	100% — 10/10	Apple Silicon; macOS 27 beta, build 26A5421a
Windows	Not implemented	0% — 0/10	Not tested
Linux	Not implemented	0% — 0/10	Neither X11 nor Wayland tested

Coverage was reviewed on 2026-09-06. It combines the macOS behavioral baseline at implementation commit ed60220 with fresh status, app/window discovery, and field-setting checks against the rebuilt backend after the platform-directory split. These are manual runtime checks, not an automated test suite. They used a separate local AppKit fixture and native applications; the fixture and private screenshots/runtime records are not included in this repository. Update the numerator and verification date only after checking actual behavior on the named OS. Creating a backend directory or passing a build does not increase this coverage.

What the macOS percentage includes

Tool family	Verified example	Important boundary
status	Version and permission preflights returned without prompting	Does not grant permissions
list_apps	Discovered actual running apps and their process IDs	Metadata only
launch_app	Launched Calculator and confirmed its process	Not a background-launch guarantee
list_windows	Enumerated exact app windows and a file-picker surface	Caller selects an exact window
get_window_state	Read controls and captured an exact window/dialog	Child dialogs need their own window ID; output can be truncated
click	Activated buttons/checkboxes and opened a selected file	Declared AXPress/AXOpen actions, not arbitrary mouse injection
set_value	Changed an editable field and checked its value	Does not submit; secure fields are refused
press_key	Changed and restored Calculator mode using shortcuts	Exact target must be foreground/focused, or activation explicitly allowed
type_text	Inserted Unicode text into an observed field	Requires verified foreground field focus; not background typing
scroll	Moved a file list using its scrollbar, checking value and row position	Explicit normalized position; not wheel gestures

This is not 100% feature completeness. Right/middle clicks, raw canvas clicks, hover, dragging, wheel/trackpad input, and background keyboard injection remain unsupported. Other macOS versions, Intel Macs, fresh permission onboarding, and general app compatibility are unverified. The build targets macOS 14+, which is not a claim that every release from 14 onward has been tested.

Background checks are narrower

Three final workflows completed while the target stayed in the background in the recorded observations: an accessible coordinate click, normalized scrollbar movement, and file selection. This is a small set of successful paths, not an overall background success rate.

The controlled checkbox check sampled unchanged foreground focus and cursor position. Scroll checks verified real content movement and unchanged foreground focus; cursor movement occurred during some interactive runs, so they do not establish an unchanged-cursor guarantee. File selection was confirmed by the fixture's independent state and the dialog closing.

Advertised page scrolling failed in the tested file picker without moving content. An explicit AXOpen returned an error after selecting the file. Both cases demonstrate why dispatch/error receipts must be followed by observation, not an automatic retry. No pointer or foreground fallback was used.

Repository layout

native/
  macos/          Swift implementation: observation, input, snapshots, settling
  windows/        Reserved backend; implementation and Windows IPC/build work pending
  linux/          Reserved backend; desktop/session implementation and validation pending
src/
  tools.mjs       Shared CLI/MCP tool names, schemas, and usage contract
  cli.mjs         JSON CLI
  mcp.mjs         Stdio MCP server
  client.mjs      Validation and daemon client
  daemon.mjs      Serialized native process and local IPC
scripts/
  build.mjs       macOS build; refuses unimplemented host platforms

The current transport uses a private Unix socket and POSIX ownership/permission checks. Its placement in src/ does not imply Windows compatibility. Windows will need an appropriate local IPC implementation; Linux will need real desktop/session verification. The backend placeholders document these boundaries without providing fake implementations.

Run on macOS

Requirements: Node.js 22+, Xcode command-line tools with the required Apple SDKs, and macOS. There are no npm dependencies to install.

1. Clone the source.

   git clone https://github.com/R44VC0RP/native-cua.git
   cd native-cua

2. Build the native backend.

   node scripts/build.mjs

   Output: .build/native-cua-driver. Windows and Linux builds currently fail with an explicit unsupported-platform message.

3. Check permissions without prompting or capturing.

   node src/cli.mjs status

   Grant any missing Accessibility or Screen Recording permission to the responsible host through macOS System Settings. Keyboard event delivery also checks event-posting permission. The tool does not change TCC settings or bypass authentication.

4. Inspect available commands and running apps.

   node src/cli.mjs help
   node src/cli.mjs apps

The CLI starts the user-local daemon when needed. After rebuilding, node src/cli.mjs daemon stop makes the next call load the new executable; stopping invalidates all sessions' snapshots. Do not remove socket/lock files blindly after a crash.

MCP

Configure your MCP client to launch node with the absolute path to src/mcp.mjs as its argument. For example, the server process is:

node /absolute/path/to/native-cua/src/mcp.mjs

Use a stable _meta.sessionID on calls, or NATIVE_CUA_SESSION in the server environment, to retain snapshot ownership across connections. Without either, each MCP connection gets an isolated owner. The CLI's --session selects an owner; session IDs are correlation, not authentication. Local IPC permissions are the security boundary.

Interaction rules

1. Discover an exact app/process and window, then obtain get_window_state.
2. Act using that observation's snapshot and element IDs. Coordinate actions require a screenshot from the same snapshot.
3. Observe again to verify the application outcome. Every action consumes its snapshot; do not reuse it or blindly repeat an uncertain action.

Accessibility clicks, direct field setting, and supported scrollbar actions do not request activation by default. Keyboard input requires verified foreground focus; focus:true explicitly permits activation/focusing where supported.

Scrolling never silently changes units. Use unit:"normalized", an exact writable AXScrollBar element, and position from 0 to 1; or explicitly request unit:"pages" with a direction on a control advertising that action. Apps may reject page actions. Unsupported raw input is refused.

Permission grants do not authorize unrelated tasks or sensitive actions. Capture only necessary windows. Secure-field input is refused. A notification-quiet result means settling, not task success; cancellation cannot undo input already delivered.

Native backend boundary

The current native process reads one JSON request per line and writes one matching response per line. Future backends should preserve this boundary while documenting unsupported capabilities and platform-specific targeting semantics.

{"id":1,"method":"status","params":{},"owner":"example-session"}

Responses contain either {"id":1,"result":{...}} or {"id":1,"error":{"code":"...","message":"...","details":{...}}}. Standard output is reserved for the protocol. The current macOS implementation is in native/macos/Protocol.swift and native/macos/Driver.swift; public tool arguments are defined in src/tools.mjs.

Keep native permissions, exact target validation, owner-scoped snapshots, and honest dispatch-versus-effect reporting in each backend. Do not implement an unsupported operation by silently switching to another engine or global input.