An installable Pi extension that exposes one OpenAI-compatible computer tool backed by the same-process @trycua/cua-driver TypeScript SDK. It controls the visible primary desktop and returns a fresh PNG screenshot after every supported action.
No Cua executable, daemon, or MCP server is required. The package pins Cua Driver 0.18.0 and installs its native library for the current host as an optional npm package.
- Node.js 22.19 or newer
- Pi with extension package support
- One of the native hosts shipped by Cua Driver 0.18.0:
| OS | Architectures | Notes |
|---|---|---|
| macOS | arm64, x64 | macOS 13 or newer; Accessibility and Screen Recording grants required |
| Windows | arm64, x64 | MSVC native package |
| Linux | arm64, x64 | glibc (gnu) hosts only |
Other architectures, musl-based Linux distributions, and hosts for which npm skipped the matching optional native package fail at first tool use with a platform-specific loading error. Package loading itself remains lightweight because the native SDK is imported lazily.
From npm:
pi install npm:@edward40/pi-computer-useFrom GitHub:
pi install git:github.com/SASUKE40/pi-computer-useFrom a local checkout:
npm install
pi install ./path/to/pi-computer-useUse -l with pi install for a project-local installation. Restart Pi after installing or use /reload when the package is already available to the current Pi process.
The process hosting Pi needs both macOS grants before Cua starts:
- In Pi, run
/computer-permissions. - Accept the Accessibility request.
- If Screen Recording is missing, System Settings opens to the appropriate Privacy & Security pane. Enable the terminal or application that hosts Pi.
- Restart that host application and Pi so macOS applies the grants to the new process.
The extension checks both grants before creating a native session. It never starts Cua first and asks later. If Pi runs inside a terminal, IDE, or desktop wrapper, grant access to that actual host application—not to this npm package.
The extension registers exactly one tool named computer. Pi supplies its schema to the model, so normal prompts are enough:
Take a screenshot and tell me what is open.
Open the browser's address bar, type https://example.com, press Enter, and wait for it to load.
Scroll down in the document until you find the API section. Do not click anything.
The parameter shape is { action: { type, ...fields } }:
| Action | Fields | Behavior |
|---|---|---|
screenshot |
none | Capture current primary desktop |
click |
x, y, button |
left, right, or wheel; wheel becomes a middle click |
double_click |
x, y |
Left double-click |
scroll |
x, y, scroll_x, scroll_y |
Positive deltas move right/down |
type |
text |
Type text verbatim |
wait |
none | Wait two seconds |
move |
x, y |
Move the pointer |
keypress |
keys |
Press one key or a key chord |
drag |
path |
Drag from the first point to the last point |
click accepts the provider-compatible back and forward button names in its flat schema, but returns an explicit validation error because Cua Driver has no side-button action. META maps to Command on macOS, Windows on Windows, and Super on Linux. CTRL, ALT/OPTION, arrows, Return, Escape, and Space are normalized too.
Coordinates are physical points in the latest returned primary-desktop screenshot. Send them back unchanged: do not perform Retina, DPI, browser zoom, or logical-pixel scaling. A display arrangement with multiple monitors is not exposed as separate targets; this extension intentionally uses Cua's strict desktop scope for the primary desktop.
Scroll deltas follow OpenAI's sign convention. Each nonzero axis is converted to ceil(abs(delta) / 120) Cua line steps, clamped from 1 through 50. Horizontal scrolling runs before vertical scrolling. A drag uses its first and last path points, 500 ms duration, and the number of supplied points as interpolation steps, capped at 200.
Tool calls execute sequentially. Native operations have a 30-second timeout and receive Pi's cancellation signal. The runtime does not retry actions: if Cua reports an error, throws, times out, or is cancelled after dispatch, the result warns that the outcome is unknown and includes a new screenshot. The model should inspect that image before deciding whether a retry is safe.
Cua runs in its standard permission mode, which is promptless for normal automation. The extension adds no per-action confirmation dialog. OS-level permission prompts still apply.
The tool mirrors the OpenAI computer action surface listed above, not every Cua MCP tool. It does not expose clipboard, window management, raw session escalation, or Cua's full tool inventory. It also does not provide Codex's app-scoped Sky accessibility operations; it operates the foreground desktop through mouse, keyboard, and screenshots.
One lazy Cua session is created for each Pi extension/session instance, with a unique public session ID. The extension requests Cua's visible agent cursor. If the host cannot provide a cursor overlay—most notably a terminal-hosted same-process runtime on macOS—it reports a warning and keeps desktop capture/input available. Guaranteeing the macOS overlay would require Cua's separately supplied GUI private-worker executable, which is intentionally outside this package's no-executable contract. On Pi session_shutdown, cleanup attempts endSession, shutdown, and uniffiDestroy in order, even if an earlier cleanup step fails.
The native SDK cannot be loaded
Confirm node -p 'process.platform + "/" + process.arch' matches a row above. On Linux, confirm the system uses glibc. Reinstall without --no-optional; Cua's platform library is an optional dependency selected by npm.
macOS says permissions are missing after I enabled them
Fully quit and restart the terminal, IDE, or app hosting Pi. Run /computer-permissions again. If Pi is launched through another host process, macOS may attribute the grant to that host.
Clicks land in the wrong place
Request a new screenshot and use its coordinates directly. Do not scale coordinates or reuse locations after a window moved, display settings changed, or the UI reflowed.
An action reports an unknown outcome
Inspect the screenshot returned in the same tool result. The action may have completed before the timeout or cancellation; retry only when the screenshot proves it did not land.
Desktop capture fails
On macOS, verify Screen Recording. On all platforms, ensure the visible desktop is unlocked and available to the Node process. The extension requires a PNG observation and rejects missing or non-PNG captures.
The agent cursor overlay is unavailable
The tool still works and includes the Cua message as a warning. Same-process macOS runtimes hosted by a terminal do not have Cua's certified AppKit main-thread adapter, so they cannot display the overlay. A GUI host with the appropriate adapter is required for that visual affordance.
npm install
npm run typecheck
npm test
npm run pack:checkThe native smoke test is deliberately opt-in because it opens a real desktop session. It only captures one screenshot and never sends mouse or keyboard input; teardown always runs:
npm run test:smokeTo additionally load the extension through Pi's real resource loader, verify its tool and command registrations, and invoke computer for a screenshot:
npm run test:pi-smokeMIT