Skip to content

feat(desktop): add opt-in cua-driver computer use - #4354

Open
injaneity wants to merge 241 commits into
pingdotgg:t3code/codex-turn-mappingfrom
injaneity:feat/cua-computer-use
Open

feat(desktop): add opt-in cua-driver computer use#4354
injaneity wants to merge 241 commits into
pingdotgg:t3code/codex-turn-mappingfrom
injaneity:feat/cua-computer-use

Conversation

@injaneity

@injaneity injaneity commented Jul 23, 2026

Copy link
Copy Markdown

what changed

adds opt-in computer use to the macos, linux, and windows desktop apps. when enabled, desktop passes the packaged cua-driver path and embedded module configuration to its local node server. the existing scoped embedded host owns the driver process and exposes its tools to codex.

packaged builds now download checksum-pinned driver assets for each supported platform and architecture. macos and linux use their release tarballs. windows uses its release zip and powershell expand-archive, which avoids incompatible gnu tar installations earlier on path.

macos keeps its existing permission-inheritance behavior. linux and windows need no additional permission-inheritance implementation and reuse the same embedded lifecycle without installing services, autostart entries, or persistent daemons.

why

the embedded host gives all three platforms one process-ownership model: t3 code starts the driver only after the user opts in, and the scoped finalizer stops it with the server. platform-specific code is limited to release assets, extraction, and executable paths.

validation

  • local: 43 focused tests, affected package typechecks, lint, and formatting pass.
  • cua-linux: 43 focused tests pass; checksum, extraction, binary execution, embedded startup, and clean shutdown pass.
  • cua-windows: 43 focused tests and affected package typechecks pass; checksum, powershell extraction, binary execution, embedded startup, and clean shutdown pass.
  • the windows remote runs in session 0, so gui interaction remains subject to the driver’s documented interactive-session requirement.

ui changes

the existing computer-use setting now appears on every supported desktop platform.

computer-use-enabled computer-use-disabled

checklist

  • this pr is small and focused
  • i explained what changed and why
  • i included before/after screenshots for ui changes
  • i tested the platform-specific paths on linux and windows hosts

Note

Add opt-in embedded cua-driver computer use to the desktop app

  • Adds an enableCua toggle to client settings (default false) exposed in the desktop General Settings panel; the feature is only shown when the desktop bridge is available.
  • On startup, DesktopApp reads enableCua and calls configureCuaDriverServerEnvironment to set cua-related process env vars (T3CODE_CUA_DRIVER_PATH, T3CODE_CUA_DRIVER_HOST_BUNDLE_ID, T3CODE_CUA_DRIVER_MODULE_URL); on failure, it shows a blocking error dialog and continues with cua disabled.
  • Server startup gains a cua-driver.start phase via startEmbeddedCuaDriver, which imports the driver module by URL, starts the host, installs Codex MCP launch args via T3CODE_CODEX_APPEND_LAUNCH_ARGS_ENV, and monitors the driver process; it cleans up on scope shutdown.
  • The desktop build script now downloads, SHA-256 verifies, extracts, and stages the @trycua/cua-driver (v0.12.2) binary for the target platform/arch into prod-resources/cua-driver, then includes it as an Electron extra resource.
  • Risk: the cua-driver binary is fetched at build time from a pinned release URL; a checksum mismatch surfaces as CuaDriverChecksumMismatchError and fails the build.

Macroscope summarized 4d2f83e.


Note

Medium Risk
Opt-in feature with scoped driver lifecycle, but it runs a native host, mutates process env for Codex MCP, and enables agents to control the user's machine when turned on.

Overview
Adds opt-in computer use for desktop (macOS, Linux, Windows) via @trycua/cua-driver 0.12.2, default off.

Settings & UI: New enableCua on client settings (contracts + General settings when desktopBridge is present). Users must restart after toggling.

Desktop startup: If enabled, configureCuaDriverServerEnvironment sets T3CODE_CUA_DRIVER_* for the local Node server (packaged driver path, host bundle id, embedded module URL). Inherited env vars are cleared when disabled so dev overrides cannot bypass opt-in. Configuration failures show an error dialog and the app continues without computer use.

Server: startEmbeddedCuaDriver starts the embedded host when env is set, wires Codex MCP via T3CODE_CODEX_APPEND_LAUNCH_ARGS, and tears down on driver exit. Codex adapter spawn args prefer live append launch args over stale per-session env snapshots.

Packaging: Desktop artifact build downloads checksum-pinned release binaries per platform/arch, stages them into prod-resources/cua-driver, and includes them in extraResources.

Reviewed by Cursor Bugbot for commit 4d2f83e. Bugbot is set up for automated code reviews on this repo. Configure here.

juliusmarminge and others added 30 commits April 17, 2026 17:29
Co-authored-by: codex <codex@users.noreply.github.com>
- Initialize provider as unchecked in a pending state
- Update initial probe message to reflect session-local status
- Type the runtime effect with `Scope`
- Build the ACP session runtime without wrapping it in `Effect.scoped`
- Use strict TurnId and ProviderItemId parsing in Codex session routing
- Decode in-memory stdio chunks in streaming mode to avoid split UTF-8 corruption
- Transfer session-owned scopes into adapter state
- Ensure runtime scopes close on stop and startup failure
- Add regression coverage for scoped lifecycle cleanup
- Close the managed native event logger when the adapter layer tears down
- Make session runtime close idempotent with an atomic closed flag
- Add coverage for flushing thread native logs on shutdown
- Use codex app-server snapshots for auth, models, and skills
- Remove legacy CLI/config discovery paths and related helpers
- Update tests for the new provider status flow
Co-authored-by: codex <codex@users.noreply.github.com>
Co-authored-by: codex <codex@users.noreply.github.com>
- Document the target orchestration graph, IDs, lifecycles, and capability model
- Add Codex app-server probe fixtures and update the probe test harness
- Introduce orchestration v2 service interfaces and error types
- Add replay runtime, fixtures, and integration coverage
- Update shared contracts and probe transcripts

Co-authored-by: codex <codex@users.noreply.github.com>
- Add Codex adapter and replay harness wiring
- Introduce in-memory orchestration projections and provider registry
- Expand orchestration contracts for turn and runtime events
Co-authored-by: codex <codex@users.noreply.github.com>
- Add context transfer IDs, schemas, and projections
- Support cheap fork creation and Codex native fork rollback
- Cover fork idempotency and replay behavior in tests
- Track remaining projection, context transfer, rollback, capability, and subagent work
- Clarify current V2 baseline and debugger-only follow-ups
- Map fork and merge-back turns into stored handoffs and transfer resolutions
- Add shell snapshot projection support plus coverage tests
- Update replay fixtures and web contracts for the new turn flow
Co-authored-by: codex <codex@users.noreply.github.com>
- Move Codex replay recording into `apps/server`
- Add Claude Agent SDK replay fixtures and test harness
- Update orchestration-v2 fixture scenarios and docs
- Move Claude provider runtime logic into its own module
- Share the SDK query runner between live and replay paths
- Add replay driver error wrapping for unexpected failures
Port orchestration V2 provider adapter wiring to the provider-instance driver registry.

Co-authored-by: codex <codex@users.noreply.github.com>
- persist the selected model on run records
- surface run model selection in the debug UI
- update replay fixtures and contracts for the new field
- Record Claude SDK transcripts across multiple prompts and restart/query modes
- Add approval and tool-call replay coverage for new orchestration fixtures
- Update Claude adapter testkit to model open/prompt/permission frames
- Derive Claude SDK query options from runtime policy
- Add read-only replay fixture and policy mapping tests
- Reuse shared approval-policy fixtures across orchestrator tests

Co-authored-by: codex <codex@users.noreply.github.com>
- add active steering and interrupt-restart replay fixtures
- update Claude adapter/orchestrator turn handling for steering
- refresh replay and integration test coverage
- add interrupt and mid-tool replay fixtures for Claude and Codex
- log Claude Agent SDK protocol frames to native event traces
- project Codex commandExecution start events into orchestration updates
- Map Cursor SDK agents and runs to V2 thread and turn lifecycles
- Update MCP capability, tool, and testing guidance for SDK-based injection
@injaneity
injaneity force-pushed the feat/cua-computer-use branch from 7b4ff33 to 731c807 Compare July 31, 2026 02:56
Comment thread scripts/build-desktop-artifact.ts
Comment thread apps/desktop/src/cua/CuaDriverServerEnvironment.ts

@macroscopeapp macroscopeapp Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Effect service conventions review: one error-modeling issue found in the new Cua modules. Everything else (namespace imports from effect/*, Schema.TaggedErrorClass usage, scoped finalizers, Path.Path acquired from the environment, whole-channel Effect.catch) matches the conventions.

Posted via Macroscope — Effect Service Conventions

Comment thread apps/desktop/src/cua/CuaDriverServerEnvironment.ts Outdated
@injaneity injaneity changed the title feat(desktop): add opt-in cua-driver for macOS computer use feat(desktop): add opt-in cua-driver computer use Aug 2, 2026

@macroscopeapp macroscopeapp Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One convention finding on the new Cua driver error definitions. See the inline comment.

Posted via Macroscope — Effect Service Conventions

Comment thread apps/desktop/src/cua/CuaDriverServerEnvironment.ts Outdated

@macroscopeapp macroscopeapp Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Effect service conventions review: two findings on error context and tagged-failure recovery in the new Cua driver modules.

Posted via Macroscope — Effect Service Conventions

Comment thread apps/desktop/src/cua/CuaDriverServerEnvironment.ts Outdated
Comment thread apps/desktop/src/app/DesktopApp.ts Outdated

@macroscopeapp macroscopeapp Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Effect service conventions review: one error-context finding in apps/server/src/cua/CuaDriverEmbedded.ts.

Posted via Macroscope — Effect Service Conventions

Comment thread apps/server/src/cua/CuaDriverEmbedded.ts Outdated

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want fixes drafted automatically? Bugbot Autofix can create code changes for findings. A team admin can enable Autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit d296a0d. Configure here.

Comment thread scripts/build-desktop-artifact.ts
@maria-rcks
maria-rcks force-pushed the t3code/codex-turn-mapping branch from a543fd4 to 378615b Compare August 3, 2026 16:01
@juliusmarminge
juliusmarminge force-pushed the t3code/codex-turn-mapping branch 3 times, most recently from e24b59c to 4213ac5 Compare August 5, 2026 14:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XL 500-999 changed lines (additions + deletions). vouch:unvouched PR author is not yet trusted in the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants