Background computer use for Claude Code, Codex, and MCP agents on macOS.
• Open Source Codex-style Computer Use • Native Swift MCP Server • App-Aware Virtual Cursor •
中文文档 • News • Features • Demo • Downloads • Quick Start • Compatibility • Agent Guide
Important
🖥️ From GUI to Agent UI. In December 1979, when Steve Jobs saw the GUI at Xerox PARC, it became obvious that computers needed a new interface. The first time I saw Codex Computer Use, I felt a smaller version of that: agents need their own lane to use real apps and coexist better with human UI work.
🧭 Background computer use. An agent should not have to steal your mouse and keyboard to get work done. It should operate in a separate lane, stay visible through an app-aware virtual cursor, and keep enough screenshot + Accessibility context to recover.
🔓 Why open source it? The official Codex Computer Use MCP is not open source. Open Claudex is an open macOS execution layer for Claude Code, Codex, and other MCP harnesses.
- [2026.05.12] 🪽 Hermes Agent Computer Use is here too. The direction is getting clearer: agents need their own background lane, not your cursor. ✨
- [2026.05.11] 🦞 OpenClaw joined the computer-use wave with a Computer Use skill. Real-app operation is becoming a standard agent primitive.
- [Launch day] 🚀 Open Claudex Computer Use is public: an open-source macOS background computer-use layer for Claude Code, Codex, and MCP agents.
Tip
I'm a human -> Continue reading this README for demos, setup, compatibility, and project context.
I'm an agent -> Read CLAUDE.md for structured operating instructions, key files, and command quick reference.
claudex-computer-use is a native Swift MCP server that lets AI agents inspect and operate real Mac apps without moving your real mouse or requiring a cloud desktop.
- For Claude Code and Codex: local stdio MCP server plus a Codex plugin scaffold.
- For real Mac apps: Safari, Notes, Finder, Calculator, TextEdit, System Settings, and best-effort WebView-heavy apps such as Feishu/Lark.
- For demos and trust: app-aware virtual cursor overlay, post-action screenshots, and Codex-style responses.
Status: 0.1.0-alpha
Not affiliated with Anthropic, OpenAI, Apple, or the official Codex Computer Use plugin.
| Native App Control | Background Cross-App Work | Feishu / Lark Best-Effort |
|---|---|---|
 |
 |
 |
| Click and read native macOS apps through Accessibility, with a visible virtual cursor. | Let the agent work in Safari and Notes while you keep using the Mac. | Operate WebView-heavy enterprise apps with mixed AX and coordinate fallbacks. Sanitized demo data only. |
Tell your coding agent:
Install Open Claudex Computer Use from https://github.com/OpenClaudex/open-claudex-computer-use and configure it as an MCP server for my agent.
Requires macOS 13+, Swift 5.9+, Accessibility permission, and Screen Recording permission. For manual setup, see Installation & Integration.
Open Claudex focuses on the native macOS execution layer:
- Reads app state through Accessibility and screenshots.
- Performs clicks, scrolling, dragging, keyboard input, text entry, and AX actions.
- Returns post-action state so agents can continue without excessive re-snapshotting.
- Shows a same-process virtual cursor for observation and recordings.
- Supports both NDJSON and
Content-LengthMCP stdio framing.
For agent-facing usage rules, tool behavior, and recovery patterns, read Agent Guide.
| Tier | Apps | Expected Behavior |
|---|---|---|
| Stable | Safari, Notes, TextEdit, Calculator, Finder, System Settings | Strong AX tree, screenshots, semantic clicks, set_value |
| Limited | Chrome, Edge, VS Code, Slack, Cursor | Partial AX, coordinate fallback, pasteboard-heavy typing |
| Best-effort | WeChat, Feishu/Lark, self-drawn or WebView-heavy surfaces | Sparse AX, unreliable frames, more fallback logic |
Details: App Compatibility Matrix
This project started from two converging workflows: Codex-style background computer use and Claude Code-style MCP extensibility. The missing piece was a reusable open-source execution layer: a local macOS MCP server that any agent harness can plug into.
Open Claudex is not a full agent harness. It is the execution engine.
- Installation & Integration
- Agent Guide
- Demo Pack
- App Compatibility Matrix
- Testing
- Codex Native Trace Kit
- Roadmap
Open Claudex focuses on the native macOS execution layer. Related projects around computer use and agent desktops:
- iFurySt/open-codex-computer-use - open-source Codex-style computer-use MCP server.
- trycua/cua - sandbox, SDK, and infrastructure for full desktop computer-use agents.
- browser-use/macOS-use - making macOS apps accessible to AI agents.
If this project helps you, please give it a ⭐ Star!