mcp-computer-use

Standalone MCP server exposing the computer_use tool verbatim — the same tool, same schema, same verdict ladder as in Hermes Agent's tools.computer_use. Backed by cua-driver.

The only difference from the original: transport. The vendor ships it as an in-process tool; we ship it as an MCP stdio tool so any MCP-speaking client (Claude Desktop, Cursor, Cline, your custom CLI, etc.) can drive a real desktop through it. The tool keeps the vendor's own name — computer_use — which is safe: host agents like Hermes prefix MCP tools with the server name (mcp__mcp_computer_use__computer_use), so it never collides with their in-process computer_use tool. Override via MCP_COMPUTER_USE_TOOL_NAME (any [a-z][a-z0-9_]* identifier) if a specific deployment wants a different name.

What you get

A single MCP tool named computer_use with 14 actions:

Action	Description
capture	Screenshot + accessibility tree (modes: som / vision / ax)
click · double_click · right_click · middle_click	Pointer input (element index or coordinate)
drag	Pointer drag (from_* → to_*)
scroll	Scroll direction + amount
type	Type text (layout-safe, with input-method hooks)
key	Key / shortcut combos (e.g. cmd+shift+t)
set_value	Select value on a control (popup / slider) without native menu
focus_app	Focus an app (optionally raise_window)
list_apps · list_windows	Inventory queries (real driver calls)
wait	Poll (capped at 30s)

Every input action returns a structured verdict:

{
  "effect":     "confirmed" | "unverifiable" | "suspected_noop",
  "verified":   true|false,
  "escalation": { "recommended": "px"|"foreground"|"stop",
                   "reason":     "..." },
  "code":       "background_unavailable" | "foreground_unsupported" | null
}

The standard escalation ladder (element+background → re-capture → coordinate+background → typed page → foreground → stop) is unchanged from the vendor.

Input is background-first — no cursor steal, no focus steal, no Space-switch. Foreground requires delivery_mode=foreground and bring_to_front=true.

Install

git clone https://github.com/phimage/mcp-computer-use
cd mcp-computer-use
python3.11 -m venv .venv
. .venv/bin/activate
pip install -e .

# sanity
python -m mcp_computer_use.server

Requires Python ≥ 3.11 and cua-driver on PATH: which cua-driver. macOS needs Accessibility + Screen Recording granted to the MCP client terminal (the vendor driver handles the rest).

Wire it into an MCP client

See examples/claude_desktop.json for a Claude-Desktop-style config. The canonical shape is:

{
  "mcpServers": {
    "computer-use": {
      "command": "python",
      "args": ["-m", "mcp_computer_use.server"],
      "env": {
        "MCP_COMPUTER_USE_APPROVAL_MODE": "deny",
        "MCP_COMPUTER_USE_SESSION_KEY": "my-client"
      }
    }
  }
}

Remote over HTTP — Hermes in Docker, server on the host

Because this tool drives the host desktop (real AX APIs, real cua-driver process), the server has to run on the host machine, not inside the container. The container (where Hermes lives) then reaches it over the network. Two transports are offered:

Transport	How to run	When
stdio (default)	python -m mcp_computer_use.server	Same machine, client spawns it as a subprocess (Claude Desktop, Cursor, Hermes on the host).
http (Streamable-HTTP)	python -m mcp_computer_use.server --transport http	Remote client — e.g. Hermes in a Docker container, or any client over the network.

1) Run the server on the host side (HTTP mode)

# On the host Mac, in a terminal:
MCP_COMPUTER_USE_APPROVAL_MODE=auto \
MCP_COMPUTER_USE_SESSION_KEY=hermes-docker \
MCP_COMPUTER_USE_HOST=0.0.0.0 \
MCP_COMPUTER_USE_PORT=8723 \
.venv/bin/python -m mcp_computer_use.server --transport http

By default it binds 0.0.0.0:8723 with a stateless HTTP session — each request is self-contained (no session id to keep), which is the right posture for a shared host service. The endpoint is http://<host>:8723/mcp; /health is a trivial liveness probe.

• MCP_COMPUTER_USE_STATELESS=1 (default) — stateless; any client can hit it.
• MCP_COMPUTER_USE_ALLOWED_HOSTS / MCP_COMPUTER_USE_ALLOWED_ORIGINS — optional DNS-rebinding / origin allow-lists (comma-separated). Empty = protection disabled, same as the MCP SDK default.
• MCP_COMPUTER_USE_APPROVAL_MODE — deny / auto / callback as before; the gate still applies in HTTP mode (the decision is made in the server process on the host, not in the container).

2) Point the Hermes-in-Docker client at it

From inside the container, register the host endpoint as a remote MCP server (Hermes supports url-based MCP servers natively):

# Replace `host.docker.internal` with whatever address the container uses to
# reach the host (Docker Desktop on Mac gives you host.docker.internal
# out of the box; on Linux you'll typically use the host's LAN/bridge IP).
hermes mcp add mcp-computer-use \
  --url "http://host.docker.internal:8723/mcp"

If Hermes prompts "does this server require authentication?", answer No for the simple setup (our server has no auth layer by default; add a reverse-proxy / API-key in front if you want a token).

Then the tool is exposed to the agent as mcp__mcp_computer_use__computer_use — the same 23-parameter schema, same 14 actions, same verdict ladder as the stdio transport. The only difference is the path across the wire: HTTP → /mcp → manager → Server → vendored handle_computer_use → cua-driver on the host.

3) What this topology gives you

• One desktop, one driver. All clients (Hermes-in Docker, Claude Desktop on the host, a browser-based MCP client) share the same physical desktop via the host cua-driver. Concurrent input will still interleave exactly like concurrent terminals; if you need true isolation, run one server per user/session (bind different ports, different SESSION_KEYs).
• Approval stays local to the server. The deny / auto / callback policy runs on the host process, not the container — so the container never needs to be able to call back, and approval is enforced even across remote hops.
• No secrets in the container. The container only holds the host's network address.

Caveats

• The server must run on a host that actually has a desktop (macOS / Windows / Linux with cua-driver working) — a container without X11/Wayland is not a valid cua-driver target.
• HTTP mode defaults to 0.0.0.0. If your host is on an untrusted network, set MCP_COMPUTER_USE_ALLOWED_HOSTS=<container-egress-host>:* to enforce the Host header, and consider TLS via a reverse-proxy (Caddy / caddy_reverse_proxy) for anything beyond a private LAN.
• cua-driver reads Accessibility / Screen Recording grants tied to the terminal on the host that started the server — grant those to that terminal.

Approvals (the piece the protocol doesn't have)

The MCP spec has no first-class "ask the user" primitive, so the vendor's approval machinery (set_approval_callback + approve_once / approve_session / always_approve / deny / timeout) is surfaced through an env-var policy:

Mode	Effect
deny (default)	All destructive actions refused until an integrator installs a callback or sets auto.
auto	All destructive actions auto-approved for the current session. Convenience for headless automation.
callback	The integrator must register a callback (see below); every destructive call goes through it.

Set it via env: MCP_COMPUTER_USE_APPROVAL_MODE=deny|auto|callback. The legacy MCP_COMPUTER_USE_AUTO_APPROVE=1 is a shortcut for auto.

Interactive approvals (callback mode) — install your own callback before constructing the server. Signature:

from mcp_computer_use.approval import interactive_callback

def my_gate(action: str, args: dict, summary: str) -> str:
    # Return: "approve_once" | "approve_session" | "always_approve"
    #         | "deny" | "timeout"
    return "approve_session"

interactive_callback(my_gate)

See examples/interactive_approval.py for a full working example.

Safe actions — capture, wait, list_apps, list_windows never hit the gate. They are read-only and always permitted.

Hard blocks — a handful of dangerous patterns are refused regardless of approval mode (e.g. rm -rf / typed via type, cmd+shift+backspace empty trash, cmd+option+shift+q force-logout). These come straight from the vendor's _BLOCKED_KEY_COMBOS table.

Tests

pip install -e ".[dev]"
pytest -v

36 tests cover:

• schema identity (the 23-param dict, the 14-action enum)
• renderer (OpenAI-style → MCP ContentBlock, including the multimodal capture shape)
• dispatch through the vendored handler (_SAFE_ACTIONS, _DESTRUCTIVE_ACTIONS, callback verdicts, blocked patterns)
• E2E over stdio using the official MCP Python client (initialize, tools/list with the default name, MCP_COMPUTER_USE_TOOL_NAME override + invalid-name rejection, tools/call, unknown-tool error)
• E2E over Streamable-HTTP using the official MCP Python client — the exact path a remote client (Hermes in Docker) uses: server launched as a subprocess on a real port, then initialize, tools/list, and a full tools/call that reaches the vendored handler across the wire

Live smoke test

.venv/bin/python examples/live_e2e.py

Spawns the real server as a subprocess, talks to it through the official MCP client, and drives the real cua-driver: list_apps, focus_app Finder, capture Finder (saves the screenshot to ~/Library/Caches/mcp-computer-use/cache/images/).

Project layout

src/mcp_computer_use/
├── __init__.py           # public API + bootstrap invocation
├── _bootstrap.py         # sys.modules alias map
├── _shims/               # one module per vendor-internal import path
│   ├── async_utils.py    # verbatim from hermes
│   ├── aux_client.py
│   ├── hermes_constants.py
│   ├── image_routing.py
│   ├── lazy_deps.py
│   ├── load_config.py
│   ├── model_tools.py
│   ├── models_dev.py
│   ├── subprocess_compat.py
│   ├── subprocess_env.py
│   ├── tool_approval.py
│   ├── tools_config.py
│   └── vision_tools.py
├── backend.py            # [verbatim] ComputerUseBackend ABC
├── cua_backend.py        # [verbatim] CuaDriverBackend + MCP-over-stdio client
├── schema.py             # [verbatim] COMPUTER_USE_SCHEMA
├── tool.py               # [verbatim] handle_computer_use + dispatch
├── vision_routing.py     # [verbatim]
├── approval.py           # [new] env-var policy adapter
├── server.py             # [new] mcp.server.Server + tool registration
└── __main__.py           # [new] `python -m mcp_computer_use.server`
tests/
├── conftest.py           # clean-env + no-backend fixtures
├── test_approval.py      # approval policy
├── test_dispatch.py      # vendor dispatch (safe / destructive / callback)
├── test_e2e_stdio.py     # full protocol via the mcp client
├── test_render.py        # renderer unit tests
└── test_schema.py        # schema identity
examples/
├── claude_desktop.json
├── interactive_approval.py
└── live_e2e.py

License & attribution

• mcp-computer-use — MIT, © 2026 phimage
• Vendored source — tools/computer_use/* from NousResearch/hermes-agent, MIT © 2025 Nous Research. Unmodified (see [verbatim] tags above).
• Driver — trycua/cua, MIT © 2025 Cua AI, Inc.