mac-agent-bench

A reproducible benchmark for macOS computer-use agents, run on a real, logged-in Mac — that measures a dimension the VM-based benchmarks structurally can't: disturbance, i.e. whether the agent hijacks your cursor and foreground while it works.

Public benchmarks (macOSWorld, MacArena, MacAgentBench, OSWorld) score task success inside a VM. None can ask the question that decides whether you'd run an agent on your own machine: did it let you keep working? This one does.

Method: same brain, different hands

Every trial is an identical claude -p "<task>" (one model) locked to a single MCP server via --strict-mcp-config, with all built-in tools denied except ToolSearch. The only variable is which MCP server (the "hands") the model may call. Success is judged by a deterministic shell/ AppleScript checker that inspects real state — never an LLM judge. A lsappinfo+Quartz monitor samples the frontmost app and cursor position at 5 Hz while the agent runs; 0·0 means the tool never touched the screen.

Full methodology, the instrument-building gotchas, and the current scorecard are in RESULTS.md.

Latest results (n=5, Claude Sonnet, one Apple Silicon Mac)

Tool	Success	Cost	Disturbance (focus·cursor)	Timeouts
Hunch	53/54	$6.03	14·3	0
Peekaboo	41/53	$31.03	80·46	11
cua-driver	31/54	$37.42	97·71	20

See RESULTS.md for the per-task table and caveats. (Disclosure: this harness was built by the authors of Hunch, one of the tools tested. That's exactly why it's open and reproducible — run it yourself, and submit your own tool as an adapter.)

Round 2 — 5 complex multi-step tasks (n=3):

Tool	Success	Cost	Disturbance (focus·cursor)	Timeouts
Hunch	15/15	$1.52	1·0	0
Peekaboo	13/15	$14.60	22·16	2
cua-driver	15/15	$17.70	22·0	0

Multi-step, cross-app, stateful and error-recovery tasks. Same story, sharper: perfect reliability, ~10x cheaper, ~5-10x faster, near-zero disturbance. Details and a focus-free-reading demonstration in RESULTS.md.

Run it

# one tool, a few tasks, 3 trials each
python run.py --tool hunch-main --tasks 01,02,03 --runs 3

# flags: --tool <adapter>  --tasks <comma-separated id prefixes>  --runs N
#        --timeout <s>      --model <claude model>

Results land in results/<stamp>-<tool>.jsonl (one row per trial) plus a full .stream.jsonl transcript per trial for audit.

Add your tool (an adapter is a few lines)

Drop a JSON file in adapters/<name>.json pointing at your MCP server:

{
  "mcp": { "mcpServers": { "yourtool": { "command": "npx", "args": ["-y", "your-mcp-server"] } } },
  "allowedTools": ["mcp__yourtool"]
}

Then python run.py --tool <name> --tasks 01,02,... --runs 5. The runner locks the agent to your server only. PRs adding adapters are welcome.

Add a task

Each task is a directory in tasks/NN-name/ with:

• prompt.txt — the tool-agnostic instruction (names the goal + app, not a method)
• task.json — { "description": "...", "timeout_s": 300, ... }
• setup.sh — resets state before the run (idempotent)
• check.sh — inspects real state, prints PASS (exit 0) or FAIL: <reason> (exit 1)
• teardown.sh — cleans up

The monitor and isolation are handled by the runner. Keep checks deterministic and programmatic.

Honest caveats

• Directional, not a universal leaderboard: small n, one model, one machine. A cross-OS or VM-based suite would rank differently — this suite is host-native and weights focus-free behavior.
• Timeouts bill $0 (the CLI's cost event never arrives), so slower tools' cost totals are floors.
• Some tasks are excluded when the OS environment (not the tool) is at fault; see RESULTS.md.

License

MIT (harness + tasks). Task ideas adapted from other suites retain their upstream licenses where noted.