Phone Harness 📱

phone-harness · let your agent control your phone.

Connect an AI agent — Claude Code, Codex, or any LLM — directly to your real phone with a thin, editable harness. iPhone through the Mac's iPhone Mirroring window, or Android over adb (USB or Wi‑Fi). No jailbreak, no Xcode, no WebDriverAgent, no app on the phone.

The Mac is the whole transport. iPhone: screencapture + Vision-framework OCR for eyes, HID-level CGEvents for hands. Android: screencap for eyes plus the phone's own accessibility tree — exact text, exact boxes — and input for hands. Nothing between the agent and the phone. The agent writes what's missing during execution in agent-workspace/agent_helpers.py.

  ● agent: wants to open Weather
  │
  ● ocr() → "Weather" at (400, 468)
  │
  ● tap(400, 468) → wait_stable() → ocr() confirms the forecast
  ✓ done

Your phone, driven by an agent.

Setup prompt

Paste into Claude Code or Codex:

Set up phone-harness for me. Clone https://github.com/ShawnPana/phone-harness into ~/.phone-harness (its canonical home), read `install.md` first, install it so `phone-harness` is a command on my PATH, and register it as an agent skill named phone-harness using `phone-harness skill` as the body, so you reach for it automatically. Then read `SKILL.md` for normal usage, and always read `src/phone_harness/helpers.py` because that is where the functions are. Then read `onboarding.md` and walk me through it.

The agent then follows onboarding.md: it asks one thing — which phone is your default, iPhone or Android — and walks you through only what needs your hands. iPhone: pairing iPhone Mirroring once, and granting the terminal Accessibility and Screen Recording in System Settings → Privacy & Security (Screen Recording takes effect after the terminal restarts). Android: turning on Developer options, then either plugging in and tapping Allow, or Wireless debugging + one 6‑digit pairing code. Then phone-harness --doctor verifies the chain, and phone-harness config set platform ios|android sets the default (both can be set up).

A fresh machine may prompt for more permissions the first time an action runs — if --doctor passes but taps or capture silently do nothing, watch for a macOS prompt. See install.md for details.

Why this works

iPhone. iPhone Mirroring (macOS Sequoia+) renders the phone as a Mac window and forwards real mouse and keyboard input as touches. That gives an agent everything it needs for real-device iOS automation:

• See — capture just the mirroring window, OCR it with Apple's Vision framework: every visible string with a tap-ready coordinate. The poor man's DOM.
• Act — CGEvents posted at the HID tap: taps, long-presses, drags/flicks, scroll gestures, unicode typing, and the app's own shortcuts (Cmd+1 Home, Cmd+2 App Switcher, Cmd+3 Spotlight). For icon-only controls found in a screenshot, use tap_image_point(x, y, image_size=...): screenshot coordinates are image pixels, while tap(x, y) expects global Mac screen points. image_point() performs the conversion when a raw tap() is needed.
• Verify — screenshot again. No DOM means the capture is the ground truth.

Things that do NOT work, learned the hard way: AppleScript click at (silently ignored — the window is a video stream with no accessibility tree), unicode key payloads (mirroring forwards raw HID keycodes, so typing must use keycodes), a slow touch-drag (barely moves an iOS list — use wheel scroll for lists, a fast flick for pages), and input while the window isn't frontmost (swallowed).

Android. adb reaches the phone directly, over USB or Wi‑Fi — no window, no focus, nothing on the Mac has to be in front. screencap is the capture; the phone's own accessibility tree (uiautomator) is the text source, so ocr() returns exact strings and boxes and tap_ui("url_bar") finds elements OCR never could; input tap/swipe/text are the hands; Back exists. Coordinates are device pixels, so a screenshot is 1:1 with tap(x, y). The harness finds the phone itself (a plugged-in one first, else the paired Wi‑Fi phone), refuses to drive a locked one, and can keep it awake for a session without touching a setting. Same helpers, same agent code.

Usage

./phone-harness <<'PY'
open_app("Notes")
tap_text("New Note")
type_text("hello from the harness")
print([o["text"] for o in ocr()][:10])
PY

Day-to-day workflow lives in SKILL.md, which install.md registers as an agent skill (phone-harness skill prints the body) so the agent reaches for it on its own.

Architecture

• SKILL.md — day-to-day usage (the agent-facing product surface)
• onboarding.md — the first-run flow the agent walks the user through
• install.md — setup reference and troubleshooting
• src/phone_harness/ — protected core:
  • transport.py — the one seam: the op vocabulary every device sits behind, and connect("ios"|"android")
  • helpers.py — the primitives pre-imported into scripts (platform-agnostic)
  • ios.py, mirror.py, background.py, ocr.py — the iPhone: window discovery, focus, capture, CGEvent input, Vision OCR
  • android.py — the Android: adb, the accessibility tree, USB/Wi‑Fi resolution and pairing, awake sessions, the android CLI
  • config.py — settings (phone-harness config) and remembered devices
  • admin.py — --doctor; run.py — the CLI (exec stdin with helpers in scope)
• agent-workspace/agent_helpers.py — helper code the agent edits; auto-loaded into every script's namespace

Both transports are stateless per call (window bounds and captures are re-queried; adb has its own server), so there is no daemon — every invocation is self-contained. State that must persist (the default platform, paired Android phones) lives in ~/.config/phone-harness and ~/.local/state/phone-harness.

Development

From a checkout, use ./phone-harness to run the working tree directly:

./phone-harness <<'PY'
print(screen_info())
PY

Limits

• iPhone: one phone, one session; unlocking the physical phone pauses mirroring. OCR sees text, not semantics — unlabeled icons need a screenshot + a vision-capable model.
• Android: the tree costs seconds on a slow phone and is unavailable on screens that never go idle (read the screenshot instead); input text is ASCII; key chords are not a thing over adb; a PIN-locked phone needs the user.
• Both: no multi-touch (no pinch), no camera/Face ID flows, DRM video renders black. Connecting the phone is always the user's job.