Skip to content

v0.1.0

Choose a tag to compare

@emazaheri emazaheri released this 01 Sep 18:08
· 6 commits to main since this release

Drive an iPhone or an iOS Simulator with an AI agent.

demo

What is in it

  • An MCP server, ~30 semantic tools over XCUIAutomation via WebDriverAgent, so any MCP client can drive a simulator or a tethered iPhone.
  • A terminal app (ios-agent), which streams the model's reasoning beside the digest it is reading and keeps the cost on screen.
  • The library beneath both (IosSession), with no MCP dependency, so an agent framework can import it directly.

Why it is not a test library

Screens reach the model as a compact digest rather than raw accessibility XML: 251 raw nodes to 12 elements on a third-party screen, 37,000 tokens to 329 on a long list. Resolution runs server-side through six tiers, so a retry costs zero model tokens. Every action returns the screen it produced, which halves round trips and is why the bundled agent spends exactly one observation per run, the same number a hand-written oracle needs. Anything irreversible is classified before it runs and asks first.

Measured, including the parts that were rejected

The eval harness was built before the agent so it could overturn the design, and it did. Of the four "deep agent" pillars, three were rejected on their own numbers and one was kept:

pillar outcome evidence
Verification kept actions 85 → 53 (−38%), cost $1.21 → $0.74
Planning rejected already at a hand-written oracle's floor on 8 of 10 tasks
Subagents rejected 5,864 prompt tokens per run against a 1M window
Memory rejected hedged, measured worse than none

Each rejection is an ADR in docs/adr/.

Verified

618 offline tests, 13 simulator integration tests, 11 golden flows, and a goal-directed agent measured over 13 tasks. Driven against an iPhone 17 Pro Max on iOS 26.6 over both USB and Wi-Fi, and against apps Apple did not write.

Requirements

macOS with Xcode. The simulator path needs no code signing; a physical device does. See docs/real-device-setup.md.

uv sync && ./scripts/prepare_wda.sh simulator
uv run ios-agent "turn on bold text"