MobileAgentMCP

An MCP server that lets an agent drive a real Android phone — read the screen as structured UI elements, tap, swipe, type, and extract clean typed fields from known app screens.

Built and verified against a realme narzo 50 Pro 5G (RMX3395, Android 14) driving Instagram 440.1.0.46.86, unrooted.

---

Why structured UI, not screenshots

A raw uiautomator hierarchy for one Instagram Reel is ~70 KB of XML. The compact element form this server returns is typically 1–3 KB for the same screen, and it is machine-readable rather than requiring vision.

Screenshots are still available, but screenshot returns a file path, not inline image data, so it can never silently blow up an agent's context.

// ui_dump output, trimmed
{
  "package": "com.instagram.android",
  "screen": "reels_viewer",
  "app_version": "440.1.0.46.86",
  "dump_ms": 358,
  "total_elements": 39,
  "elements": [
    {"i": 12, "id": "clips_author_username", "text": "ally.verma_", "c": [402,1964], "f": "C"},
    {"i": 18, "id": "like_count", "text": "The like number is 9732. View likes.", "c": [990,1041], "f": "C"}
  ]
}

i is the index used by tap(i=…). c is the tap centre. f flags: C=clickable, S=scrollable, *=selected.

---

Install

pip install -r requirements.txt

Requires adb on PATH or at ADB_PATH. Enable USB debugging on the phone.

Register with Claude Code

.mcp.json in the repo root already does this. From the repo:

claude mcp add mobileagent -- python -m mobileagent.server

Or rely on the checked-in .mcp.json when Claude Code opens this directory.

---

Tools

Device — devices, device_info, foreground_app, launch_app, list_apps

Reading the screen — ui_dump, find_element, extract_fields, screenshot

Acting — tap, swipe, text_input, press_key

Registry / maintenance — check_drift, record_baseline, registry_info

App-specific — reset_reels_feed

---

Selector registry and drift detection

App UIs change without warning. Selectors live as data in src/mobileagent/selectors/<app>.json, pinned to the app version they were verified against.

# after an app update, or when extraction starts returning nulls
check_drift()      # -> exactly which resource-ids vanished or appeared
record_baseline()  # -> pin the new version once you have re-verified

Three rules the registry enforces, each learned the hard way:

1. Values often sit on anonymous child nodes, not on the resource-id that names them. Instagram's caption lives under clips_caption_component and the audio track under clips_author_info_component; media_album_art_button merely carries the literal string "Audio". Every element therefore reports its anchor = nearest ancestor id.

2. Intermittent anchors are marked optional so they never read as drift. Instagram's scrubber appears in only ~29 % of dumps; treating it as required makes most dumps look broken and trains you to ignore the warning that matters.

3. Fail loud. A missing anchor returns null and appears in _unavailable. It never falls back to a guess — a plausible wrong value is worse than a gap.

Known app issues

Encoded in the registry so they aren't misdiagnosed as drift:

Issue	Behaviour
reels_overlay_missing	The first reel after entering the Reels tab often renders with no engagement overlay — counts, caption and audio absent while username and video are fine. An Instagram bug. extract_fields detects it and returns data_warning with recovery steps; do not re-baseline on it. Recover with swipe(direction="up") or reset_reels_feed().

---

Backends

Phase	Backend	State
1	adb + uiautomator2 from the host	current
2	on-device AccessibilityService app	planned

Phase 2 will sit behind the same tool surface, so agent-side code does not change.

They cannot run at the same time. uiautomator2 is a UiAutomation, which is itself a special AccessibilityService, and Android permits only one. Phase 2 is a swap, not a merge.

---

Platform notes

Not every target needs device automation, and two have sanctioned APIs that are strictly better than driving a phone:

Target	Recommended route
Reddit	Official API — covers the authenticated home feed. Don't scrape.
Google	Custom Search JSON API + Programmable Search Engine.
Instagram	No sanctioned path; device automation is the route.
X / Twitter	API is paid and gated; scraping is aggressively detected.

Automated collection generally breaches these platforms' terms regardless of transport — wrapping it in MCP changes the convenience, not the permission. Use throwaway accounts and keep volume human-plausible.

Out of scope by design: defeating consent dialogs, CAPTCHA handling, device-identity spoofing, root exploits, and any anti-bot evasion. Android's security model is treated as a boundary, not an obstacle. In particular, MediaProjection consent is one tap per capture session and cannot legitimately be bypassed — not even by a device-owner or privileged app.

---

Device setup (ColorOS / realme)

ColorOS kills background services aggressively. For anything long-running:

Settings > Apps > App management > <app> > Battery usage
  -> Allow background activity
  -> Allow auto launch

plus adb shell cmd deviceidle whitelist +<package>. Without the UI toggle the ADB whitelist alone is not enough — services die within seconds.