feat(ai/cli): device-tools MCP server — Claude Code drives the emulator, no API key - #46
Merged
Conversation
Adds an in-process MCP server (mirrors permission_server.rs — a tiny_http Streamable-HTTP server) that exposes UMIDE's emulator device tools to the Claude Code CLI backend, so the in-panel session can drive the Android device (screenshot → reason → tap) on the machine's `claude` login — no API key. Tools: device_screenshot, describe_ui, device_tap, device_swipe, device_type, device_key, device_logs. Each reuses the built-in agent's device plumbing — the ai.rs free fns (android_screenshot/adb_input/android_describe_ui/ android_logs), reached via `super::super::` (ai/cli is a descendant of ai, so no visibility change to ai.rs) — inheriting the #41 argv-direct adb path. Verified live on a real Pixel_9a (ignored test `live_device_mcp_drives_the_ device`, run with --ignored): over HTTP exactly as Claude Code calls it — tools/list → 7 tools; device_screenshot → an 856 KB base64 PNG; describe_ui → live uiautomator parse; device_tap(540,1200) → tap executed on the device. Four pure unit tests cover the tool catalogue, keycode map, MCP-name and ToolOutput→MCP conversion. This is the standalone core; the app wiring (start the server in CliRunner, merge it into --mcp-config, thread the viewed device's serial, auto-allow the read tools in the permission bridge) follows.
dev-josias
added a commit
that referenced
this pull request
Jul 1, 2026
- README + docs/index.html: fuller emulator hardware controls (Home/Back/Recents/Power, volume, rotate, keyboard, screenshot) and the key-free CLI-agent angle (no API key; uses your existing login), plus a README bullet for the new chat sessions. - CLAUDE.md Current status: add a "Landed since #41" summary (input-channel reconnect #37, cmd.exe fixes #40/#41, DeviceInfo.serial #44, device-tools MCP core #46, AI panel redesign + sessions #48/#52, Volume/Rotate #49, per-device gRPC port #51). Device-MCP app wiring is noted as a handed-off follow-up (not advertised as shipped since it isn't wired yet).
dev-josias
added a commit
that referenced
this pull request
Jul 1, 2026
Context told the agent it's a mobile IDE; this lets it act on that. Wires Windows's device_server (#46) into the Claude Code backend so the in-panel, key-free session can screenshot/tap/read-logs on the running emulator — the see->act->verify loop, end to end. The 5 seams: - runner.rs: CliRunner gains a `serial`; run() starts a DeviceServer alongside the permission bridge (Claude only, best-effort); build_args merges its mcp_config_entry into the single --mcp-config object (--strict-mcp-config requires one JSON) and appends DEVICE_NOTE so the agent knows the tools exist. - ai.rs: spawn_cli_turn takes selected_device, resolves the Android serial, and passes it to CliRunner::new. - ai_assistant_view.rs: the Launch::Cli arm forwards active_device (mirrors LLM). - permission_server.rs: adds mcp_config_entry(); auto-allows the device *read* tools (device_screenshot/describe_ui/device_logs) while writes (tap/swipe/type/key) still prompt via the approval card. Compiles + clippy clean; 24 CLI unit tests green. Live device-driving verified by the Windows session on the Pixel per HANDOFF (claude 401 on this Mac blocks a local live run).
dev-josias
added a commit
that referenced
this pull request
Jul 1, 2026
… emulator (#53) * feat(cli): give the Claude Code backend UMIDE's mobile-first context The CLI agents felt generic — Claude Code only knew "you're inside the UMIDE editor" plus the approval model, with no idea it's a mobile IDE. It'd act like a generic code agent pointed at a folder. Add a UMIDE_CONTEXT preamble (mirroring the built-in SYSTEM_PROMPT) to Claude's --append-system-prompt in both write and read-only modes: UMIDE is a Rust IDE for cross-platform MOBILE dev (React Native / Flutter), the open project is a mobile app (prefer mobile idioms/tooling), it embeds a live Android emulator (+ iOS sim on macOS), and changes are meant to be seen and tested on the running device in a see->act->verify loop. Claude Code is the key-free default (auto-picked when no API key), so this is the path most users get. Codex (prompt-on-stdin, context via AGENTS.md) and Gemini (no append flag) have no clean system-prompt hook here — context for those is a follow-up. Build + clippy clean; CLI-args tests green. * feat(cli): wire the device-MCP so Claude Code can drive the emulator Context told the agent it's a mobile IDE; this lets it act on that. Wires Windows's device_server (#46) into the Claude Code backend so the in-panel, key-free session can screenshot/tap/read-logs on the running emulator — the see->act->verify loop, end to end. The 5 seams: - runner.rs: CliRunner gains a `serial`; run() starts a DeviceServer alongside the permission bridge (Claude only, best-effort); build_args merges its mcp_config_entry into the single --mcp-config object (--strict-mcp-config requires one JSON) and appends DEVICE_NOTE so the agent knows the tools exist. - ai.rs: spawn_cli_turn takes selected_device, resolves the Android serial, and passes it to CliRunner::new. - ai_assistant_view.rs: the Launch::Cli arm forwards active_device (mirrors LLM). - permission_server.rs: adds mcp_config_entry(); auto-allows the device *read* tools (device_screenshot/describe_ui/device_logs) while writes (tap/swipe/type/key) still prompt via the approval card. Compiles + clippy clean; 24 CLI unit tests green. Live device-driving verified by the Windows session on the Pixel per HANDOFF (claude 401 on this Mac blocks a local live run).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Lets the Claude Code CLI backend drive the embedded Android device — screenshot → reason → tap — on the machine's
claudelogin, no API key. A new in-process MCP server (ai/cli/device_server.rs) exposes UMIDE's device tools to Claude, so the same in-panel session that edits code can also operate the emulator.Requested to make the product usable key-free (the device-driving demo/QA loop otherwise required the BYO-key LLM path).
Verified live (real Pixel_9a, machine
claudelogin, no API key)Two
#[ignore]tests (run with--ignored):live_device_mcp_drives_the_device— hits the MCP server over HTTP exactly as Claude does:tools/list→ 7 tools;device_screenshot→ an 856 KB base64 PNG;describe_ui→ live uiautomator parse;device_tap(540,1200)→ tap executed.live_claude_drives_device_via_mcp— spawns the realclaudeCLI against the server and asserts it drove the device:Plus 4 pure unit tests (tool catalogue, keycode map, MCP-name, ToolOutput→MCP).
clippy/fmtclean on Windows.How
device_server.rsmirrorspermission_server.rs(atiny_httpStreamable-HTTP MCP server:initialize/tools/list/tools/call/ping).device_screenshot,describe_ui,device_tap,device_swipe,device_type,device_key,device_logs. Each reuses the built-in agent's device plumbing — theai.rsfree fns (android_screenshot/adb_input/android_describe_ui/android_logs), reached viasuper::super::(noai.rsvisibility change), so behaviour is identical and inherits the fix(ai): run adb directly (argv) so device tools work on Windows #41 argv-direct adb path.Remaining: app wiring (small, in
ai/cli/ai.rs— the Mac's area)The server is proven; to expose it from the in-panel Claude Code backend (not just the tests), four mapped seams remain:
runner.rs— startDeviceServerinCliRunner::runand merge its entry into the single--mcp-configJSON; add a device-serial field toCliRunner.ai.rs— add aselected_deviceparam tospawn_cli_turn, forward toCliRunner::new.ai_assistant_view.rs:824— passactive_device.get_untracked()(mirrors the LLM path at:821).permission_server.rs— add the read-only device tools (…device_screenshot/…describe_ui/…device_logs) tois_read_only; writes keep prompting.Coordinating on the board since that's the Mac's active area.