Show: phone-eye — the day it configured its own phone (vision + UI-tree fusion, any MCP client)

[boheastill]

Show: phone-eye — the day it configured its own phone

boheastill/phone-eye — five MCP tools that give any agent eyes and a finger on a real Android phone: phone_look (vision + UI-tree fusion), phone_tap/swipe/type, phone_screenshot. Listed in awesome-dsh-plugin under vision.

I built phone-eye to close a loop that kept annoying me: the agent can read code, run commands, and browse docs — but it never sees the one thing that matters at the end, the actual phone screen. So every "works on my desktop" bug turned me into a screenshot courier and a human coordinate decoder.

The story I didn't plan

Day one of dogfooding. Phone connected over Wi-Fi adb, tools wired, everything verified — except the machine I ran the test from had never been authorized on that phone. Which means, if you've done Android work, you know what was sitting on the screen:

a "Allow USB debugging?" RSA dialog.

I didn't touch the phone. The agent did this:

phone_look("any dialogs on screen? read the buttons with exact bounds")

[vision]
… dialog asking "允许 USB 调试吗?" showing an RSA key fingerprint,
"一律允许…" checkbox checked, two buttons …

[ui-tree]
允许 USB 谐试吗? @[266,1391][813,1479]
一律允许使用这台计算机进行调试 @[110,1890][930,2054]
取消 @[110,2164][523,2301]
确定 @[556,2164][970,2301]

It read the dialog with both channels — the vision model got the meaning, the UI tree got the exact bounds — then:

phone_tap(763, 2232)   → dialog still there (tap landed low; it noticed)
phone_tap(763, 2200)   → dialog gone
adb shell echo         → authorized

The tool that exists to let agents operate phones used itself to get permission to operate the phone. That closed the loop for me more than any test suite could: look → act → verify by looking again is exactly the workflow it's for, and it survived its own first failure by re-reading the screen.

Why two channels

Every screen-reading tool I found bets on one:

• UI-tree-only tools die on game canvases, images, anything the accessibility tree can't see.
• Vision-only answers drift on coordinates.

phone_eye's phone_look fuses both: the vision model answers what is this, the uiautomator dump supplies exactly where. When one channel fails, the other carries. The story above is the fusion working — the vision answer was in Chinese (screen locale), the tree still gave me tappable bounds.

The pitch in one breath

Five verbs, no framework, no on-device install, no root — adb over USB or Wi-Fi, any MCP client (dsh via dsh-mcp-client, Claude Code, Codex, Cursor…). Vision is pluggable: any MCP server exposing describe_image(path, question) — cloud, or a LAN-local Qwen-VL so no pixel leaves your network. There's a CI smoke test asserting the five verbs on every push, because a tools-list that lies is worse than no tool.

Worked examples in the repo: mobile web QA loop, surviving an OEM setup wizard, form regression checks.

Try it: github.com/boheastill/phone-eye — install is git clone + pip install + one env var. Device reports welcome in the "Verified on" table.

Happy to answer questions about the adb/vision split, the MIUI screencap quirks, or multi-device plans (Phase 2).