feat(ai): harden + extend agent device control — consent gate (E1) + adb timeout/retry (F2) + describe_ui (B4) - #36
Merged
Conversation
The LLM agent's device tools (tap/swipe/type_text/press_key) now require an explicit per-session approval before the first one runs. EditorTools carries a shared Option<bool> consent latch: None until asked, then Some(granted) so the user is prompted exactly once. A new ApprovalKind::DeviceControl surfaces the card; approving unlocks device input for the rest of the session, rejecting returns a clear error to the agent for every device action. Reads and file/command tools are unaffected (they keep their own gates). The outcome->granted mapping is factored into consent_granted() and unit-tested.
adb_sh() ran adb/simctl/idb via a bare Command::output() with no timeout, so a hung adb — device mid-boot, offline, or a wedged daemon — would block the agent turn indefinitely. It now runs through output_with_timeout (20s cap) and retries exactly once on a transient failure: a timeout, or an adb daemon-restart/attach race (device offline / error: closed / daemon not running / protocol fault / device still connecting). Genuine failures (bad command, missing binary) return immediately without a wasted retry. The transient classifier is factored into adb_stderr_is_transient() and unit-tested.
dev-josias
changed the title
New read-only device tool: dumps the Android view hierarchy via `uiautomator dump` and renders it as a compact, tappable element listing — one line per labelled or interactive node with its center coordinate, label, short class and id. This is the accessibility fallback for when a screenshot is ambiguous (custom-rendered React Native / Flutter surfaces where pixels don't reveal structure): the agent gets element text plus a coordinate it can tap. Purely additive: a new read tool (no consent gate, like screenshot/read_logs), a new spec, and a dispatch arm. iOS returns a clear Android-only message. Parsing (parse_ui_dump / bounds_center / xml_unescape) is factored out and unit-tested; output is capped at 200 elements so a deep tree can't flood context. System prompt updated to advertise the tool.
dev-josias
changed the title
dev-josias
added a commit
that referenced
this pull request
Jul 1, 2026
docs(handoff): report 3 Windows cmd.exe device-command bugs in #36 (proven fixes)
This was referenced Jul 1, 2026
dev-josias
added a commit
that referenced
this pull request
Jul 1, 2026
#43) Refresh the Current status: the closed loop (see->act->see) is now landed — A2/B3 (#30), E1+F2+B4 (#36), G2 producer+consumer (#35/#39), and the Windows adb direct-argv fix (#41). Remaining follow-ups: macOS active_device producer, a panel-provided DeviceInfo.serial, and the live Pixel demo capture (Windows).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three small, cohesive changes to the agent's emulator-control tools (Mac-owned
ai.rs/ai_assistant_view.rs; no overlap with Windows PR #35). Each is an atomic commit.E1 — gate emulator control behind per-session consent
Device tools (
tap/swipe/type_text/press_key) now require a one-time per-session approval before the first one runs — the last un-gated mutating surface (edits + commands were already per-action approved).ApprovalKind::DeviceControl;EditorToolscarries anArc<Mutex<Option<bool>>>latch (Noneuntil asked →Some(granted)), so the user is prompted exactly once, not per tap.F2 — time-box device shell commands + retry transient adb
adb_sh()had no timeout, so a hungadbcould freeze the agent turn.output_with_timeout(20s cap).device offline/error: closed/daemon not running/protocol fault/device still connecting). Genuine failures return immediately.B4 —
describe_uiaccessibility tool (Android)New read-only tool:
uiautomator dump→ compact listing of labelled/tappable nodes with center coordinates, class and id. Accessibility fallback when a screenshot is ambiguous (custom-rendered RN/Flutter UIs). iOS returns an Android-only message. Output capped at 200 elements.Verification
cargo build -p umide-appgreen; fmt + clippy clean on touched codecargo test -p umide-app --lib ai::→ 33 passed (new:device_consent_maps_approve_and_reject,adb_transient_errors_are_retried,bounds_center_computes_midpoint,parse_ui_dump_lists_labelled_and_tappable_nodes,parse_ui_dump_reports_empty_hierarchy)cargo test -p umide_agent→ greendescribe_uireturns sane bounds/text (per HANDOFF).