Skip to content

v2.1.0

Choose a tag to compare

@igorzheludkov igorzheludkov released this 31 Jul 21:45
· 225 commits to main since this release

Latency work across tap, swipe and the screenshot tools, driven by profiling against a real simulator, plus a new diagnosis for coordinate taps that hit nothing.

tap is roughly 2x faster

Most of a tap was fixed sleeps and repeated subprocess work rather than anything the device needed:

  • Adaptive settle. The 800ms sleep before the post-tap screenshot (and its zero-diff retries) is replaced by polling until two consecutive frames match. Measured on device: a tab switch is stable at t+239ms while a heavier transition needs t+1557ms, so one constant was both too slow and too short. A fast-settling tap: 2384ms → 1343ms.
  • No more fixed 300ms fiber delay. measureInWindow results are polled (probe eval ~15ms) instead of slept on, across up to three depth attempts. Fiber strategy: 1081ms → 166-200ms.
  • One accessibility dump per tap. The strategy runs 2-3 predicates over the same screen and each re-dumped the tree (axe describe-ui ~210ms, or a uiautomator dump on Android). A two-pass text lookup: 462ms → 238ms, scaling with tree size.
  • Device pixel ratio reuses the frame already captured instead of taking its own screenshot: 448ms → 215ms on the first call of a session.
  • Device inventory is cached for 30s instead of 5s, saving a simctl list + adb devices (140-330ms) on most interactions. Safe because a failed resolve retries once against a fresh inventory.

swipe

iOS passed no duration and inherited AXe's ~1s drag, while Android had always used 300ms. Both now flick for 300ms: 1227ms → 558ms. A 1s drag was also too slow to trigger inertial scrolling, so iOS swipes now fling and cover more content per call — matching Android's existing behaviour. Also fixed a stale scale-factor lookup that could mis-convert coordinates on multi-device setups.

Screenshot tools

ios_screenshot gathered its framebuffer, accessibility probe and screen state one after another; they describe the same moment and are independent. Now run in parallel: 1142ms → 601ms, identical output. android_screenshot got the same treatment across its four legs.

New: coordinate taps explain their misses

A coordinate tap that changed nothing used to report only "no visual change detected", leaving an agent to screenshot and guess again. It now says what occupied the point and where to go instead, distinguishing three cases that were previously conflated:

  • the aim missed — names what was hit plus the nearest reachable pressable, with re-tappable coordinates and a direction
  • the target sits behind an overlay — visible in the screenshot but unreachable, so dismiss the overlay first
  • the aim was correct — the point was inside a reachable pressable, so the handler is the suspect, not the coordinates

No tool was added or removed; the MCP tool surface is unchanged.