Skip to content

Poll AX readiness when snapshot_ui returns an empty hierarchy #408

@dpearson2699

Description

@dpearson2699

Summary

Could XcodeBuildMCP add AX readiness polling/retry behavior when snapshot_ui / describe-ui returns the known empty hierarchy shape on iOS simulators?

This is a follow-up to #290 and the investigation in #312. I'm not asking to revive the proactive defaults write approach, since the maintainer investigation showed that early writes to com.apple.Accessibility can make the simulator AX state worse. This request is specifically for a readiness/recovery layer around the UI automation tools.

Background

In #290, the reported failure shape was:

  • a bare AXApplication node
  • zero-size frame, e.g. {{0, 0}, {0, 0}}
  • children: []
  • tool result still looks like a successful capture, even though no usable UI hierarchy is available

The maintainer follow-up in #312 found that this is most likely an AX daemon / CoreSimulator readiness timing issue rather than the accessibility defaults being the direct gate. The comment also mentioned that XcodeBuildMCP may want to add a polling readiness probe later.

I hit a related practical variant in a downstream project workflow: the app had been launched while simulator AX introspection was unavailable, and snapshot_ui kept returning the empty hierarchy until the app process was relaunched. After relaunch, the same XcodeBuildMCP snapshot_ui call returned a populated hierarchy with the expected app frame and elements.

That makes the current behavior difficult for agent workflows: the tool reports SUCCEEDED, but the hierarchy is effectively unusable, so the agent has to infer whether the app is blank, accessibility metadata is missing, the simulator AX bus is not ready, or the app needs relaunch/reboot.

Proposed enhancement

When a UI hierarchy capture returns the known empty AX shape, XcodeBuildMCP could treat it as "AX not ready" rather than an ordinary successful empty app tree.

Possible behavior:

  1. Detect the suspicious shape:
    • root application node only, or no meaningful descendants
    • root frame is zero-size or otherwise invalid
    • children: []
  2. Poll/retry for a bounded period, e.g. 15-30 seconds, before returning the final result.
  3. If still empty, return an actionable diagnostic in the tool result, such as:
    • AX hierarchy is unavailable / simulator AX service may not be ready
    • try relaunching the app
    • if persistent, try simctl shutdown <UDID> && simctl boot <UDID>
  4. Optionally, for tools that already own launch/build-run flow, consider relaunching the target app once after the readiness timeout before failing/returning the diagnostic.

Why this would help

This keeps the fix aligned with the findings in #312:

  • no proactive accessibility flag writes during the risky startup window
  • no assumption that app-side accessibility metadata is broken
  • clearer distinction between "the app has no accessible UI" and "the simulator AX pipeline is not ready"
  • more robust behavior for fresh simulator boots and agent-driven UI verification

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions