Skip to content

fix: support xctrace-only iOS physical devices#1375

Draft
thymikee wants to merge 4 commits into
mainfrom
agent/ios-xctest-physical-device-fallback
Draft

fix: support xctrace-only iOS physical devices#1375
thymikee wants to merge 4 commits into
mainfrom
agent/ios-xctest-physical-device-fallback

Conversation

@thymikee

@thymikee thymikee commented Jul 23, 2026

Copy link
Copy Markdown
Member

Summary

Support xctrace-only iOS 15/16 physical devices through a small backend seam while keeping CoreDevice as the default path.

  • Carry the selected physical-device backend as a precise internal DeviceInfo discriminant without exposing it publicly.
  • Route XCTest runner traffic to cable-connected devices through macOS usbmuxd, using Node HTTP parsing over the upgraded socket.
  • Keep CoreDevice lifecycle behavior unchanged and return targeted guidance for unsupported fallback operations.
  • Document fallback behavior, transport requirements, and limitations.

Closes #1361
Closes #1355

Validation

  • pnpm check:affected --base origin/main --run (format, lint, typecheck, layering, fallow, and build passed; nine unrelated concurrent timeout cases passed in isolated reruns)
  • pnpm test:coverage --maxWorkers=4 (540 files; 4,568/4,569 passed, with the sole unrelated timeout passing immediately in isolation; all provider-integration tests passed)
  • pnpm build:xcuitest
  • Final chained gate: format check, typecheck, and 51 focused tests
  • Live thymikee-iphone smoke with the source-built CLI: opened Settings, resolved and tapped id="com.apple.settings.general", settled on General, found the hittable Software Update row, and captured a post-interaction screenshot; session closed cleanly
  • No iOS 15/16 hardware was available, so the fallback transport is covered by an end-to-end fake-usbmuxd regression test that verifies device lookup, connect framing, network-order port, and command response

@thymikee

Copy link
Copy Markdown
Member Author

Review: xctrace-only iOS physical-device support

Reviewed the full diff (26 files, +745/−234). This is a clean, well-scoped feature. The CoreDevice path is refactored out verbatim into physical-device-coredevice.ts (readiness error taxonomy preserved), a new xctest backend is introduced behind a small control interface, and CoreDevice-only operations (install/inventory/app-name lookup/deep links/launch args) fail with targeted guidance instead of cryptic tool errors. Test coverage for the new module is solid.

Overview

  • New IosPhysicalDeviceBackendRegistry records each physical device as coredevice or xctest from discovery evidence (devicectl vs xctrace), with CoreDevice winning on overlap.
  • resolveIosPhysicalDeviceControl(device) returns a backend-specific control (ensureReady / launchApp / terminateApp / resolveRunnerTunnelIp). XCTest readiness uses xcrun xcdevice wait; app lifecycle uses new activate/terminate runner commands; screenshots come back in-band as base64.
  • Install / reinstall / uninstall / app inventory / app-name resolution are gated to CoreDevice.

Findings

1. [design, medium] Backend selection is process-global side-channel state, decoupled from DeviceInfo.
The registry is populated only by recordDiscoveredIosPhysicalDeviceBackends, called inside listAppleDevices. Any device that reaches command execution without discovery having run in the same process silently defaults to coredevice — so an xctrace-only device would attempt CoreDevice ops and fail with confusing devicectl readiness errors.

I traced the reachable paths:

  • Common local path is fine: resolveTargetDevice → listLocalDeviceInventory → listAppleDevices populates the registry before ensureDeviceReady, and it's a module singleton so it stays warm for the session's lifetime.
  • Daemon-restart-mid-session is not reachable — SessionStore keeps sessions in an in-memory Map (only tombstones/logs/scripts hit disk), so a restart drops the session and the next command re-runs discovery. Good.
  • Injected inventory bypasses it: readInjectedDeviceInventory short-circuits before listAppleDevices, so an injected physical device is never recorded. Edge affordance today, but it's a live gap.

The deeper point is the latent invariant "discovery must precede command execution, in-process" that nothing enforces by type. Since the resolved DeviceInfo is what actually flows to every consumer, consider carrying the backend on DeviceInfo at discovery time (or resolving it lazily from device evidence in the plugin) so it can't desync from device lifetime. As-is it works, but it's fragile to any future non-discovery device source (leases, providers, a future persisted-session feature).

2. [maintainability, low] activate runner case is a no-op that returns success.

case .activate:
  return Response(ok: true, data: DataPayload(message: "app activated"))

This is correct only because .activate is classified isLifecycle: false, so prepareActiveCommandContext runs activateTarget(bundleId:) in the preflight before this switch. That's non-obvious and quietly breaks if anyone reclassifies activate as lifecycle. Two asks:

  • Add a one-line comment noting the activation is performed by the preflight and the case body is intentionally a confirming no-op.
  • .terminate guards on a non-empty appBundleId but .activate doesn't — an activate with a missing bundle would activate the runner host and still report "app activated". A matching guard would keep the two symmetric and fail loudly.

3. [dead-code, low] deviceBackend (plugin) + deviceBackendForDevice (capabilities) look test-only.
Production consumers all read the registry through resolveIosPhysicalDeviceControl; the capability accessor is exercised only by devices.test.ts / physical-device-control.test.ts. Given fallow's dead-code gating (recent chore: sweeps), either route a production read through it or confirm it's intended public capability surface so it doesn't get flagged.

4. [nit] Duplicated constants. IOS_DEVICE_READY_TIMEOUT_MS / IOS_DEVICE_READY_COMMAND_TIMEOUT_BUFFER_MS (15_000 / 3_000) are now defined in both physical-device-control.ts and physical-device-coredevice.ts. Worth sharing one source.

5. [nit] parseIosReadyPayload back-compat wrapper in device-ready.ts — confirm it still has a non-test consumer, otherwise it's dead surface after the move to parseIosDeviceDetailsPayload.

Things I checked that are fine

  • resolveIosApp's new xctest guard is correctly placed after the trimmed.includes('.') bundle-ID short-circuit, so opening by bundle ID (the required path for these devices) still works; only bare app names are rejected.
  • Inline base64 screenshots travel the device HTTP path (fetch body), so no small message ceiling to worry about.
  • CoreDevice readiness error taxonomy (probe_inconclusive / not_ready / timeout hints) is preserved through the move.
  • isMacOs is imported in plugin.ts; deviceBackend correctly excludes macOS and simulators.

Validation

Live physical-device validation is still pending (noted in the description). Given the whole feature targets rarely-available iOS 15/16 hardware, I'd treat finding #1's injected-inventory gap and a real-device smoke of open → screenshot → terminate as the two things to close before relying on this in the field.

@thymikee

Copy link
Copy Markdown
Member Author

P1: The new xctrace-only backend cannot reach its XCTest runner. resolveRunnerTunnelIp deliberately returns null, so runner-transport.ts gives a physical device only http://127.0.0.1:<port>/command. That is the Mac’s loopback, while RunnerTests.swift starts the NWListener on the phone. There is no usbmux/iproxy/other port-forwarding path, and physical devices have no simulator transport fallback. As a result, open can launch xcodebuild and log AGENT_DEVICE_RUNNER_LISTENER_READY, but its first activate probe will poll the wrong host and time out—the primary acceptance path cannot complete. Add a real host-to-device transport for xctest-backed devices and a regression/integration test that fails without it. Then provide the required live iOS 15/16 open → interaction/screenshot → close evidence. Current CI is also incomplete (only CodeQL/analysis checks are present).

@github-actions

github-actions Bot commented Jul 23, 2026

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
JS raw 1.8 MB 1.8 MB +9.0 kB
JS gzip 577.7 kB 580.6 kB +2.9 kB
npm tarball 692.3 kB 695.4 kB +3.2 kB
npm unpacked 2.4 MB 2.4 MB +10.4 kB

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 29.1 ms 28.6 ms -0.5 ms
CLI --help 63.3 ms 59.4 ms -3.9 ms

Top changed chunks:

Chunk Raw diff Gzip diff
dist/src/runner-disposal.js +14.5 kB +4.2 kB
dist/src/internal/daemon.js -7.1 kB -1.9 kB
dist/src/apps.js +481 B +164 B
dist/src/cli-help.js +440 B +153 B
dist/src/session.js +29 B +23 B

@github-actions

github-actions Bot commented Jul 23, 2026

Copy link
Copy Markdown
PR Preview Action v1.8.1

QR code for preview link

🚀 View preview at
https://callstack.github.io/agent-device/pr-preview/pr-1375/

Built to branch gh-pages at 2026-07-23 09:43 UTC.
Preview will be ready when the GitHub Pages deployment is complete.

@thymikee

Copy link
Copy Markdown
Member Author

Re-reviewed at 05167e9: the earlier transport finding is fixed. Xctrace-only devices now route XCTest traffic through usbmuxd, and the fake-usbmuxd regression exercises lookup, Connect, network-order port encoding, and command response. I found no new functional code issue in the targeted delta.

Readiness is still blocked by the confirmed Fallow failure: clearDeviceTunnelIpCache is re-exported from runner-transport.ts without a production consumer. Remove the unused export and rerun the failing check. The current-device smoke is useful, but an actual xctrace-only iOS 15/16 open → interaction/screenshot → close run remains residual validation risk because that hardware was unavailable.

@thymikee

Copy link
Copy Markdown
Member Author

Addressed the remaining review blocker in 293e90245: removed the test-only clearDeviceTunnelIpCache re-export from runner-transport.ts and imported it from its owning module in the colocated test. pnpm check:production-exports, the 7 focused transport tests, and pnpm typecheck pass locally.

Also repeated the live source-built smoke on unlocked thymikee-iphone: open Settings → selector tap on General → verify hittable Software Update → screenshot → close. The iOS 15/16 fallback remains covered by the fake-usbmuxd regression because no xctrace-only hardware is available.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant