v0.12.0

Added

• Android support for stream-video, and a top-level cross-platform stream-video verb (#78). sim-use android stream-video --format h264 streams native adb screenrecord H.264 Annex B passthrough to stdout (variable frame rate, survives screenrecord's per-invocation time limit by restarting mid-stream — each new segment re-emits SPS/PPS, which ffplay/ffmpeg accept); mjpeg / raw / ffmpeg provide a screencap-based JPEG loop (~7–8 FPS ceiling) with byte-identical framing to the iOS formats. The new top-level sim-use stream-video routes by UDID shape: bgra stays iOS-only, h264 is Android-only for now, and the platform-mismatch cases fail with a pointer to the right alternative. Streaming now also treats consumer hangup (ffplay quit, head -c done) as an orderly end-of-stream on Android instead of dying on SIGPIPE. SIM_USE_SCREENRECORD_TIME_LIMIT=<seconds> (debug) forces short screenrecord segments so the restart path can be exercised on API ≥ 34 devices; it applies to Android record-video and stream-video alike. --json is rejected on every stream-video surface — stdout carries the raw video bytes and the envelope would corrupt the stream (the run summary goes to stderr). The Android segment loops only restart screenrecord after a clean exit 0 (time limit); a screenrecord that dies mid-run now aborts with its stderr instead of being blindly restarted — record-video still finalizes and keeps the partial MP4. (#80)
• sim-use android record-video: the Android recording engine moved out of the top-level forwarder into AndroidBackend, giving record-video the same three-surface layout as every other cross-platform verb (top-level + ios + android). Flags and behavior are unchanged; sim-use record-video with an Android serial routes to the same implementation. (#79)
• swipe / touch --coordinate-space native|ui (top-level and ios subcommands): explicit coordinates keep their device-native portrait default, and ui opts them into the visual space describe-ui prints — endpoints are orientation-calibrated per command (per batch for batch steps), so outline coordinates stay correct on a rotated device (#66). touch accepts ui in the atomic --down --up form only: a split touch could straddle a rotation and land its two halves in different spaces. A failed calibration degrades to native dispatch and says so via the advisory envelope key. On Android the flag is accepted and ignored — on the top-level commands and the android swipe / android touch direct commands alike (display coordinates already rotate with the UI). (#74)
• describe-ui --no-raw (top-level, ios describe-ui, and android describe-ui): with --json, omit the raw accessibility tree from the envelope. data.raw typically dominates the payload on real app screens and is only useful for debugging sim-use itself; outline / entries / lists are unaffected. (#63, #69 — thanks @4faramita!)
• SIM_USE_HID_TRANSPORT=indigo|dtuhid debug override to force a specific iOS HID transport (default: automatic per-boot selection). The per-UDID daemon keeps the environment it was spawned with — combine with SIM_USE_NO_DAEMON=1 or restart the daemon for ad-hoc experiments. (#68)
• SIM_USE_DEBUG=1 turns on the logger's stderr sink (with debug-level detail) on every construction path, including ios batch without --verbose, making internal info-lines — notably the HID transport-selection signals logged at connection creation — visible in normal runs and in the daemon logfile. Same daemon-environment caveat as SIM_USE_HID_TRANSPORT. (#67, #72)
• Keeping output small section in skills/sim-use/SKILL.md: steers agents to prefer the text outline, pair --json with --no-raw, verify via outline instead of screenshots, reuse the verify read as the next observe, and batch known sequences. (#63 — thanks @4faramita!)
• Viewer: GET /api/snapshot now forwards the CLI's calibrated interface orientation as screen.orientation (omitted when describe-ui reports none — Android, legacy daemons), and the SPA tags a rotated screen next to the W×H readout, e.g. 874×402 (landscape-right). (#57, #71)
• make e2e-matrix (scripts/e2e-matrix.sh): iOS E2E across the supported host environments — Xcode 26.x / Xcode 27.x × Device Hub closed at boot (the classic Simulator.app workflow, legacy indigo HID) / Device Hub open at boot (CoreDevice dtuhid). One leg runs the full suite (default x27-hub, the primary workflow once Xcode 27 ships; --full <leg>|all|none to change), the rest run the new smoke tier (test-runner.sh --smoke: describe-ui, tap, type, scroll); legs whose Xcode is not installed are skipped. The package builds once on the xcode-select toolchain (build_products/ is toolchain-locked); each leg swaps only the runtime Xcode — SIM_USE_TEST_DEVELOPER_DIR, which the test harness injects as DEVELOPER_DIR into every process the suites spawn — and boots a device whose iOS runtime matches the leg's Xcode generation (runtimes are system-wide, so "newest available" would cross-contaminate). Every leg is double-gated against silently green-running the wrong combination: dtuhidd process-tree checks before and after the suites, plus a SIM_USE_DEBUG probe that must report the matching HID transport-selection predicate. Per-leg logs and a combination-evidence file land in .build/e2e-matrix/<timestamp>/. test-runner.sh now also accepts arbitrary repeatable swift test --filter patterns (previously one suite from a fixed whitelist) and resolves its default simulator by exact device name on the newest available runtime (previously a substring match that could pick an unavailable device or an "… Pro Max" sibling). (#77)
• The agent-eval suite can now pin exactly which sim-use binary a run evaluates: make eval ARGS="-b <path>" / scripts/eval.sh --sim-use <path> / run.py --sim-use <path> (default remains whatever sim-use resolves to on PATH). The wrapper and runner print sim-use under test: <real path> (<version>) up front and the report header records it, so a run can never silently exercise the wrong binary — and development builds under .build/ can be evaluated directly. New repo skill .claude/skills/run-evals/ orchestrates the whole flow for agents and contributors: environment prep (Device Hub closed, fixtures installed), binary selection, cost confirmation, and verdict triage; e2e/agent-evals/README.md documents the new prereqs and flags. (#65)
• E2E coverage for the capture surfaces on both platforms: new ScreenshotTests (iOS) and AndroidScreenshotTests / AndroidRecordVideoTests suites — PNG-magic and AVAsset-loadable-MP4 assertions, top-level platform routing, --output - stdout mode, and a forced-segment-restart recording case — registered in the respective E2E runners. New unit suites pin VideoOutputFile path resolution, the VideoFrameUtilities frame processing (including the pass-through fast path), the shared flag validators, and the process-control primitives. (#81)

Changed

• record-video now captures a real H.264 stream and muxes it straight into the MP4 (passthrough) instead of polling screenshots and re-encoding each frame. iOS drives FBSimulatorVideoStream in eager H.264 mode at a constant --fps (frames laid out at exactly 1/fps, so the rate is honored and playback is smooth); Android pipes adb screenrecord --output-format=h264 at its native variable frame rate, stitching across screenrecord restarts past the API < 34 per-invocation limit. --fps is now 1–60 (default 30, was 1–30 default 10) and is ignored on Android; --quality/--scale map to bitrate/size on both platforms. A mid-recording rotation stops Android capture (an MP4 track can't change frame size). (#52, #70 — thanks @subdiox!)
• The pinned idb ref is bumped from 76639e4d (2025-05-29) to 1f6943f8 (2026-07-23) (#68) — a migration, not a version bump (upstream Swiftified the HID layer and much of FBControlCore, moved the build to XcodeGen-generated projects producing static frameworks, and rewrote the video layer in async Swift). What it buys:
  • Device Hub workflows now just work on Xcode 27. The HID transport is selected automatically per simulator boot: simulators whose legacy HID was suppressed at boot (booted while Device Hub was open) are driven through dtuhidd's CoreDevice HID service (upstream's new FBSimulatorDTUHIDTransport), everything else through the legacy SimulatorKit path. tap/type/swipe deliver in both states — no reboot dance.
  • Multi-touch verbs ride upstream's native two-finger API (FBSimulatorHIDEvent.twoFingerTouch); the whole gesture is assembled into one composite event so the transport is drained exactly once per gesture, as the DTUHID transport requires.
  • The FB frameworks are statically linked into the sim-use binary.* The release payload shrinks to the executable plus the two resource bundles (no Frameworks/ directory, no framework codesigning), and the entire dev-loop rpath machinery — Package.swift slice rpaths, scripts/stage-fb-frameworks.sh, release-time rpath stripping — is retired. Package.swift instead wires the idb PrivateHeaders Clang module maps, force-loads the three ObjC-bearing archives (their category-only members export no referenced symbol and would otherwise be dropped — deliberately per-archive rather than -ObjC, which makes Xcode 26.5-built binaries trip a Swift task-allocator abort via CompanionUtilities), and weak-links the CoreSimulator / AccessibilityPlatformTranslation .tbd stubs at the final link.
  • All four patches/idb/ patches are retired: xcode27-simulatorkit-sharedframeworks and multi-touch-spike are superseded upstream; headerpad-shims targeted the removed checked-in project's shim build phase; fbprocess-runtime-rename renamed a class that no longer exists (and was never referenced by sim-use).
• gesture --screen-width/--screen-height on iOS: for single-finger presets they now describe a visual-space canvas and default to the auto-detected screen size of the current orientation (previously always 390×844, iPhone 15 portrait — wrong for iPads and every landscape state); 390×844 survives only when the screen size cannot be probed. Pinch/rotate presets keep the raw device-native portrait canvas with the fixed 390×844 default. (#73, #74)
• Building the XCFrameworks now requires XcodeGen (brew install xcodegen) — the idb checkout generates FBSimulatorControl.xcodeproj from project.yml. The frameworks are built without library evolution, so build_products/ is locked to the toolchain that produced it: re-run ./scripts/build.sh dev after switching Xcode versions (CI keys its cache on the Xcode version for the same reason). (#68)
• Internal: the platform-neutral video plumbing (H.264 Annex B parsing, passthrough muxing, AVAssetWriter encoding, frame/output-path utilities) moved out of iOSSimBackend into a new SimUseVideo target, and the generic process-control helpers (CancellationFlag, SignalObserver, …) into SimUseCore. Pure restructuring — no CLI behavior change; unblocks hosting Android video verbs inside AndroidBackend (#78 groundwork, #79).

Fixed

• The iOS HID transport auto-selection no longer races dtuhidd's boot-time attach (#67). dtuhidd appears 0–3 s after boot on a simulator booted while Device Hub is open; a HID verb landing inside that window probed the process tree too early, auto-selected the legacy Indigo transport, and the cached connection pinned the wrong transport for the entire boot — type exited 0 while delivering nothing, with no error to trigger recovery. A connection whose transport was auto-selected while launchd_sim uptime is under 15 s is now served once but never reused, so the next command re-derives the selection after the attach window closes. Steady-state behavior (uptime ≥ 15 s at first HID verb) is unchanged. (#72)
• iOS directional gesture presets (gesture scroll-up/down/left/right, swipe-from-*-edge) are now orientation-aware (#66). They previously emitted device-native portrait axes, so on a rotated device scroll-up pointed 90° (landscape) or 180° (upside-down) away from its name — no scroll, or an unintended row navigation. Preset math now runs in the current visual space (auto-calibrated per command, the same #34 mapping tap selectors use) and the endpoints are transformed into HID coordinates; this covers both the standalone command and ios batch gesture steps (which share the batch-wide calibration). When the orientation cannot be determined the gesture falls back to the legacy portrait dispatch and says so via the advisory envelope key. Explicit swipe/touch coordinates remain device-native portrait by contract; pinch/rotate presets are likewise unchanged. (#73)
• iOS describe-ui no longer goes blind when a system document picker (or another remote-process presentation) is on screen while the host app stays frontmost (#64). The frontmost tree comes back as an empty shell in that state — a bare or full-screen-framed AXApplication with no content — and previously produced zero entries even though --point could hit the visible picker. The fetch now detects the shell and retries once with upstream's cross-process content discovery (coverage-grid hit-testing), surfacing the picker's elements with a remote_content_recovery advisory that notes the recovered hierarchy is flat. The hot path is untouched: healthy and sparse-but-valid trees never pay for the retry, and on full-coverage native screens the discovery grid probes nothing. (#75)
• record-video frame rate is no longer capped at the ~8–10 fps the screenshot-polling loop topped out at (iOS honors --fps up to 60 with smooth constant-rate playback; an Android emulator captured ~50 fps under motion). Recording no longer burns CPU decoding/re-encoding every frame. (#52, #70)
• Android record-video now preserves a static final frame through the requested stop time, and waits for stdout chunks already being delivered before finalizing the MP4. (#52, #70)
• The iOS Playground's Batch Login Flow screen no longer parks its Continue / Sign In buttons inside the band the system text-suggestion strip occupies below a focused field (#76). The strip swallowed the tap at the AX level, stranding the batch login-flow E2E cases on the email screen — the only red in a full iOS E2E pass; the email field also disables autocorrection so a typed password can no longer be mangled into the field above. Fixture-only change: no sim-use behavior is affected. (#77)

Removed

• The Device-Hub-poisoned-simulator guard (DeviceHubHIDSuppression, shipped in 0.11.0) and its SIM_USE_SKIP_DTUHIDD_CHECK escape hatch: automatic transport selection routes those simulators through dtuhidd itself, so the state the guard refused is now simply functional. Forcing the legacy transport onto a suppressed simulator (via the new debug override) fails loudly with upstream's diagnostic. (#68)
• scripts/build.sh subcommands strip, sign-frameworks, and sign-xcframeworks — static archives have no nested frameworks and their signatures are irrelevant at runtime (the executable's signature covers the linked code). (#68)