feat: iOS and Android simulator capture mode (v0.20260317.0)#1
Merged
ranaroussi merged 10 commits intomainfrom Mar 17, 2026
Merged
feat: iOS and Android simulator capture mode (v0.20260317.0)#1ranaroussi merged 10 commits intomainfrom
ranaroussi merged 10 commits intomainfrom
Conversation
- mode: 'simulator' with platform: 'ios' records the simulator screen - Uses xcrun simctl io recordVideo, SIGINT to stop - Auto-detects booted simulator or boots by name - CLI: --mode simulator --platform ios [--device-name, --os, --codec] - iOS test app: SwiftUI Hello World with button + UI tests - 48 tests passing (3 new: command required, platform required, android stub) - Smoke tested: real 5.3s recording of Safari on iPhone 17 Pro sim
xcodebuild test clones the simulator by default, making the recording capture an idle screen. The clone is invisible to simctl list, so clone detection isn't viable. Instead: detect xcodebuild test commands and warn users to add -parallel-testing-enabled NO -disable-concurrent-destination-testing flags, which prevent cloning and run tests on the original sim. Also provide a helpful error message if the recording file ends up empty (common symptom of cloned sim recording).
Post-processes xcodebuild test recordings with red circle + ripple indicators at tap timestamps, matching the Playwright cursor style. - xcresult.ts: parses xcresult bundle for tap activity summaries - touch-overlay.ts: ffmpeg filter to overlay red dot + ring at tap times - simulator.ts: auto-detects xcodebuild tests and applies overlay Uses geq filter to generate proper circular overlays (not drawbox). Tap timestamps extracted from xcresult ActivitySummary entries. Currently places indicators at screen center (element coordinate extraction can be added later).
Add ProofTapLogger.swift -- a XCUITest extension that logs element frames (x, y, width, height) to proof-taps.json on the simulator. Tests use element.proofTap() instead of element.tap(). After recording, simulator.ts reads the tap log from the simulator's app container and passes coordinates to the ffmpeg overlay filter. Each tap's red dot + ripple ring is now positioned exactly on the tapped element center, scaled from UIKit points to video pixels. Falls back to screen center if no tap log is found.
Add simulator capture to architecture diagram, directory structure, data flow (full recording + tap overlay pipeline), CLI flags, gotchas (xcodebuild cloning, XCUITest HID injection, simctl framebuffer, tap log location, point-to-pixel scaling), and lessons learned (post-processing vs runtime, SIGINT for simctl, xcresult limitations).
Add simulator-android.ts with: - assertAndroidReady() -- checks adb is available, finds SDK path - resolveAndroidDevice() -- finds running emulator or boots AVD by name - startAndroidRecording() -- adb shell screenrecord with 175s segment chaining for recordings >3 minutes - pullAndMergeRecording() -- adb pull + ffmpeg concat for multi-segment - enableTouchIndicators() / disableTouchIndicators() -- toggle show_touches (Android renders tap dots in screenrecord natively) Update simulator.ts to route platform=android to captureAndroid() instead of throwing a stub error. Tap indicators work automatically via Android's show_touches developer option -- no ProofTapLogger equivalent needed.
Switch Android recording from adb shell screenrecord to adb emu screenrecord which saves directly to the host filesystem. The shell screenrecord approach produces 0 frames on Apple Silicon emulators using the gfxstream graphics backend. Add ffmpeg tap overlay for Android: captureAndroid reads /tmp/proof-android-taps.json (or $PROOF_TAP_LOG) and overlays red dot + ripple at each tap position. Overlay uses scaleFactor=1 since adb emu screenrecord outputs at native device resolution. Add scaleFactor param to overlayTouchIndicators so Android (scale=1) and iOS (auto-detected 2x/3x from video width) can share the same overlay pipeline. Fix test runner: exclude test files from tsc compilation to stop dist/*.test.js from landing in dist/ and failing with missing cli.ts. Scope bun test to src/ and test-app/cli/ to exclude Playwright spec files that bun cannot run.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a new
simulatorcapture mode for recording iOS Simulator and Android emulator screens while running UI tests, with automatic tap indicator overlays.iOS
xcrun simctl io recordVideo(SIGINT to stop)--device-name+--osxcodebuild testis missing clone-prevention flagselement.tap()withelement.proofTap()to log element coordinates toproof-taps.jsonin the app containerAndroid
adb emu screenrecord(QEMU monitor protocol) -- works on Apple Silicon whereadb shell screenrecordproduces 0 frames due to the gfxstream graphics backend.webmoutput to.mp4via ffmpeg$PROOF_TAP_LOG(default/tmp/proof-android-taps.json) and overlays indicators at native device pixel coordinates (scale=1)CLI
Fixes
tsccompilation (staledist/*.test.jswere failing with missingcli.ts)bun testtosrc/andtest-app/cli/to skip Playwright.spec.tsfilesTests
51 pass, 0 fail