feat(ios): embedded iOS simulator (no Simulator.app) — I1 headless frame acquisition - #72
Merged
Merged
Conversation
…r.app) Proven live end-to-end: a real 1179x2556 iPhone-16 home screen captured through UMIDE's own gRPC client from a FULLY HEADLESS simulator, with Simulator.app closed. The load-bearing discovery: a `simctl boot`-ed device has no rendering surface — idb's screenshot returns "No Image available to encode" and even `simctl io screenshot` hangs ~60s. Booting via `idb_companion --boot` (IosCompanion::boot_headless) attaches the CoreSimulator framebuffer WITHOUT opening Simulator.app, and the `screenshot` RPC then returns a live frame. So the embedded panel's frame source is the on-demand screenshot RPC polled at a target fps (robust regardless of screen activity); H.264 video_stream is a future low-latency optimization (doesn't emit over companion v1.1.8). This is the iOS twin of the Android emulator gRPC path: - proto/idb.proto: a wire-compatible SUBSET of facebook/idb v1.1.8 (video_stream + hid + screenshot); trimming also sidesteps a prost name collision in the full file (DapResponse's nested `message Event` vs its `oneof event`). - build.rs compiles it alongside emulator_controller.proto (client only). - ios_grpc.rs: IosCompanion (spawn a serving companion, read its gRPC port, boot_headless, availability probe for graceful fallback), connect(), screenshot(), video_stream_h264(), payload_bytes(). - examples/ios_grab_frame.rs: the milestone proof (screenshot path primary, video_stream characterized), the iOS twin of grab_frame. idb_companion is an external dep (brew install idb-companion); available() gates the embedded path so the panel can fall back to Simulator.app when it's absent. Next: I2 wire the panel's embedded-iOS view onto this + boot_headless; I3 pointer/keyboard via the hid RPC.
…re, no Simulator.app)
…l loop) start_ios_stream(udid, frame_signal, native_size, status) -> IosStreamHandle: the iOS twin of start_emulator_stream. Boots the sim headless (IosCompanion::boot_headless — framebuffer, no Simulator.app), serves it over gRPC, then polls the screenshot RPC, decodes each PNG to a DecodedFrame (software RGBA) and pushes it into the same frame_signal the VideoFrame panel view renders. native_size feeds pointer mapping; status drives the panel's connecting overlay. Handle drop ends the loop and kills the companion (device stays booted). macOS-only module (cfg gate in panel/mod.rs). Measured live on an iPhone 16 (12/12 frames decoded, 1179x2556): the screenshot RPC costs ~450ms/call at full res, so the effective rate is ~2 fps — a usable "see your app" PREVIEW, not smooth animation. video_stream H.264 does not emit via tonic against companion v1.1.8 (bidi call hangs even with the framebuffer attached) — documented as the future rate optimization. Examples: ios_poll_stream (sustained loop proof), ios_grab_frame (UMIDE_FORCE_VIDEO characterization). Next: I2b wire the panel's embedded-iOS view onto this; I3 hid input.
… Simulator.app) Live-verified end-to-end in the app: click Start on an iPhone 16 → the panel boots the sim HEADLESS and renders its home screen inside UMIDE's right dock, with NO Simulator.app window (confirmed: Simulator.app not running). New macOS `ios_panel_portable` (the iOS twin of `android_panel_portable`): a device list whose Start boots headless via start_ios_stream (never launch_device / `open -a Simulator`), a floem `video_frame` fed by the screenshot-poll stream, a status overlay (Booting…/Connecting…), and a Stop/Hide sidebar. `ios_platform_view` chooses this embedded path when idb_companion is installed (embedded_ios_available), else falls back to the Simulator.app overlay `platform_panel` — so machines without the companion are unaffected. Header carries an EMBEDDED badge. Stop drops the stream handle (ends the poll loop + kills the companion) and shuts the device down. Pointer/keyboard over the idb hid RPC is the next slice (I3); this delivers the visible embedded view the whole brick was for.
… (hid RPC) The embedded device is now fully interactive — ALL THREE modalities verified live in the app, Simulator.app closed: a click inside the panel opened Settings on the iPhone; a drag scrolled its list; typing entered "wifi" into its search field. - ios_grpc: hid_tap / hid_swipe / hid_key over the companion's client-streaming `hid` RPC (one short stream per gesture — the pattern idb ui tap uses); char_to_hid (USB-HID usage page 0x07 incl. shifted chars) + hid_usage names; `describe` RPC added to the proto subset (ScreenDimensions only — proto3 skips the rest) for the pixels-per-point scale. - Coordinate space settled EMPIRICALLY (examples/ios_tap_test): HID touch is in screen POINTS, not pixels — a points-space tap opened Safari. The panel maps view → native pixels (same letterbox as Android) → ÷ point_scale (describe; @3x fallback). - ios_stream: IosInput (the iOS twin of EmulatorInput — bounded queue into a worker task with its own connection to the same companion), point_scale + input_out signals from start_ios_stream. - Panel: PointerDown/Up → tap-or-swipe (8pt slop, measured gesture duration — the idb HID has no touch-move, so gestures resolve on release); KeyDown → chars + Enter/Backspace/Tab/Escape/arrows. The video_frame is focusable, so clicking into the device routes typing to it.
…rick complete on PR #72
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Big brick, first slice. Embed the iOS simulator directly in UMIDE without the separate Simulator.app window — the iOS half of what the embedded Android emulator already does. Proven live end-to-end: a real 1179×2556 iPhone-16 home screen captured through UMIDE's own gRPC client from a fully headless simulator (Simulator.app closed).
The load-bearing discovery
A
simctl boot-ed device has no rendering surface — idb'sscreenshotreturns "No Image available to encode" and evensimctl io screenshothangs ~60s. Booting viaidb_companion --boot(IosCompanion::boot_headless) attaches the CoreSimulator framebuffer without opening Simulator.app, and thescreenshotRPC then returns a live frame.Architecture decision: the embedded panel's frame source is the on-demand
screenshotRPC polled at a target fps — robust regardless of screen activity, decoded via theimagecrate to RGBA and fed to the existingVideoFrame/frame_signalpipeline. H.264video_streamis a future low-latency optimization (doesn't emit over companion v1.1.8).What's in I1
proto/idb.proto— a wire-compatible subset of facebook/idb v1.1.8 (video_stream+hid+screenshot); trimming also sidesteps a prost codegen name collision in the full file.build.rscompiles it besideemulator_controller.proto(client only).src/ios_grpc.rs—IosCompanion(spawn serving companion, read gRPC port,boot_headless,available()for graceful fallback),connect(),screenshot(),video_stream_h264(),payload_bytes(). Unit-tested port parser.examples/ios_grab_frame.rs— the milestone proof (iOS twin ofgrab_frame).External dep:
brew install idb-companion;available()gates the embedded path so the panel falls back to Simulator.app when it's absent. macOS-only (iOS Simulator is macOS-only) — does not touch the Windows/Linux build.Next
boot_headless(noopen -a Simulator).hidRPC.