From c345e5e90d5084e381dec9f02e48007f15a9abe8 Mon Sep 17 00:00:00 2001 From: DjDeveloperr Date: Tue, 5 May 2026 18:33:16 -0400 Subject: [PATCH] Improve Studio stream controls and rotation sync --- README.md | 34 ++--- cli/DFPrivateSimulatorDisplayBridge.h | 1 + cli/DFPrivateSimulatorDisplayBridge.m | 42 +++++++ cli/XCWPrivateSimulatorSession.h | 1 + cli/XCWPrivateSimulatorSession.m | 4 + cli/native/XCWNativeBridge.h | 1 + cli/native/XCWNativeBridge.m | 8 ++ cli/native/XCWNativeSession.h | 1 + cli/native/XCWNativeSession.m | 4 + client/src/api/types.ts | 1 + client/src/app/AppShell.tsx | 104 +++++++++++---- client/src/app/uiState.ts | 1 - .../src/features/simulators/SimulatorMenu.tsx | 30 ++--- .../src/features/stream/streamWorkerClient.ts | 42 +++++-- client/src/features/stream/useLiveStream.ts | 11 +- client/src/styles/components.css | 12 +- docs/api/rest.md | 10 +- docs/guide/video.md | 2 +- scripts/studio-provider-bridge.mjs | 14 ++- server/src/api/routes.rs | 118 +++++++++++++----- server/src/native/bridge.rs | 4 + server/src/native/ffi.rs | 1 + server/src/simulators/registry.rs | 1 + server/src/simulators/session.rs | 1 + skills/simdeck/SKILL.md | 95 +++++++------- 25 files changed, 369 insertions(+), 174 deletions(-) diff --git a/README.md b/README.md index 39e95ce..962efba 100644 --- a/README.md +++ b/README.md @@ -36,11 +36,12 @@ view inside the editor. ## Features - Local simulator video stream over browser-native WebRTC H.264 -- Full simulator control & inspection using private accessibility APIs +- Full simulator control & inspection using private accessibility APIs - available using `simdeck` CLI +- Real-time screen `describe` command using accessibility view tree - available in token-efficient format for agents - CoreSimulator chrome asset rendering for device bezels - NativeScript, React Native, UIKit and SwiftUI runtime inspector plugins to view app's view hierarchy live - `simdeck/test` for fast JS/TS app tests that can query accessibility state and drive simulator controls. -- SimDeck Studio for automatic PR deployments to on-demand simulators +- SimDeck Studio for sharing Simulator streams & automatic PR deployments to on-demand simulators ## Documentation @@ -59,9 +60,10 @@ To focus a specific simulator by name or UDID, pass it as the only argument: simdeck "iPhone 17 Pro Max" ``` -Use `simdeck ui --open` or `simdeck daemon start` when you want a reusable background daemon instead. -The no-subcommand lifecycle shortcuts are `simdeck -d` for detached start, `simdeck -k` to kill the background daemon, and `simdeck -r` to restart it. -The served loopback browser UI receives the generated API access token automatically. LAN browsers pair with the printed code before receiving the API cookie. +`simdeck -d` for detached start, `simdeck -k` to kill the background daemon, and `simdeck -r` to restart it. + +The served loopback browser UI receives the generated API access token automatically. +LAN clients should pair with the printed code before receiving the API cookie. SimDeck Studio providers run the daemon on loopback and use `scripts/studio-provider-bridge.mjs` for outbound control-plane communication @@ -69,7 +71,7 @@ with Studio. Studio hosts the browser UI and proxies SimDeck REST requests over that bridge while WebRTC media still negotiates directly between the browser and runner through ICE. -Expose a local simulator through Studio with one command: +Expose a local simulator through SimDeck Studio with one command: ```sh simdeck studio expose "iPhone 17 Pro" @@ -77,15 +79,7 @@ simdeck studio expose "iPhone 17 Pro" The command starts or reuses the local daemon, creates an ephemeral Studio session, prints a unique `https://simdeck.djdev.me/simulator/...` URL, and keeps -the outbound bridge alive until you press Ctrl-C. It uses software H.264 by -default with realtime stream settings for remote viewing, and prints the active -codec/profile when it starts. Studio defaults to the `smooth` stream quality -profile (`1170` longest edge, dynamic up to `60` fps). Use -`--stream-quality quality|balanced|fast|smooth|economy|ci-software` to override it, -or pass `--video-codec hardware` when a dedicated hardware encoder is preferable. -The remote viewer renders live video with the browser's native video element; -the canvas is only used for input geometry. Remote viewers can choose 15, 30, -or 60 fps in the browser stream menu. +the outbound bridge alive until you press Ctrl-C. CLI commands automatically use the same warm daemon: @@ -117,14 +111,6 @@ more important than full-resolution smoothness: simdeck daemon start --video-codec software --low-latency ``` -Local browser streams default to realtime WebRTC delivery with the `quality` -profile on VideoToolbox H.264: full resolution, 120 fps, and a high bitrate floor. On -high-refresh local displays, raise the local stream target explicitly: - -```sh -simdeck daemon restart --local-stream-fps 240 -``` - Restart the CoreSimulator service layer when `simctl` reports a stale service version or the live display gets stuck before the first frame: @@ -255,7 +241,7 @@ React Fiber commits. ## VS Code -Install the `nativescript.simdeck` extension from the VS Code Marketplace, then +Install the `nativescript.simdeck-vscode` extension from the VS Code Marketplace, then run `SimDeck: Open Simulator View` from the Command Palette. The extension opens the simulator inside a VS Code panel and auto-starts the local daemon when it is not already reachable. diff --git a/cli/DFPrivateSimulatorDisplayBridge.h b/cli/DFPrivateSimulatorDisplayBridge.h index 2fb1552..c4bce91 100644 --- a/cli/DFPrivateSimulatorDisplayBridge.h +++ b/cli/DFPrivateSimulatorDisplayBridge.h @@ -38,6 +38,7 @@ NS_SWIFT_NAME(PrivateSimulatorDisplayBridge) @property (nonatomic, readonly, getter=isDisplayReady) BOOL displayReady; @property (nonatomic, readonly) NSString *displayStatus; @property (nonatomic, readonly) CGSize displaySize; +@property (nonatomic, readonly) NSInteger rotationQuarterTurns; - (nullable CVPixelBufferRef)copyPixelBuffer CF_RETURNS_RETAINED; diff --git a/cli/DFPrivateSimulatorDisplayBridge.m b/cli/DFPrivateSimulatorDisplayBridge.m index dfa930a..437e653 100644 --- a/cli/DFPrivateSimulatorDisplayBridge.m +++ b/cli/DFPrivateSimulatorDisplayBridge.m @@ -1455,6 +1455,32 @@ static double DFNormalizedDegrees(double value) { return normalized; } +static NSInteger DFRotationQuarterTurnsForDegrees(double degrees) { + NSInteger turns = (NSInteger)llround(DFNormalizedDegrees(degrees) / 90.0) % 4; + if (turns < 0) { + turns += 4; + } + return turns; +} + +static void DFReconcileRotationWithDisplaySize(double *rotationDegrees, CGSize displaySize) { + if (rotationDegrees == NULL || displaySize.width <= 0.0 || displaySize.height <= 0.0) { + return; + } + + double aspectDelta = fabs(displaySize.width - displaySize.height); + if (aspectDelta < 1.0) { + return; + } + + NSInteger currentTurns = DFRotationQuarterTurnsForDegrees(*rotationDegrees); + BOOL displayIsLandscape = displaySize.width > displaySize.height; + BOOL rotationIsLandscape = (currentTurns % 2) != 0; + if (displayIsLandscape != rotationIsLandscape) { + *rotationDegrees = displayIsLandscape ? 90.0 : 0.0; + } +} + static NSArray * DFInterestingSelectorsForObject(id object) { if (object == nil) { return @[]; @@ -2758,6 +2784,7 @@ - (nullable instancetype)initWithUDID:(NSString *)udid size_t width = CVPixelBufferGetWidth(pixelBuffer); size_t height = CVPixelBufferGetHeight(pixelBuffer); strongSelf->_displayPixelSize = CGSizeMake((CGFloat)width, (CGFloat)height); + DFReconcileRotationWithDisplaySize(&strongSelf->_deviceRotationDegrees, strongSelf->_displayPixelSize); [strongSelf notifyDelegateOfFrame:pixelBuffer]; DFRunOnMainAsync(^{ if (strongSelf->_headlessHostWindow != nil) { @@ -3435,6 +3462,21 @@ - (CGSize)displaySize { return size; } +- (NSInteger)rotationQuarterTurns { + __block NSInteger turns = 0; + dispatch_block_t work = ^{ + turns = DFRotationQuarterTurnsForDegrees(self->_deviceRotationDegrees); + }; + + if (dispatch_get_specific(DFPrivateSimulatorCallbackQueueKey) != NULL) { + work(); + } else { + dispatch_sync(_callbackQueue, work); + } + + return turns; +} + - (BOOL)sendTouchAtNormalizedX:(double)normalizedX normalizedY:(double)normalizedY phase:(DFPrivateSimulatorTouchPhase)phase diff --git a/cli/XCWPrivateSimulatorSession.h b/cli/XCWPrivateSimulatorSession.h index 680d43b..90dc69a 100644 --- a/cli/XCWPrivateSimulatorSession.h +++ b/cli/XCWPrivateSimulatorSession.h @@ -23,6 +23,7 @@ typedef void (^XCWPrivateSimulatorEncodedFrameHandler)(NSData *sampleData, @property (nonatomic, readonly, getter=isDisplayReady) BOOL displayReady; @property (nonatomic, copy, readonly) NSString *displayStatus; @property (nonatomic, readonly) CGSize displaySize; +@property (nonatomic, readonly) NSInteger rotationQuarterTurns; @property (nonatomic, readonly) NSUInteger frameSequence; - (BOOL)waitUntilReadyWithTimeout:(NSTimeInterval)timeout; diff --git a/cli/XCWPrivateSimulatorSession.m b/cli/XCWPrivateSimulatorSession.m index bed1b15..18e9eac 100644 --- a/cli/XCWPrivateSimulatorSession.m +++ b/cli/XCWPrivateSimulatorSession.m @@ -229,6 +229,10 @@ - (CGSize)displaySize { return size; } +- (NSInteger)rotationQuarterTurns { + return _displayBridge.rotationQuarterTurns; +} + - (NSUInteger)frameSequence { __block NSUInteger sequence = 0; dispatch_sync(_stateQueue, ^{ diff --git a/cli/native/XCWNativeBridge.h b/cli/native/XCWNativeBridge.h index b2be93a..bc02dae 100644 --- a/cli/native/XCWNativeBridge.h +++ b/cli/native/XCWNativeBridge.h @@ -74,6 +74,7 @@ void xcw_native_session_request_refresh(void * _Nonnull handle); void xcw_native_session_request_keyframe(void * _Nonnull handle); void xcw_native_session_reconfigure_video_encoder(void * _Nonnull handle); char * _Nullable xcw_native_session_video_encoder_stats(void * _Nonnull handle, char * _Nullable * _Nullable error_message); +int32_t xcw_native_session_rotation_quarter_turns(void * _Nonnull handle); bool xcw_native_session_send_touch(void * _Nonnull handle, double x, double y, const char * _Nonnull phase, char * _Nullable * _Nullable error_message); bool xcw_native_session_send_multitouch(void * _Nonnull handle, double x1, double y1, double x2, double y2, const char * _Nonnull phase, char * _Nullable * _Nullable error_message); bool xcw_native_session_send_key(void * _Nonnull handle, uint16_t key_code, uint32_t modifiers, char * _Nullable * _Nullable error_message); diff --git a/cli/native/XCWNativeBridge.m b/cli/native/XCWNativeBridge.m index 02aeb13..bdbaa45 100644 --- a/cli/native/XCWNativeBridge.m +++ b/cli/native/XCWNativeBridge.m @@ -702,6 +702,14 @@ void xcw_native_session_reconfigure_video_encoder(void *handle) { } } +int32_t xcw_native_session_rotation_quarter_turns(void *handle) { + @autoreleasepool { + NSInteger turns = [XCWNativeSessionFromHandle(handle) rotationQuarterTurns]; + NSInteger normalized = ((turns % 4) + 4) % 4; + return (int32_t)normalized; + } +} + bool xcw_native_session_send_touch(void *handle, double x, double y, const char *phase, char **error_message) { @autoreleasepool { NSError *error = nil; diff --git a/cli/native/XCWNativeSession.h b/cli/native/XCWNativeSession.h index b3a6070..db3fd3f 100644 --- a/cli/native/XCWNativeSession.h +++ b/cli/native/XCWNativeSession.h @@ -16,6 +16,7 @@ NS_ASSUME_NONNULL_BEGIN - (void)requestKeyFrame; - (void)reconfigureVideoEncoder; - (NSDictionary *)videoEncoderStats; +- (NSInteger)rotationQuarterTurns; - (BOOL)sendTouchAtX:(double)x y:(double)y phase:(NSString *)phase diff --git a/cli/native/XCWNativeSession.m b/cli/native/XCWNativeSession.m index 71aa2d9..d96c28d 100644 --- a/cli/native/XCWNativeSession.m +++ b/cli/native/XCWNativeSession.m @@ -106,6 +106,10 @@ - (NSDictionary *)videoEncoderStats { return [self.session videoEncoderStats]; } +- (NSInteger)rotationQuarterTurns { + return self.session.rotationQuarterTurns; +} + - (BOOL)sendTouchAtX:(double)x y:(double)y phase:(NSString *)phase diff --git a/client/src/api/types.ts b/client/src/api/types.ts index f8345d4..44a65b5 100644 --- a/client/src/api/types.ts +++ b/client/src/api/types.ts @@ -4,6 +4,7 @@ export interface PrivateDisplayInfo { displayWidth: number; displayHeight: number; frameSequence: number; + rotationQuarterTurns?: number; } export interface SimulatorMetadata { diff --git a/client/src/app/AppShell.tsx b/client/src/app/AppShell.tsx index 05c184e..b3f9580 100644 --- a/client/src/app/AppShell.tsx +++ b/client/src/app/AppShell.tsx @@ -40,6 +40,7 @@ import type { TouchPhase, } from "../api/types"; import { AccessibilityInspector } from "../features/accessibility/AccessibilityInspector"; +import { isEditableTarget } from "../features/input/keycodes"; import { useKeyboardInput } from "../features/input/useKeyboardInput"; import { usePointerInput } from "../features/input/usePointerInput"; import { simulatorRuntimeLabel } from "../features/simulators/simulatorDisplay"; @@ -68,6 +69,7 @@ import { clampZoom, computeChromeScreenBorderRadius, computeChromeScreenRect, + normalizeQuarterTurns, screenAspectRatio, shellSize, } from "../features/viewport/viewportMath"; @@ -81,6 +83,7 @@ import { ACCESSIBILITY_SOURCE_STORAGE_KEY, clearLegacyVolatileUiState, DEFAULT_VIEWPORT_STATE, + DEBUG_VISIBLE_STORAGE_KEY, HIERARCHY_VISIBLE_STORAGE_KEY, readPersistedUiState, readStoredAccessibilitySource, @@ -107,21 +110,13 @@ const REMOTE_STREAM_DEFAULTS: StreamConfig = { fps: 30, quality: "balanced", }; -const STREAM_CONFIG_SYNC_INTERVAL_MS = 5000; +const STREAM_CONFIG_SYNC_INTERVAL_MS = 1500; const STREAM_CONFIG_USER_CHANGE_GRACE_MS = 1000; const STREAM_ENCODER_VALUES = new Set([ "auto", "hardware", "software", ]); -const STREAM_QUALITY_VALUES = new Set([ - "balanced", - "ci-software", - "economy", - "fast", - "quality", - "smooth", -]); clearLegacyVolatileUiState(); interface StreamQualityResponse { @@ -202,6 +197,20 @@ function simulatorDisplayReady(simulator: SimulatorMetadata): boolean { ); } +function normalizeSimulatorRotationQuarterTurns( + simulator: SimulatorMetadata | null, +): number | null { + const display = simulator?.privateDisplay; + if ( + !simulator?.isBooted || + !display?.displayReady || + !Number.isFinite(display.rotationQuarterTurns) + ) { + return null; + } + return normalizeQuarterTurns(display.rotationQuarterTurns ?? 0); +} + function mergeAccessibilitySources( ...sources: unknown[] ): AccessibilitySource[] { @@ -261,7 +270,9 @@ export function AppShell({ refresh, simulators, } = useSimulatorList({ remote: remoteStream }); - const [debugVisible, setDebugVisible] = useState(false); + const [debugVisible, setDebugVisible] = useState(() => + readStoredFlag(DEBUG_VISIBLE_STORAGE_KEY), + ); const [hierarchyVisible, setHierarchyVisible] = useState(() => readStoredFlag(HIERARCHY_VISIBLE_STORAGE_KEY), ); @@ -513,7 +524,7 @@ export function AppShell({ streamConfigUserChangeAtRef.current = Date.now(); setStreamConfigReady(true); setStreamConfigApplyKey((current) => current + 1); - setStreamConfig((current) => ({ ...current, quality })); + setStreamConfig((current) => ({ ...current, maxEdge: undefined, quality })); }, []); useEffect(() => { @@ -597,6 +608,12 @@ export function AppShell({ (shouldRenderChrome && !chromeProfileReady) || (viewportChromeProfile && chromeUrl), ); + const simulatorRotationQuarterTurns = + normalizeSimulatorRotationQuarterTurns(selectedSimulator); + + useEffect(() => { + writeStoredFlag(DEBUG_VISIBLE_STORAGE_KEY, debugVisible); + }, [debugVisible]); useEffect(() => { writeStoredFlag(HIERARCHY_VISIBLE_STORAGE_KEY, hierarchyVisible); @@ -855,6 +872,20 @@ export function AppShell({ } }, [isBooted]); + useEffect(() => { + if (simulatorRotationQuarterTurns == null) { + return; + } + setRotationQuarterTurns((current) => { + const normalizedCurrent = normalizeQuarterTurns(current); + if (normalizedCurrent === simulatorRotationQuarterTurns) { + return current; + } + beginZoomAnimation(); + return simulatorRotationQuarterTurns; + }); + }, [simulatorRotationQuarterTurns]); + useEffect(() => { setChromeLoaded(!chromeRequired); }, [chromeRequired, chromeUrl]); @@ -918,6 +949,30 @@ export function AppShell({ }; }, [menuOpen]); + useEffect(() => { + function handleWindowKeyDown(event: KeyboardEvent) { + if ( + isEditableTarget(event.target) || + event.altKey || + event.metaKey || + !event.ctrlKey || + !event.shiftKey || + event.key.toLowerCase() !== "d" + ) { + return; + } + + event.preventDefault(); + event.stopImmediatePropagation(); + setDebugVisible((current) => !current); + } + + window.addEventListener("keydown", handleWindowKeyDown); + return () => { + window.removeEventListener("keydown", handleWindowKeyDown); + }; + }, []); + useEffect(() => { setPan((currentPan) => { const nextPan = clampPan( @@ -1496,15 +1551,14 @@ export function AppShell({ if (!selectedSimulator) { return; } + beginZoomAnimation(); if (sendControl(selectedSimulator.udid, { type: "rotateLeft" })) { setRotationQuarterTurns((current) => (current + 3) % 4); - setStreamStamp(Date.now()); return; } void runAction(async () => { await rotateLeft(selectedSimulator.udid); setRotationQuarterTurns((current) => (current + 3) % 4); - setStreamStamp(Date.now()); }, false); }} onOpenBundlePrompt={promptForBundleID} @@ -1513,15 +1567,14 @@ export function AppShell({ if (!selectedSimulator) { return; } + beginZoomAnimation(); if (sendControl(selectedSimulator.udid, { type: "rotateRight" })) { setRotationQuarterTurns((current) => (current + 1) % 4); - setStreamStamp(Date.now()); return; } void runAction(async () => { await rotateRight(selectedSimulator.udid); setRotationQuarterTurns((current) => (current + 1) % 4); - setStreamStamp(Date.now()); }, false); }} onStreamEncoderChange={updateStreamEncoder} @@ -1760,12 +1813,21 @@ function normalizeStreamQuality( value: string | undefined, fallback: StreamQualityPreset, ): StreamQualityPreset { - const normalized = value?.trim().toLowerCase() as - | StreamQualityPreset - | undefined; - return normalized && STREAM_QUALITY_VALUES.has(normalized) - ? normalized - : fallback; + const normalized = value?.trim().toLowerCase(); + if (normalized === "quality") { + return "quality"; + } + if ( + normalized === "balanced" || + normalized === "smooth" || + normalized === "fast" + ) { + return "balanced"; + } + if (normalized === "economy" || normalized === "ci-software") { + return "economy"; + } + return fallback; } function normalizeStreamFps( diff --git a/client/src/app/uiState.ts b/client/src/app/uiState.ts index eb70239..6c298e3 100644 --- a/client/src/app/uiState.ts +++ b/client/src/app/uiState.ts @@ -81,7 +81,6 @@ export function clearLegacyVolatileUiState(): void { return; } - window.localStorage.removeItem(DEBUG_VISIBLE_STORAGE_KEY); try { const parsed = JSON.parse( window.localStorage.getItem(UI_STATE_STORAGE_KEY) ?? "{}", diff --git a/client/src/features/simulators/SimulatorMenu.tsx b/client/src/features/simulators/SimulatorMenu.tsx index 92a124b..f220b0d 100644 --- a/client/src/features/simulators/SimulatorMenu.tsx +++ b/client/src/features/simulators/SimulatorMenu.tsx @@ -72,12 +72,6 @@ export function SimulatorMenu({ ) ? [] : [{ label: String(streamConfig.fps), value: streamConfig.fps }]; - const activeQualityOption = STREAM_QUALITY_OPTIONS.some( - (option) => option.value === streamConfig.quality, - ) - ? [] - : [{ label: streamConfig.quality, value: streamConfig.quality }]; - return (
- ), - )} + {STREAM_QUALITY_OPTIONS.map((option) => ( + + ))}
@@ -246,9 +238,7 @@ const STREAM_QUALITY_OPTIONS: Array<{ value: StreamQualityPreset; }> = [ { label: "Quality", value: "quality" }, - { label: "Smooth", value: "smooth" }, { label: "Balanced", value: "balanced" }, - { label: "Fast", value: "fast" }, { label: "Economy", value: "economy" }, ]; diff --git a/client/src/features/stream/streamWorkerClient.ts b/client/src/features/stream/streamWorkerClient.ts index ff06dff..6c0f4b2 100644 --- a/client/src/features/stream/streamWorkerClient.ts +++ b/client/src/features/stream/streamWorkerClient.ts @@ -200,11 +200,13 @@ class WebRtcStreamClient implements StreamClientBackend { private reconnectDelayMs = WEBRTC_RECONNECT_BASE_DELAY_MS; private reconnecting = false; private remoteMode = false; - private reportedVideoConfig = false; + private reportedVideoHeight = 0; + private reportedVideoWidth = 0; private receiverStatsInterval = 0; private receiverStatsSeen = false; private shouldReconnect = false; private streamConfigGeneration = 0; + private streamTarget: StreamConnectTarget | null = null; private telemetryChannel: RTCDataChannel | null = null; private stats: StreamStats = createEmptyStreamStats(); private video: HTMLVideoElement | null = null; @@ -272,12 +274,14 @@ class WebRtcStreamClient implements StreamClientBackend { const generation = ++this.connectGeneration; this.shouldReconnect = true; this.remoteMode = Boolean(target.remote); + this.streamTarget = target; if (!wasReconnecting) { this.reconnectDelayMs = WEBRTC_RECONNECT_BASE_DELAY_MS; } this.resetFrameStateForNewConnection(); this.diagnostics = createWebRtcDiagnostics(); - this.reportedVideoConfig = false; + this.reportedVideoHeight = 0; + this.reportedVideoWidth = 0; this.receiverStatsSeen = false; this.onMessage({ type: "status", @@ -285,12 +289,18 @@ class WebRtcStreamClient implements StreamClientBackend { }); try { - if (!target.remote) { - await postStreamConfigWithAuthRetry(target.streamConfig); - if (generation !== this.connectGeneration) { - return; + try { + await postStreamConfigWithAuthRetry(target.streamConfig, { + remote: target.remote, + }); + } catch (error) { + if (!target.remote) { + throw error; } } + if (generation !== this.connectGeneration) { + return; + } const health = await fetchHealth().catch(() => null); if (generation !== this.connectGeneration) { return; @@ -401,7 +411,7 @@ class WebRtcStreamClient implements StreamClientBackend { this.clearIceRestartTimeout(); this.iceRestartInFlight = false; this.reconnectDelayMs = WEBRTC_RECONNECT_BASE_DELAY_MS; - if (this.reportedVideoConfig) { + if (this.reportedVideoWidth > 0 && this.reportedVideoHeight > 0) { this.onMessage({ type: "status", status: { detail: "WebRTC media connected", state: "streaming" }, @@ -450,6 +460,7 @@ class WebRtcStreamClient implements StreamClientBackend { this.shouldReconnect = false; this.reconnecting = false; this.connectGeneration += 1; + this.streamTarget = null; this.clearReconnectTimeout(); this.clearDisconnectGraceTimeout(); this.clearIceRestartTimeout(); @@ -471,6 +482,11 @@ class WebRtcStreamClient implements StreamClientBackend { if (generation !== this.streamConfigGeneration) { return; } + const target = this.streamTarget; + if (target && this.shouldReconnect) { + await this.connect({ ...target, streamConfig: config }); + return; + } this.sendControl({ forceKeyframe: true, type: "streamControl" }); } @@ -535,7 +551,8 @@ class WebRtcStreamClient implements StreamClientBackend { this.video.remove(); } this.video = null; - this.reportedVideoConfig = false; + this.reportedVideoHeight = 0; + this.reportedVideoWidth = 0; this.controlChannel?.close(); if (activeWebRtcControlChannel === this.controlChannel) { activeWebRtcControlChannel = null; @@ -1001,10 +1018,14 @@ class WebRtcStreamClient implements StreamClientBackend { } private reportVideoConfig(width: number, height: number) { - if (this.reportedVideoConfig) { + if ( + this.reportedVideoWidth === width && + this.reportedVideoHeight === height + ) { return; } - this.reportedVideoConfig = true; + this.reportedVideoWidth = width; + this.reportedVideoHeight = height; this.onMessage({ type: "video-config", size: { height, width }, @@ -1222,7 +1243,6 @@ function postStreamConfig(config: StreamConfig): Promise { return fetch(apiUrl("/api/stream-quality"), { body: JSON.stringify({ fps: config.fps, - maxEdge: config.maxEdge, profile: config.quality, videoCodec: config.encoder, }), diff --git a/client/src/features/stream/useLiveStream.ts b/client/src/features/stream/useLiveStream.ts index 54c90ad..d3eccf5 100644 --- a/client/src/features/stream/useLiveStream.ts +++ b/client/src/features/stream/useLiveStream.ts @@ -290,14 +290,7 @@ export function useLiveStream({ return () => { workerClient.disconnect(); }; - }, [ - canvasElement, - simulator?.isBooted, - simulator?.udid, - paused, - remote, - streamConfig?.encoder, - ]); + }, [canvasElement, simulator?.isBooted, simulator?.udid, paused, remote]); useEffect(() => { if ( @@ -313,7 +306,9 @@ export function useLiveStream({ paused, simulator?.isBooted, streamConfigApplyKey, + streamConfig?.encoder, streamConfig?.fps, + streamConfig?.maxEdge, streamConfig?.quality, ]); diff --git a/client/src/styles/components.css b/client/src/styles/components.css index 145fd0c..c1f26dc 100644 --- a/client/src/styles/components.css +++ b/client/src/styles/components.css @@ -1157,7 +1157,17 @@ } .device-anchor.animated { - transition: transform 180ms ease; + transition: transform 240ms cubic-bezier(0.2, 0, 0, 1); +} + +.device-anchor.animated .device-frame { + transition: + width 240ms cubic-bezier(0.2, 0, 0, 1), + height 240ms cubic-bezier(0.2, 0, 0, 1); +} + +.device-anchor.animated .device-presentation { + transition: transform 240ms cubic-bezier(0.2, 0, 0, 1); } .device-bezel { diff --git a/docs/api/rest.md b/docs/api/rest.md index a848e99..7570599 100644 --- a/docs/api/rest.md +++ b/docs/api/rest.md @@ -87,8 +87,11 @@ quality. ``` `videoCodec` accepts `hardware` or `software` from the UI, and the API also -accepts `auto`. `fps` is clamped to the local stream range. `profile` accepts -`quality`, `balanced`, `fast`, `smooth`, `economy`, or `ci-software`. +accepts `auto`. `fps` is clamped to the local stream range. Browser viewers show +three profiles: `quality`, `balanced`, and `economy`. The API still accepts the +legacy `fast`, `smooth`, and `ci-software` profiles for CLI/provider +compatibility. When `profile` is provided, its resolution preset is applied; +send `maxEdge` without `profile` for a custom resolution cap. ## Simulator inventory @@ -110,7 +113,8 @@ Returns every simulator known to the native bridge, enriched with any session st "displayStatus": "running", "displayWidth": 1170, "displayHeight": 2532, - "frameSequence": 8124 + "frameSequence": 8124, + "rotationQuarterTurns": 0 } } ] diff --git a/docs/guide/video.md b/docs/guide/video.md index 01e8305..e59c851 100644 --- a/docs/guide/video.md +++ b/docs/guide/video.md @@ -32,7 +32,7 @@ It is CLI-only because it is meant for less capable machines where freshness matters more than maximum smoothness. The requested encoder mode is reported to clients in the JSON `videoCodec` field on `GET /api/health`. -The browser UI exposes stream controls for encoder, FPS, and quality. Local browser sessions default to hardware H.264, 120 fps, and `quality`/full resolution with FPS choices of 30, 60, and 120. Remote browser sessions default to software H.264, 30 fps, and `balanced` with FPS choices of 15, 30, and 60. +The browser UI exposes stream controls for encoder, FPS, and three quality choices: `quality`, `balanced`, and `economy`. Local browser sessions default to hardware H.264, 120 fps, and `quality`/full resolution with FPS choices of 30, 60, and 120. Remote browser sessions default to software H.264, 30 fps, and `balanced` with FPS choices of 15, 30, and 60. ## Remote WebRTC ICE diff --git a/scripts/studio-provider-bridge.mjs b/scripts/studio-provider-bridge.mjs index 7ae27f8..9166cb4 100644 --- a/scripts/studio-provider-bridge.mjs +++ b/scripts/studio-provider-bridge.mjs @@ -124,12 +124,18 @@ if (isMainModule()) { providerToken, }, ); + if (stopped) { + break; + } if (!next || !next.request) { await sleep(250); continue; } runProviderRequest(next.request); } catch (error) { + if (stopped) { + break; + } console.error( `[simdeck-provider-bridge] ${error instanceof Error ? error.message : String(error)}`, ); @@ -169,7 +175,10 @@ async function registerProvider() { }); registered = true; lastRegisterAt = Date.now(); - if (shouldStopForLocalMetadata(metadata, localDaemonProcessExited())) { + if ( + !stopped && + shouldStopForLocalMetadata(metadata, localDaemonProcessExited()) + ) { providerMarkedTerminal = true; stopped = true; await markProviderFailed( @@ -204,6 +213,9 @@ function localDaemonProcessExited() { } function updateLocalAvailability(metadata) { + if (stopped) { + return; + } if (metadata.ok) { localUnavailableSince = 0; return; diff --git a/server/src/api/routes.rs b/server/src/api/routes.rs index 99ea193..6d908f9 100644 --- a/server/src/api/routes.rs +++ b/server/src/api/routes.rs @@ -128,6 +128,14 @@ struct ActiveStreamQualityState { video_codec: String, } +#[derive(Debug, Eq, PartialEq)] +struct StreamQualityLimits { + max_edge: u32, + fps: u32, + min_bitrate: u32, + bits_per_pixel: u32, +} + const STREAM_QUALITY_PROFILES: &[StreamQualityProfile] = &[ StreamQualityProfile { id: "ci-software", @@ -179,6 +187,8 @@ const STREAM_QUALITY_PROFILES: &[StreamQualityProfile] = &[ }, ]; +const VISIBLE_STREAM_QUALITY_PROFILE_IDS: &[&str] = &["quality", "balanced", "economy"]; + static STREAM_CONFIG_LOCK: OnceLock> = OnceLock::new(); #[derive(Deserialize, Clone)] @@ -644,26 +654,7 @@ async fn set_stream_quality( .filter(|value| !value.is_empty()) .map(stream_quality_profile) .transpose()?; - let max_edge = payload - .max_edge - .or_else(|| profile.map(|profile| profile.max_edge)) - .unwrap_or(1440) - .clamp(320, 4096); - let fps = payload - .fps - .or_else(|| profile.map(|profile| profile.fps)) - .unwrap_or(30) - .clamp(10, 240); - let min_bitrate = payload - .min_bitrate - .or_else(|| profile.map(|profile| profile.min_bitrate)) - .unwrap_or(3_000_000) - .clamp(200_000, 60_000_000); - let bits_per_pixel = payload - .bits_per_pixel - .or_else(|| profile.map(|profile| profile.bits_per_pixel)) - .unwrap_or(4) - .clamp(1, 10); + let limits = resolved_stream_quality_limits(&payload, profile); let _stream_config_guard = STREAM_CONFIG_LOCK .get_or_init(|| StdMutex::new(())) @@ -672,23 +663,26 @@ async fn set_stream_quality( let current = current_stream_quality_state(active_video_codec(&state.config)); let next_video_codec = video_codec.unwrap_or(current.video_codec.as_str()); let next_profile = profile.map(|profile| profile.id).unwrap_or("custom"); - if current.max_edge == max_edge - && current.fps == fps - && current.min_bitrate == min_bitrate - && current.bits_per_pixel == bits_per_pixel + if current.max_edge == limits.max_edge + && current.fps == limits.fps + && current.min_bitrate == limits.min_bitrate + && current.bits_per_pixel == limits.bits_per_pixel && current.profile == next_profile && current.video_codec == next_video_codec { return Ok(json(json_value!(stream_quality_response(&state.config)))); } - env::set_var("SIMDECK_REALTIME_MAX_EDGE", max_edge.to_string()); - env::set_var("SIMDECK_REALTIME_FPS", fps.to_string()); - env::set_var("SIMDECK_LOCAL_STREAM_FPS", fps.to_string()); - env::set_var("SIMDECK_REALTIME_MIN_BITRATE", min_bitrate.to_string()); + env::set_var("SIMDECK_REALTIME_MAX_EDGE", limits.max_edge.to_string()); + env::set_var("SIMDECK_REALTIME_FPS", limits.fps.to_string()); + env::set_var("SIMDECK_LOCAL_STREAM_FPS", limits.fps.to_string()); + env::set_var( + "SIMDECK_REALTIME_MIN_BITRATE", + limits.min_bitrate.to_string(), + ); env::set_var( "SIMDECK_REALTIME_BITS_PER_PIXEL", - bits_per_pixel.to_string(), + limits.bits_per_pixel.to_string(), ); if let Some(profile) = profile { env::set_var("SIMDECK_STREAM_QUALITY_PROFILE", profile.id); @@ -710,7 +704,11 @@ fn stream_quality_response(config: &Config) -> Value { "ok": true, "quality": stream_quality_state_value(&quality), "videoCodec": video_codec, - "profiles": STREAM_QUALITY_PROFILES.iter().map(stream_quality_profile_value).collect::>() + "profiles": STREAM_QUALITY_PROFILES + .iter() + .filter(|profile| VISIBLE_STREAM_QUALITY_PROFILE_IDS.contains(&profile.id)) + .map(stream_quality_profile_value) + .collect::>() }) } @@ -795,6 +793,34 @@ fn stream_quality_profile_value(profile: &StreamQualityProfile) -> Value { }) } +fn resolved_stream_quality_limits( + payload: &StreamQualityPayload, + profile: Option, +) -> StreamQualityLimits { + StreamQualityLimits { + max_edge: profile + .map(|profile| profile.max_edge) + .or(payload.max_edge) + .unwrap_or(1440) + .clamp(320, 4096), + fps: payload + .fps + .or_else(|| profile.map(|profile| profile.fps)) + .unwrap_or(30) + .clamp(10, 240), + min_bitrate: payload + .min_bitrate + .or_else(|| profile.map(|profile| profile.min_bitrate)) + .unwrap_or(3_000_000) + .clamp(200_000, 60_000_000), + bits_per_pixel: payload + .bits_per_pixel + .or_else(|| profile.map(|profile| profile.bits_per_pixel)) + .unwrap_or(4) + .clamp(1, 10), + } +} + fn env_u32(name: &str, fallback: u32, minimum: u32, maximum: u32) -> u32 { env::var(name) .ok() @@ -3448,9 +3474,10 @@ mod tests { compact_accessibility_snapshot, element_matches_selector, first_matching_element, inspector_available_sources, normalize_inspector_node, normalize_screen_point_from_snapshot, normalized_gesture_coordinates, - parse_lsof_tcp_listener, split_filter_values, suppress_native_ax_translation_error, - tap_point_from_snapshot, trim_tree_depth, AccessibilityHierarchySource, - ElementSelectorPayload, InspectorSession, InspectorSessionTransport, SOURCE_NATIVE_AX, + parse_lsof_tcp_listener, resolved_stream_quality_limits, split_filter_values, + stream_quality_profile, suppress_native_ax_translation_error, tap_point_from_snapshot, + trim_tree_depth, AccessibilityHierarchySource, ElementSelectorPayload, InspectorSession, + InspectorSessionTransport, StreamQualityLimits, StreamQualityPayload, SOURCE_NATIVE_AX, SOURCE_NATIVE_SCRIPT, SOURCE_REACT_NATIVE, SOURCE_SWIFTUI, SOURCE_UIKIT, }; use serde_json::{json, Value}; @@ -3495,6 +3522,31 @@ mod tests { )); } + #[test] + fn named_stream_quality_profile_controls_resolution_over_stale_max_edge() { + let payload = StreamQualityPayload { + profile: Some("quality".to_owned()), + video_codec: None, + max_edge: Some(1280), + fps: None, + min_bitrate: None, + bits_per_pixel: None, + }; + + assert_eq!( + resolved_stream_quality_limits( + &payload, + Some(stream_quality_profile("quality").unwrap()) + ), + StreamQualityLimits { + max_edge: 4096, + fps: 60, + min_bitrate: 60_000_000, + bits_per_pixel: 10, + } + ); + } + #[test] fn first_matching_element_searches_descendants() { let found = first_matching_element(&accessibility_snapshot(), &selector()).unwrap(); diff --git a/server/src/native/bridge.rs b/server/src/native/bridge.rs index 1a7c901..477cc25 100644 --- a/server/src/native/bridge.rs +++ b/server/src/native/bridge.rs @@ -658,6 +658,10 @@ impl NativeSession { } } + pub fn rotation_quarter_turns(&self) -> i32 { + unsafe { ffi::xcw_native_session_rotation_quarter_turns(self.handle).rem_euclid(4) } + } + pub unsafe fn set_frame_callback( &self, callback: Option, diff --git a/server/src/native/ffi.rs b/server/src/native/ffi.rs index 8ca3b7a..e87d8fb 100644 --- a/server/src/native/ffi.rs +++ b/server/src/native/ffi.rs @@ -184,6 +184,7 @@ unsafe extern "C" { handle: *mut c_void, error_message: *mut *mut c_char, ) -> *mut c_char; + pub fn xcw_native_session_rotation_quarter_turns(handle: *mut c_void) -> i32; pub fn xcw_native_session_set_frame_callback( handle: *mut c_void, callback: Option, diff --git a/server/src/simulators/registry.rs b/server/src/simulators/registry.rs index 9572aa3..d3cd8ac 100644 --- a/server/src/simulators/registry.rs +++ b/server/src/simulators/registry.rs @@ -147,6 +147,7 @@ impl SessionRegistry { "displayWidth": 0, "displayHeight": 0, "frameSequence": 0, + "rotationQuarterTurns": 0, }) }); json!({ diff --git a/server/src/simulators/session.rs b/server/src/simulators/session.rs index d0dde6f..7ae4f31 100644 --- a/server/src/simulators/session.rs +++ b/server/src/simulators/session.rs @@ -258,6 +258,7 @@ impl SimulatorSession { "displayWidth": self.inner.display_width.load(Ordering::Relaxed), "displayHeight": self.inner.display_height.load(Ordering::Relaxed), "frameSequence": self.inner.frame_sequence.load(Ordering::Relaxed), + "rotationQuarterTurns": self.inner.native.rotation_quarter_turns(), "encoder": self.inner.native.video_encoder_stats(), }) } diff --git a/skills/simdeck/SKILL.md b/skills/simdeck/SKILL.md index 60f0934..f6791cc 100644 --- a/skills/simdeck/SKILL.md +++ b/skills/simdeck/SKILL.md @@ -1,14 +1,12 @@ --- name: simdeck -description: Agent guide for SimDeck, iOS Simulator control panel. Use for simulator lifecycle, app install/launch, live viewing, UI inspection, touch/keyboard automation, screenshots, logs, pasteboard, hardware controls, and repeatable simulator flows. +description: Use for simulator lifecycle, app install/launch, live viewing, UI inspection, touch/keyboard automation, screenshots, logs, pasteboard, hardware controls, and repeatable simulator flows. --- # SimDeck Agent Guide SimDeck automates iOS Simulators. Use the CLI for automation and the browser UI for live human visibility. Works with UIKit, SwiftUI, React Native, Expo, and NativeScript apps. -## Start And View - SimDeck uses one warm daemon per project. Check it with `simdeck daemon status`; start it or open the browser UI when needed: ```bash @@ -27,47 +25,9 @@ simdeck daemon start --video-codec software --low-latency simdeck ui --bind 0.0.0.0 --advertise-host 192.168.1.50 --open ``` -`simdeck` without a subcommand starts a foreground workspace daemon, prints local and LAN HTTP URLs, prints a six-digit pairing code for LAN browsers, and stops on `q` or Ctrl-C. The optional single argument is a simulator name or UDID to select by default. Use `-d` for detached start, `-k` to kill the background daemon, and `-r` to restart it. - -Viewer: `http://127.0.0.1:4310` or `http://127.0.0.1:4310?device=`. -The browser uses WebRTC H.264 video for both hardware and software encoders. -Local browser streams default to realtime WebRTC delivery with the `quality` -profile on VideoToolbox H.264: full resolution, 120 fps, and a high bitrate floor. -Add `--low-latency` on less capable runners to cap software H.264 at 15 fps, -drop stale pending frames more aggressively, and cap the longest edge at 1170 px -before latency piles up. -For local high-refresh testing, pass `--local-stream-fps <15-240>` on `ui`, -`daemon start`, or `daemon restart`. The default stays 60 fps; higher values are -for local high-refresh displays. -For remote browsers where Safari stalls but Chrome works, run the daemon with a -TURN server and relay-only ICE: -`SIMDECK_WEBRTC_ICE_SERVERS=turns:turn.example.com:5349?transport=tcp`, -`SIMDECK_WEBRTC_ICE_USERNAME`, `SIMDECK_WEBRTC_ICE_CREDENTIAL`, and -`SIMDECK_WEBRTC_ICE_TRANSPORT_POLICY=relay`. -SimDeck Studio provider runners keep SimDeck bound to loopback and run -`scripts/studio-provider-bridge.mjs` as an outbound bridge; Studio hosts the UI -and proxies REST requests through that bridge while WebRTC media negotiates -directly with the runner. -For an ad-hoc local provider that can be opened from another browser or phone, -run `simdeck studio expose "iPhone 17 Pro"` and keep that process running. It -prints the unique Studio simulator URL and active stream settings. This defaults -to software H.264 with realtime stream settings so remote viewers drop stale -frames instead of building latency. Studio providers default to the `smooth` -stream quality profile (1170 px, dynamic up to 60 fps, higher bitrate to reduce -artifacts); override with -`--stream-quality quality|balanced|fast|smooth|economy|ci-software`, or pass -`--video-codec hardware` when a dedicated hardware encoder is preferable. The -remote Studio viewer exposes 15, 30, and 60 fps choices in the stream menu. - -The local viewer gets the API token automatically. LAN browsers pair with the printed code before receiving the API cookie. Direct HTTP calls need `X-SimDeck-Token` or `Authorization: Bearer `. - -For fastest agent loops against a known daemon, export: - -```bash -export SIMDECK_SERVER_URL=http://127.0.0.1:4310 -``` +`simdeck` alone starts a foreground workspace daemon, prints URLs. The optional single argument is a simulator name or UDID to select by default. Use `-d` for detached start, `-k` to kill the background daemon, and `-r` to restart it. -Hot controls then delegate through the selected daemon instead of cold-starting native control each time. This is supported for launch/open-url, normalized touch/tap/swipe/gesture, key/key-sequence/key-combo, hardware buttons, dismiss-keyboard, home/app-switcher, rotate, and appearance toggles. Use direct commands when you need screen-coordinate selector resolution, install/uninstall, screenshots, pasteboard, or batch. +Viewer: usually `http://127.0.0.1:4310` or `http://127.0.0.1:4310?device=`. ## Device And App @@ -87,7 +47,7 @@ simdeck open-url https://example.com simdeck toggle-appearance ``` -Build apps with project tooling. SimDeck controls the simulator. +Build apps with project tooling. ## Fast Agent Inspection @@ -175,7 +135,7 @@ simdeck pasteboard get Use `--stdin` or `--file` for text with quotes, newlines, shell variables, or shell-sensitive characters. -## Timing And Batch +## Timing, Batch Input dispatch success does not prove the app reacted. Prefer selector waits/asserts, then use screenshot/logs/viewer when visual evidence matters. @@ -216,21 +176,56 @@ await simdeck.batch(udid, [ ```bash simdeck screenshot --output screen.png simdeck screenshot --stdout > screen.png -simdeck stream --frames 120 > stream.h264 simdeck logs --seconds 30 --limit 200 simdeck chrome-profile ``` -Use screenshots for still evidence. Use `stream` when a diagnostic needs raw -H.264 samples for an external player or capture pipeline. +Use screenshots for still evidence. Prefer describe for token-efficient state dumps, if they have enough context. ## Default Loop -1. Serve, list, boot/select ``, optionally open viewer. +1. Serve, list, boot/select ``, optionally open viewer if in-app browser available 2. Build with project tools; install and launch with SimDeck. 3. Use one `describe --format agent --max-depth 4` to understand an unfamiliar screen. 4. Interact with selectors first; use coordinates only when needed. 5. Verify with `waitFor`/`assert`/`query`, not repeated full `describe` dumps. 6. Batch known flows; keep `describe` as a failure/debug artifact. -Final check: UDID explicit, daemon URL set for fast loops when targeting a specific daemon, selectors/coordinates inspected, timing intentional, complex text uses `--stdin`/`--file`, results verified, CLI/API/inspector changes reflected here and in docs. +### Optional inspector plugins + +For a richer hierarchy, if user wants to opt-in + +### NativeScript Inspector + +NativeScript apps can connect directly to the running server from JS and expose +their view hierarchy plus raw UIKit backing views + +```ts +import { startSimDeckInspector } from "@nativescript/simdeck-inspector"; + +if (__DEV__) { + startSimDeckInspector({ port: 4310 }); +} +``` + +The runtime connects to `GET /api/inspector/connect` as a WebSocket + +### React Native Inspector + +React Native apps can expose their component tree and Metro dev-mode source +locations with the inspector package: + +```ts +import { AppRegistry } from "react-native"; +import { startSimDeckReactNativeInspector } from "react-native-simdeck"; +import App from "./App"; + +if (__DEV__) { + startSimDeckReactNativeInspector({ port: 4310 }); +} + +AppRegistry.registerComponent("Example", () => App); +``` + +Call it before `AppRegistry.registerComponent(...)` so the package can capture +React Fiber commits.