Remote Interaction Performance and Floating Menu

Remote Interaction Performance and Floating Menu

中文 · Runtime Main Path

This page maintains current invariants for multi-touch, control backpressure, performance metrics, Surface lifecycle, and the floating menu. Pixel measurements from one device are test evidence, not cross-device rules.

Input and control queue

MotionEvent -> RemoteTouchEventDispatcher -> ControlViewModel
            -> ScrcpyControllerTransport -> control socket

• The dispatcher assigns stable scrcpy pointer ids to Android pointers and maintains DOWN → MOVE → UP/CANCEL.
• First pointer down requests unbuffered dispatch. MOVE uses only the latest coordinates from the current event and does not replay historical samples.
• Surface or size changes and component disposal release pointers that remain down, preventing stuck remote touches.
• Active pointers use pressure 1; UP uses 0, independent of vendor touch-pressure reporting.

ScrcpyControllerTransport has one dedicated sender loop that serializes control-socket writes. Each pointer retains only its latest unsent MOVE. Ordered messages such as DOWN, UP, key, text, and clipboard flush all pending MOVE values before entering the ordered queue. Stale positions may be dropped, but the last position cannot cross a state barrier.

Game mode adds a global 5ms minimum MOVE interval and ensures the target observes at least 20ms between DOWN and UP; DOWN is never delayed. The control sender and video decoder use dedicated threads at display scheduling priority. None of these optimizations changes the video → optional audio → control socket connection order.

The control socket uses blocking reads without a timeout, with sending and device-message reception in separate execution domains. Idle keepalive and non-droppable messages such as clipboard must share the same ordering model as MOVE coalescing.

While a remote session is connected, Activity hardware volume-key events are also forwarded to the target. Odd-numbered presses are consumed locally; every second press is forwarded to the target and also passed through so the controller device can still change its own volume. The handler is removed when the screen is disposed or no longer connected. Compatibility mode routes the same keys through ADB input instead of the control socket.

Video and Surface

• Game-mode decoder output polling has a short wait window but does not intentionally drop decoded output.
• Decode and render counters separately track MediaCodec output and Surface submission.
• Repeated switches to the same Surface object are deduplicated so Compose recomposition does not repeat setOutputSurface().
• Both SurfaceView and TextureView use the same click-gesture tracker. A handled, single-pointer gesture that remains within Android touch slop calls performClick() for accessibility; movement, cancellation, and multi-pointer gestures remain remote touch input and do not become clicks.
• Controller/remote orientation and aspect-ratio decisions use the actual Compose window container size, including multi-window changes, instead of configuration screenWidthDp/screenHeightDp.
• Rotation policy is explicit: Local sets SCREEN_ORIENTATION_UNSPECIFIED; Target follows the remote frame with portrait or sensor-landscape restrictions.
• Surface recreation is a rendering lifecycle concern and does not automatically trigger ADB or full scrcpy reconnection. Real decoder failures enter recovery through SessionEvent.

Performance overlay

The overlay appears when the global setting is enabled and a video stream exists; reconnection is not required. It is anchored at the controller fullscreen container's top-left and does not follow remote aspect ratio, letterboxing, or rotation. It does not consume touch.

Metric	Source	Limitation
Decode FPS	MediaCodec output count	Not the remote application's FPS
Render FPS	Outputs submitted to Surface	Not final SurfaceFlinger presentation
Video	scrcpy video payload bytes	Excludes other application traffic
Network TX/RX	Current UID TrafficStats	May include other traffic under the UID
Wi-Fi Link TX/RX	Negotiated WifiInfo rate	PHY indicator, not actual throughput; network sessions only

Deltas are sampled about once per second with a monotonic clock. If Decode and Render fall together, inspect encoding load, network, size, and the local decoder. If Decode is stable but Render is lower, inspect Surface, display composition, and foreground state. Tune from lower size and bitrate, change one variable at a time, and record these metrics instead of treating one bitrate table as universal.

Game-mode background behavior

After a game session connects, file/in-memory logging and background mDNS discovery pause to reduce unrelated competition. ADB, scrcpy sockets, health checks, and foreground protection remain active. The floating ball remains controlled by the session's Show Floating Ball switch, while the FPS/network-rate overlay remains controlled by the global performance-stats switch; both may be displayed in game mode and are not forcibly disabled. The layout inspector also remains independent. Connection loss restores background modules and follows normal reconnection policy.

Floating-menu coordinates

• Ball and menu use controller Activity-window coordinates, never remote video dimensions.
• FloatingMenuWindowBoundsProvider derives safe bounds from orientation, navigation bars, and display cutouts.
• At an edge, the window remains inside valid bounds while a Canvas drawing offset hides 3/5 and leaves 2/5 visible. Negative window coordinates are not required.
• The first tap while hidden reveals the ball; the second opens the menu. Dragging inward beyond the threshold also reveals it.
• Balls A and B keep aligned centers. Independent directional movement of A is disabled while the menu is visible.

The menu normally sits above the ball with a 16dp gap. When room is insufficient, the ball first moves beyond boundsTop + menuHeight + gap; menu height changes trigger recalculation. Portrait may use the application-window top while landscape uses the safe bounds top. Acceptance uses dynamic insets, actual view dimensions, and the spacing formula rather than fixed px values from one device.

Regression anchors

• TouchTransportTimingTest: MOVE throttling and minimum hold duration.
• RemoteScreenOrientationTest: Local/Target activity-orientation policy.
• FloatingMenuEdgeSnapTest: four edges, safe insets, 2/5 visibility, menu gap, and orientation differences.
• Device multi-touch: two or more fingers retain identity and the last MOVE before UP.
• Accessibility: a stationary handled tap invokes performClick() once on both video view types, while drag and multi-touch do not.
• Rotation matrix: controller and remote portrait/landscape combinations keep the ball and overlay in controller coordinates.
• Performance setting: off by default, enable without reconnect, disappear immediately when disabled.