Remote Interaction Performance and Floating Menu

Remote Interaction Performance and Floating Menu

中文 · Runtime Main Path

This page maintains current invariants for multi-touch, control backpressure, performance metrics, Surface lifecycle, and the floating menu. Pixel measurements from one device are test evidence, not cross-device rules.

Input and control queue

MotionEvent -> RemoteTouchEventDispatcher -> ControlViewModel
            -> ScrcpyControllerTransport -> control socket

• The dispatcher assigns stable scrcpy pointer ids to Android pointers and maintains DOWN → MOVE → UP/CANCEL.
• First pointer down requests unbuffered dispatch. MOVE uses only the latest coordinates from the current event and does not replay historical samples.
• Surface or size changes and component disposal release pointers that remain down, preventing stuck remote touches.
• Active pointers use pressure 1; UP uses 0, independent of vendor touch-pressure reporting.

ScrcpyControllerTransport has one dedicated sender loop that serializes control-socket writes. Each pointer retains only its latest unsent MOVE. Ordered messages such as DOWN, UP, key, text, and clipboard flush all pending MOVE values before entering the ordered queue. Stale positions may be dropped, but the last position cannot cross a state barrier.

Game mode adds a global 5ms minimum MOVE interval and ensures the target observes at least 20ms between DOWN and UP; DOWN is never delayed. The control sender and video decoder use dedicated threads at display scheduling priority. None of these optimizations changes the video → optional audio → control socket connection order.

The control socket uses blocking reads without a timeout, with sending and device-message reception in separate execution domains. Idle keepalive and non-droppable messages such as clipboard must share the same ordering model as MOVE coalescing.

Video and Surface

• Game-mode decoder output polling has a short wait window but does not intentionally drop decoded output.
• Decode and render counters separately track MediaCodec output and Surface submission.
• Repeated switches to the same Surface object are deduplicated so Compose recomposition does not repeat setOutputSurface().
• Surface recreation is a rendering lifecycle concern and does not automatically trigger ADB or full scrcpy reconnection. Real decoder failures enter recovery through SessionEvent.

Performance overlay

The overlay appears when the global setting is enabled and a video stream exists; reconnection is not required. It is anchored at the controller fullscreen container's top-left and does not follow remote aspect ratio, letterboxing, or rotation. It does not consume touch.

Metric	Source	Limitation
Decode FPS	MediaCodec output count	Not the remote application's FPS
Render FPS	Outputs submitted to Surface	Not final SurfaceFlinger presentation
Video	scrcpy video payload bytes	Excludes other application traffic
Network TX/RX	Current UID TrafficStats	May include other traffic under the UID
Wi-Fi Link TX/RX	Negotiated WifiInfo rate	PHY indicator, not actual throughput; network sessions only

Deltas are sampled about once per second with a monotonic clock. If Decode and Render fall together, inspect encoding load, network, size, and the local decoder. If Decode is stable but Render is lower, inspect Surface, display composition, and foreground state. Tune from lower size and bitrate, change one variable at a time, and record these metrics instead of treating one bitrate table as universal.

Game-mode background behavior

After a game session connects, file/in-memory logging and background mDNS discovery pause to reduce unrelated competition. ADB, scrcpy sockets, health checks, and foreground protection remain active. The floating ball remains controlled by the session's Show Floating Ball switch, while the FPS/network-rate overlay remains controlled by the global performance-stats switch; both may be displayed in game mode and are not forcibly disabled. The layout inspector also remains independent. Connection loss restores background modules and follows normal reconnection policy.

Floating-menu coordinates

• Ball and menu use controller Activity-window coordinates, never remote video dimensions.
• FloatingMenuWindowBoundsProvider derives safe bounds from orientation, navigation bars, and display cutouts.
• At an edge, the window remains inside valid bounds while a Canvas drawing offset hides 3/5 and leaves 2/5 visible. Negative window coordinates are not required.
• The first tap while hidden reveals the ball; the second opens the menu. Dragging inward beyond the threshold also reveals it.
• Balls A and B keep aligned centers. Independent directional movement of A is disabled while the menu is visible.

The menu normally sits above the ball with a 16dp gap. When room is insufficient, the ball first moves beyond boundsTop + menuHeight + gap; menu height changes trigger recalculation. Portrait may use the application-window top while landscape uses the safe bounds top. Acceptance uses dynamic insets, actual view dimensions, and the spacing formula rather than fixed px values from one device.

Regression anchors

• TouchTransportTimingTest: MOVE throttling and minimum hold duration.
• FloatingMenuEdgeSnapTest: four edges, safe insets, 2/5 visibility, menu gap, and orientation differences.
• Device multi-touch: two or more fingers retain identity and the last MOVE before UP.
• Rotation matrix: controller and remote portrait/landscape combinations keep the ball and overlay in controller coordinates.
• Performance setting: off by default, enable without reconnect, disappear immediately when disabled.