Encoding and Decoding

Encoding and Decoding

中文 · Runtime Main Path

This is the canonical entry for the codec catalog, capability model, automatic selection, metadata, and video/audio runtime. See Session Configuration and Connection Lifecycle for ownership of user preferences, device capability cache, and current-run fallback.

Format catalog

CodecCatalog is the single mapping from format names to MIME types:

Media	Format	MIME
Video	H.264	video/avc
Video	H.265	video/hevc
Video	AV1	video/av01
Video	VP9	video/x-vnd.on2.vp9
Video	VP8	video/x-vnd.on2.vp8
Audio	Opus	audio/opus
Audio	AAC	audio/mp4a-latm
Audio	FLAC	audio/flac
Audio	RAW	audio/raw

Automatic selection must intersect structured EncoderCapability.codec/mimeType with DecoderCapability.mimeTypes. Implementation-name inference is allowed only as a last fallback for a user-entered implementation without detection data. c2.* and OMX.* identify implementation families or vendors, not formats.

Selection and cache

• ScrcpyConfig.user*Encoder/Decoder represents user selection. DeviceCapabilityCache stores remote capability and automatic results.
• ScrcpyOptions.getFinal*() only implements “non-empty user value, otherwise cache”; it does not prove the combination can run.
• CodecSelector selects by MIME intersection. Fixed implementations are tried strictly first. If they cannot form a compatible pair, the reason is logged, automatic selection runs, and ignoredUserSelection is set.
• Automatic encoders prefer non-alias hardware, then hybrid, unknown, and software. Video decoders have the same acceleration preference plus a bonus for low-latency capability on the current MIME.
• Disabling hardware decoding truly limits candidates to software decoders.
• A changed device serial invalidates the complete capability cache. Do not infer one physical-device identity across USB, TCP, and mDNS transports.

Remote capability detection runs after the server asset is available but must finish before the final launch command reads configuration. Results are written only to the session that initiated detection; switching or cancellation must not allow an old task to update another session.

Metadata and protocol boundary

All required sockets first connect sequentially as video → optional audio → control. The dummy byte is validated only on video, after which metadata is read.

The current video header is a 64-byte device name, 4-byte codec id, 4-byte session flags, 4-byte width, and 4-byte height. Fixed fields use an exact-read loop; one read() call is never assumed complete. Codec ids support H.264, H.265, AV1, VP8, and VP9. Size and session flags are validated before stream creation.

The audio header is a 4-byte codec id. 0 means the server explicitly disabled audio; configuration error and unknown ids fail. A stream is created only for valid Opus, AAC, FLAC, or RAW. Metadata failure closes created streams and all sockets instead of passing misaligned bytes to a decoder.

Video runtime

• Each scrcpy codec-config packet is parsed independently. H.264 and H.265 extract every Annex-B NAL in that packet and require the needed SPS/PPS or VPS/SPS/PPS; VP8/VP9 support no-CSD startup. The client no longer aggregates unrelated packets in a fixed NAL buffer.
• scrcpy PTS values go directly to MediaCodec in microseconds.
• Decoder candidates must pass creation, size preflight, configure, and start. Failed candidates are rejected only for the current session.
• Runtime decoder fallback retains the current configuration and the complete GOP beginning at the latest key frame. After the replacement decoder is configured, packets are replayed with their original PTS before live input resumes, avoiding an unconditional wait for the next remote key frame. Packet and bootstrap-cache ceilings now scale with the app heap: each is at least 4 MiB, the packet ceiling is at most 32 MiB, and the bootstrap cache is at most 16 MiB. On a 48 MiB heap both are 6 MiB. An oversized or incomplete GOP is discarded as a unit.
• Configuration packets and key frames are critical input. If MediaCodec has no input buffer on the first attempt, existing output is drained before one longer retry. Ordinary frames retain the short timeout so input backpressure does not grow remote-control latency.
• On Android 11+, a decoder declaring support may receive KEY_LOW_LATENCY; this bonus never replaces format, size, and real startup validation.
• If a fixed decoder rejects the negotiated size, or scrcpy reports a capture/encoding size failure, automatic reconnect pauses and the UI asks before applying a temporary maxSize. Approval follows visible tiers 1920 → 1080 → 720, allows at most three recovery attempts, and never persists the override.
• Repeated switches to the same Surface are deduplicated. A Surface failure is not automatically a connection failure.
• GOP replay applies only to a real MediaCodec rebuild within the same video stream. Normal SurfaceView, full-screen TextureView, and dummy-Surface transitions do not trigger it, and the path does not add a persistent texture or EGL forwarding layer.

Audio runtime

• The selected decoder name reaches createByCodecName(). In non-fixed mode, creation, configure, start, or runtime failure may reject the candidate and try another.
• Opus parses OpusHead, pre-skip, codec delay, and seek preroll; AAC parses AudioSpecificConfig; FLAC parses STREAMINFO.
• RAW plays PCM directly without a compressed decoder.
• AudioTrack writes must handle partial writes without discarding remaining PCM. Audio buffering and errors are diagnosed separately from video.

Troubleshooting order

1. Compare server launch parameters with the actual codecs announced by video and audio socket headers.
2. Confirm that remote encoder and local decoder MIME types match instead of inferring from names.
3. Separate metadata, packet, decoder configure/start, and Surface/output stages.
4. Identify whether the run used user selection, cached selection, or fallback, and verify session id and device serial.
5. Treat server-disabled audio (codec id 0) separately from audio configuration failure.

Regression anchors

• CodecCatalogTest, CodecSelectorTest: catalog, intersection, and ranking.
• CodecRemoteDetectionTest, ServerCodecResolutionTest: remote detection and launch parameters.
• VideoCodecHeaderTest, ScrcpyStreamProtocolTest: header and packet boundaries.
• VideoDecoderSizePolicyTest, VideoCaptureRecoveryPolicyTest, VideoPacketCodecModeTest, VideoPresentationTimestampTest, VideoDecoderBootstrapCacheTest, VideoMemoryPolicyTest: video runtime, recovery, PTS, and memory/cache boundaries.
• AudioDecoderBootstrapperTest, AudioStreamHeaderTest, ScrcpyAudioPacketTest: audio runtime.
• SessionCodecFallbackTest: current-session fallback and persistence boundary.

The device matrix should cover H.264/H.265 across SoC families and AV1/VP8/VP9 where supported. Audio should cover Opus/AAC/FLAC/RAW, rotation, Surface recreation, disconnection, and long playback. Defaults never replace measured first-frame time, sustained FPS, end-to-end latency, power, and temperature.