Encoding and Decoding

Encoding and Decoding

中文 · Runtime Main Path

This is the canonical entry for the codec catalog, capability model, automatic selection, metadata, and video/audio runtime. See Session Configuration and Connection Lifecycle for ownership of user preferences, device capability cache, and current-run fallback.

Format catalog

CodecCatalog is the single mapping from format names to MIME types:

Media	Format	MIME
Video	H.264	video/avc
Video	H.265	video/hevc
Video	AV1	video/av01
Video	VP9	video/x-vnd.on2.vp9
Video	VP8	video/x-vnd.on2.vp8
Audio	Opus	audio/opus
Audio	AAC	audio/mp4a-latm
Audio	FLAC	audio/flac
Audio	RAW	audio/raw

Automatic selection must intersect structured EncoderCapability.codec/mimeType with DecoderCapability.mimeTypes. Implementation-name inference is allowed only as a last fallback for a user-entered implementation without detection data. c2.* and OMX.* identify implementation families or vendors, not formats.

Selection and cache

• ScrcpyConfig.user*Encoder/Decoder represents user selection. DeviceCapabilityCache stores remote capability and automatic results.
• ScrcpyOptions.getFinal*() only implements “non-empty user value, otherwise cache”; it does not prove the combination can run.
• CodecSelector selects by MIME intersection. Fixed implementations are tried strictly first. If they cannot form a compatible pair, the reason is logged, automatic selection runs, and ignoredUserSelection is set.
• Automatic encoders prefer non-alias hardware, then hybrid, unknown, and software. Video decoders have the same acceleration preference plus a bonus for low-latency capability on the current MIME.
• Disabling hardware decoding truly limits candidates to software decoders.
• A changed device serial invalidates the complete capability cache. Do not infer one physical-device identity across USB, TCP, and mDNS transports.

Remote capability detection runs after the server asset is available but must finish before the final launch command reads configuration. Results are written only to the session that initiated detection; switching or cancellation must not allow an old task to update another session.

Metadata and protocol boundary

All required sockets first connect sequentially as video → optional audio → control. The dummy byte is validated only on video, after which metadata is read.

The current video header is a 64-byte device name, 4-byte codec id, 4-byte session flags, 4-byte width, and 4-byte height. Fixed fields use an exact-read loop; one read() call is never assumed complete. Codec ids support H.264, H.265, AV1, VP8, and VP9. Size and session flags are validated before stream creation.

The audio header is a 4-byte codec id. 0 means the server explicitly disabled audio; configuration error and unknown ids fail. A stream is created only for valid Opus, AAC, FLAC, or RAW. Metadata failure closes created streams and all sockets instead of passing misaligned bytes to a decoder.

Video runtime

• H.264 parses SPS/PPS; H.265 supports combined or split CSD; VP8/VP9 support no-CSD startup.
• scrcpy PTS values go directly to MediaCodec in microseconds.
• Decoder candidates must pass creation, size preflight, configure, and start. Failed candidates are rejected only for the current session.
• Runtime decoder fallback retains the current configuration and the complete GOP beginning at the latest key frame. After the replacement decoder is configured, packets are replayed with their original PTS before live input resumes, avoiding an unconditional wait for the next remote key frame. The cache has a 16 MiB total byte limit; an oversized or incomplete GOP is discarded as a unit and fallback waits for the next key frame instead of evicting its configuration or key-frame anchor and replaying an undecodable tail.
• Configuration packets and key frames are critical input. If MediaCodec has no input buffer on the first attempt, existing output is drained before one longer retry. Ordinary frames retain the short timeout so input backpressure does not grow remote-control latency.
• On Android 11+, a decoder declaring support may receive KEY_LOW_LATENCY; this bonus never replaces format, size, and real startup validation.
• If size is unsupported and a smaller recommendation exists, the current run may reconnect with a bounded temporary maxSize reduction. It is not persisted.
• Repeated switches to the same Surface are deduplicated. A Surface failure is not automatically a connection failure.
• GOP replay applies only to a real MediaCodec rebuild within the same video stream. Normal SurfaceView, full-screen TextureView, and dummy-Surface transitions do not trigger it, and the path does not add a persistent texture or EGL forwarding layer.

Audio runtime

• The selected decoder name reaches createByCodecName(). In non-fixed mode, creation, configure, start, or runtime failure may reject the candidate and try another.
• Opus parses OpusHead, pre-skip, codec delay, and seek preroll; AAC parses AudioSpecificConfig; FLAC parses STREAMINFO.
• RAW plays PCM directly without a compressed decoder.
• AudioTrack writes must handle partial writes without discarding remaining PCM. Audio buffering and errors are diagnosed separately from video.

Troubleshooting order

1. Compare server launch parameters with the actual codecs announced by video and audio socket headers.
2. Confirm that remote encoder and local decoder MIME types match instead of inferring from names.
3. Separate metadata, packet, decoder configure/start, and Surface/output stages.
4. Identify whether the run used user selection, cached selection, or fallback, and verify session id and device serial.
5. Treat server-disabled audio (codec id 0) separately from audio configuration failure.

Regression anchors

• CodecCatalogTest, CodecSelectorTest: catalog, intersection, and ranking.
• CodecRemoteDetectionTest, ServerCodecResolutionTest: remote detection and launch parameters.
• VideoCodecHeaderTest, ScrcpyStreamProtocolTest: header and packet boundaries.
• VideoDecoderSizePolicyTest, VideoPacketCodecModeTest, VideoPresentationTimestampTest, VideoDecoderBootstrapCacheTest: video runtime, PTS, and complete-GOP cache boundaries.
• AudioDecoderBootstrapperTest, AudioStreamHeaderTest, ScrcpyAudioPacketTest: audio runtime.
• SessionCodecFallbackTest: current-session fallback and persistence boundary.

The device matrix should cover H.264/H.265 across SoC families and AV1/VP8/VP9 where supported. Audio should cover Opus/AAC/FLAC/RAW, rotation, Surface recreation, disconnection, and long playback. Defaults never replace measured first-frame time, sustained FPS, end-to-end latency, power, and temperature.