egfx: extend fuzz coverage to OpenH264-wrapper, ZGFX, multi-frame state, and encoder round-trip

[glamberson]

Part of #1120. This issue is an egfx-scoped child of the main fuzzing umbrella, focusing on the codec-pipeline depth specific to MS-RDPEGFX. It sits alongside #1122 (Arbitrary impls) and #1124 (advanced fuzz targets) under the #1120 umbrella, not as a parallel framing.

Why a separate child issue rather than comments on #1120

#1120 is broad and workspace-wide. The egfx-specific fuzz coverage gaps below are six concrete targets across four oracle classes, scoped to one crate's codec pipeline. That is enough scope to warrant its own tracking surface as a child issue rather than getting buried in #1120's comment thread (where our 2026-05-18 issuecomment-4483012331 on the ZGFX + memory-budget question is currently sitting).

Background

PR #1271 (merged 2026-05-25) extended the pdu_decode oracle in crates/ironrdp-fuzzing/src/oracles/mod.rs to include egfx wire-PDU types: GfxPdu, CacheToSurfacePdu, EgfxRawCapabilitySet, Avc420BitmapStream<'_>, Avc444BitmapStream<'_>, QuantQuality, Point, Color, CapabilitiesAdvertisePdu. Two real bugs surfaced and were fixed in-PR (Avc444BitmapStream::decode split_at panic on malformed streamLen; Avc420BitmapStream::decode Vec::with_capacity(num_regions) ASan OOM on attacker-controlled u32).

PR #1291 (merged 2026-05-25) added pdu_round_trip for ironrdp-pdu types, but did not extend into ironrdp-egfx's own encoders.

The result is that egfx is fuzzed at the wire-PDU-decode layer but the pipeline below that is not. The unfuzzed areas are below.

Coverage gaps

Surface	Wire-PDU decode	Round-trip	State-machine	Memory-budget
GfxPdu top-level	covered (#1271)	gap	gap	gap
Avc420BitmapStream wire	covered (#1271, 1 bug fixed)	gap	gap	gap
Avc444BitmapStream wire	covered (#1271, 1 bug fixed)	gap	gap	gap
Wire-decoded AVC bytes → OpenH264 input wrapper	gap	n/a	gap	gap
Annex B vs length-prefixed AVC handling	gap	gap	gap	n/a
Multi-frame H.264 sequence	gap	gap	gap	gap
ZGFX compression/decompression	gap	gap	n/a	gap
egfx encoders (encode_avc420_bitmap_stream etc.)	n/a	gap	n/a	n/a
Surface lifecycle state machine	n/a	n/a	gap	n/a
Cache state	n/a	n/a	gap	gap
EgfxRawCapabilitySet	covered (#1271)	gap	n/a	n/a

Proposed fuzz targets

Filing per-target PRs will follow as the work lands; this issue tracks the set.

1. egfx_avc420_decode — fuzz the wrapper layer that takes a decoded Avc420BitmapStream's bitmap_data and feeds it to H264Decoder::decode(&[u8]). OpenH264 itself is fuzzed by OSS-Fuzz; the IronRDP wrapper is not. Target: the input-construction code including any Annex B vs AVC-length-prefix conversion and any frame-header munging. Catches: panics in the wrapper, OOM via attacker-controlled chunking, contract violations in H264Decoder trait callers.

2. egfx_avc444_decode — same shape, MS-RDPEGFX 2.2.4.4 luma/chroma split. AVC444 has the auxiliary chroma stream and a tagged dispatch that AVC420 does not; merits a separate target.

3. egfx_zgfx_decompress — ZGFX is distinct from ironrdp-bulk's MPPC/NCRUSH/XCRUSH; lives in crates/ironrdp-graphics/src/zgfx/. No fuzz target today. Raised in our 2026-05-18 issuecomment-4483012331 on Advanced fuzzing with arbitrary #1120 as still-TODO. Decompression bombs and corrupted-history paths are the high-value catches.

4. egfx_round_trip — encoder/decoder symmetry for ironrdp-egfx's own encoders (encode_avc420_bitmap_stream, Avc420Region::encode, the new server-encode paths from PR feat(graphics,egfx): add progressive RFX server encode and mixed-codec frames #1198). Same shape as PR test(fuzz): add pdu_round_trip oracle and target #1291's pdu_round_trip but scoped to egfx encoders. Maps to Add advanced fuzz targets to verify more invariants (round-trip, framing, state-machine…) #1124's task 1 (round_trip) extended into egfx. Catches the same class of bug PR fix(pdu): cover BitmapCacheV3 in CapabilitySet encoder #1313 fixed in ironrdp-pdu's CapabilitySet encoder (decoder-accepted variant missing from encoder match).

5. egfx_multi_frame — the workspace currently has no multi-frame oracle pattern (per configs/ironrdp.md codec_pr_requirements). H.264 decoding maintains reference-picture state, SPS/PPS context, and decoder configuration across frames. Surface caching, frame acks, and codec state in egfx all carry forward. Single-shot fuzzers cannot find frame-to-frame state corruption. First codec to require it lands the pattern.

6. egfx_surface_state — state-machine fuzzing for the surface lifecycle (CreateSurface / ResetGraphics / WireToSurface1 / MapSurfaceToWindowPdu / DeleteSurface orderings + cache state transitions). Maps to Add advanced fuzz targets to verify more invariants (round-trip, framing, state-machine…) #1124's task 3 (connector_state_machine) narrowed to egfx surface state.

Memory-budget oracle (cross-cutting)

The codec memory-budget oracle remains the shape question raised in our 2026-05-18 issuecomment-4483012331 on #1120. Once that shape is settled, targets 3 (ZGFX) and the H.264 wrapper targets (1, 2) gain memory-budget oracles. Landing the targets above does not strictly require the memory-budget oracle answer (the targets work as panic/sanitizer-only oracles in the interim).

Sequencing

1. egfx_round_trip (target 4) is the smallest and most closed-form (mirrors PR test(fuzz): add pdu_round_trip oracle and target #1291's shape). Best to land first.
2. egfx_avc420_decode + egfx_avc444_decode (targets 1, 2) target the highest-value attack surface (H.264 wrapper) but require careful design around OpenH264's stateful decoder. Second pass.
3. egfx_zgfx_decompress (target 3) is closed-form in shape (similar to bulk_* targets) but has the open memory-budget oracle question.
4. egfx_multi_frame (target 5) establishes the workspace pattern for multi-PDU state-carrying oracles. Largest design work; expected via a dedicated PR with a separate design doc.
5. egfx_surface_state (target 6) maps to Add advanced fuzz targets to verify more invariants (round-trip, framing, state-machine…) #1124 task 3 and may be better tracked there once that task is factored.

Lamco-shaped commitment

We have downstream consumer interest in egfx via lamco-rdp-server and lamco-rdp-wasm. We will drive the per-target PRs ourselves, scoped per the sequencing above. Filing as a child issue under #1120 so the workspace has a public tracking surface and per-target PRs can reference it back.

Provenance

• Parent: Advanced fuzzing with arbitrary #1120 (main fuzzing umbrella)
• Sibling: Implement Arbitrary for more PDU structs #1122 (Arbitrary impls), Add advanced fuzz targets to verify more invariants (round-trip, framing, state-machine…) #1124 (advanced fuzz targets), CI: compile/test a feature matrix including arbitrary (std/no_std) #1123 (CI feature matrix, impl complete via PR ci: add check features matrix gate with dynamic CI fan-out #1283)
• Reference shapes:
  • PR feat(fuzz): add egfx PDU decoders to pdu_decode oracle + fix two surfaced bugs #1271 (egfx PDU decoder fuzz coverage; closed-bug record + reference shape for target 4's encoder counterpart)
  • PR test(fuzz): add pdu_round_trip oracle and target #1291 (pdu_round_trip oracle pattern, the reference shape for target 4 extended to egfx)
  • PR feat(fuzz): add bulk decompression fuzz targets (MPPC, NCRUSH, XCRUSH) #1285 (bulk_* fuzz targets, the reference shape for target 3)
  • PR fix(pdu): cover BitmapCacheV3 in CapabilitySet encoder #1313 (CapabilitySet encoder fix; reference for the bug class target 4 catches)
• Open question on parent: our 2026-05-18 issuecomment-4483012331 on Advanced fuzzing with arbitrary #1120 (ZGFX + memory-budget shape, pending response)

[glamberson]

Target 4 (round_trip) filed as #1317. Smallest and most closed-form of the six targets per the sequencing notes. Smoke fuzz at PR-filing time: 14.9M iterations across 121 seconds at ~125K exec/s sustained, zero crashes. egfx encoder surface for the five covered types is clean against this oracle.

[glamberson]

Design and placement notes for the two remaining targets in this umbrella, filed now so the harness-pattern conversations can run in parallel with the other targets landing.

Target 5 (egfx_multi_frame)

The workspace currently has no multi-frame oracle pattern. H.264 decoding maintains reference-picture state, SPS/PPS context, and decoder configuration across frames; surface caching and codec state in egfx carry forward. Single-shot fuzzers cannot find frame-to-frame state corruption.

Proposed harness shape: an Arbitrary-derived Vec<EgfxPduShape> driving a single decoder + surface-cache pair across the iteration. Each PDU shape is itself Arbitrary over the existing per-PDU generators. The harness:

1. Constructs one OpenH264Decoder and one surface-cache state at iteration start.
2. Iterates the PDU sequence, feeding each to the appropriate decode entry point in order.
3. Lets all errors and panics propagate to libFuzzer.

Trade-offs vs alternatives:

• Length-prefix framing on raw fuzz bytes: simpler harness; loses the structure the Arbitrary impls already provide; corpus mutation is less directed.
• Encoder-reflected well-formed sequences: highest corpus quality; requires reverse-driving the encoders (slow); doesn't exercise the failure paths the oracle is supposed to catch.

Arbitrary-derived Vec<Shape> is the middle ground.

Open shape question: the EgfxPduShape enum can live in ironrdp-fuzzing (oracle-private) or in ironrdp-egfx itself behind a #[cfg(feature = "arbitrary")] gate. The latter lets other oracles compose against the same shape; the former keeps egfx's public surface clean. Lamco's preference is the gated variant in ironrdp-egfx, with the oracle-private path as the fallback if the feature-gated surface raises concerns.

Target 6 (egfx_surface_state)

The original framing mapped target 6 to #1124's task 3 (connector_state_machine) narrowed to egfx surface state. #1325 has since landed as the acceptor-side sibling of #1124 for the broader state-machine fuzz pattern.

Placement: target 6 stays under this umbrella. The egfx surface lifecycle (CreateSurface / ResetGraphics / WireToSurface1 / MapSurfaceToWindowPdu / DeleteSurface orderings) is meaningfully different from the connector state machine in shape, state set, and receive-orientation. Folding into #1124 would force generic-vs-specific compromises. Multi-frame (target 5) and surface-state are conceptually adjacent (both carry codec/surface state across PDUs); the same umbrella preserves the shared-state framing. #1124 retains the broader connector state-machine scope; cross-referencing from there is sufficient.

Sequencing

Target 5 (multi-frame) lands first; target 6 follows since the multi-frame harness pattern informs the surface-state harness pattern. Both follow the AVC wrapper targets (#1326 / #1327) in maintainer-review priority. Lamco drives both implementations under this issue.