fix(server): emit RGB-channel QOI for opaque captures so ironrdp-session can decode

[clintcan]

Fixes the broken QOI codec path between ironrdp-server and ironrdp-client/ironrdp-session. Found while answering Marc-Andre Lureau (@elmarco)'s "did you try [QOI]?" question on #1322.

What's broken

In crates/ironrdp-server/src/encoder/mod.rs::qoi_encode, the four *A32 PixelFormat variants (ARgb32, ABgr32, BgrA32, RgbA32) map to the matching *a variant of qoi::RawChannels. In qoicoubeh, those variants produce a QOI header with channels = Rgba. The decoder in ironrdp-session/src/fast_path.rs::qoi_apply only supports Channels::Rgb; the Channels::Rgba arm is literally warn!("Unsupported RGBA QOI data") and drops the frame.

So the moment QOI actually gets negotiated between an ironrdp-server-based server and an ironrdp-client/ironrdp-session-based client, the server emits a header the client refuses, and the client renders a blank window plus one warning per frame.

Reproduction (before the fix)

Loopback against macrdp 0.5.49 (which uses ironrdp-server from Devolutions/IronRDP rev 879ffed8):

$ ironrdp-viewer --features qoi,qoiz ... 127.0.0.1:3390
WARN ironrdp_session::fast_path: Unsupported RGBA QOI data
WARN ironrdp_session::fast_path: Unsupported RGBA QOI data
... (412 warnings in ~12s; viewer renders nothing)

From the connector trace, the server advertised [NsCodec, RemoteFx, ImageRemoteFx, Qoi, QoiZ], the client confirmed [RemoteFx, Qoi, QoiZ], and the server picked QOI per its codec selection ladder. Every encoded frame was emitted as RGBA.

Fix

Map every 4-byte input format — whether it nominally has an alpha byte or an "X" filler — to the *x sibling of RawChannels. The qoi crate picks Channels::Rgb for *x/*r/*g/*b and Channels::Rgba only for *a, so this forces 3-channel output that the existing decoder accepts.

Server-side bitmap captures are functionally opaque (the alpha byte from a screen capture is either premultiplied to 0 or set to 0xFF), so discarding it at encode time is consistent with how the rest of the legacy bitmap path already handles these formats.

After the fix

Same loopback session:

• 0 Unsupported RGBA QOI data warnings
• viewer renders the desktop normally

Test plan

• cargo build -p ironrdp-server --features qoi — clean
• cargo build -p ironrdp-server (default) — clean
• cargo clippy -p ironrdp-server --features qoi,qoiz,helper,__bench --all-targets -- -D warnings — clean
• cargo fmt --check — clean
• Loopback reproduction with ironrdp-viewer --features qoi,qoiz against macrdp (a ironrdp-server consumer) — 412 → 0 RGBA warnings

Notes

• The four *X32 variants are unchanged; they already mapped to the *x RawChannels and produced correct RGB output.
• No client-side change. A future PR could also add a real Channels::Rgba decode path in qoi_apply (just ignore the alpha plane), which would let clients consume RGBA QOI from third-party encoders if any exist. Out of scope here.

[Marc-Andre Lureau (elmarco)]

thanks for the investigation and the fix!

If the server is correctly encoding RGBA bitmap, why not fix the client instead?

[clintcan]

Thanks for the review!

Fair point: both fixes are reasonable, and I think they're complementary rather than alternatives.

The server-side change here is partly a bandwidth optimization: every screen-capture *A32 frame has an alpha byte that's always 0xFF for opaque content, so encoding it costs one extra byte per QOI_OP_RGB/QOI_OP_RGBA literal chunk for zero information gain.

There's actually existing precedent in the same function; the four *X32 variants already map to the *x RawChannels for exactly this reason (the X filler byte is dropped). Extending the same treatment to *A32 keeps the function consistent.

Long story short, the server can actually emit either format.

But you're right that the client should also handle RGBA QOI. Today it can't decode legitimate RGBA frames (the Channels::Rgba arm is a warn! + drop), which is a real gap whenever a server has a reason to emit them (layered composition, premultiplied alpha preservation, etc.). I'd be happy to open a follow-up adding the RGBA decode path on the client side, it's a few lines either widening apply_rgb24 or stripping alpha before the existing call. They're independent PRs and can land in either order.

This is good for future divergence (or third-party RDP servers with real alpha channels).

If you'd prefer to close this in favor of just the client-side fix, I'm fine with that too — but I'd argue the server side is worth keeping as a bandwidth optimization regardless, since screen captures don't carry useful alpha — every consumer I know of feeds opaque content.

[clintcan]

Marc-Andre Lureau (@elmarco) — opened #1341 as the companion client-side fix. TL;DR: I think both changes have independent value (server is a bandwidth optimization on opaque captures; client is correctness against any server that legitimately emits Channels::Rgba), and #1341 carries the full reasoning. Happy to land them in either order, or to close this one if you'd rather only take the client-side path.

[Marc-Andre Lureau (elmarco)]

Marc-Andre Lureau (Marc-Andre Lureau (@elmarco)) — opened #1341 as the companion client-side fix. TL;DR: I think both changes have independent value (server is a bandwidth optimization on opaque captures; client is correctness against any server that legitimately emits Channels::Rgba), and #1341 carries the full reasoning. Happy to land them in either order, or to close this one if you'd rather only take the client-side path.

fair enough, I don't think we care enough about rgba support at this point. We can revisit later.

[Benoît Cortier (CBenoit)]

LGTM! If Marc-Andre Lureau (@elmarco) is good with the changes too, we can merge