Green/purple/blue/etc. artifacts when hardware transcoding on AMD Radeon RX 5x00 series GPUs and Ryzen 4000 series APUs

[hifihedgehog]

Test environment

PMS Version: 1.21.0.3616
PMP Version: Plex Web Version 4.43.4 and Plex iOS 7.10.1
Platform: Windows 10 64-Bit for Server; Windows and iOS for clients

Steps to reproduce

1. Use an AMD Ryzen 4000 series APU with integrated graphics or a discrete AMD Radeon RX 5x00 series GPU. In this example, I am using the Ryzen 7 PRO 4750G. (Update: Based on forum comments, this seems to be common only to the Radeon RX 5x00 GPU and Ryzen 4000 series APUs, which share the same updated features in their Video Core Next (VCN) 2.0 codec block.)
2. On your Plex server, go to Settings -> Transcoder and ensure both "Use hardware acceleration when available" and "Use hardware-accelerated video encoding" are checked.
3. Attempt to play a video locally or on a separate device (platform independent for the client) at any at-playback transcoded bitrate, or attempt to sync a pre-transcoded optimized copy of the video. In my case, this behavior occurs both on my Plex server and on my iPhone 12 Pro Max.

Current behavior

1. PMP green/purple/blue artifacts on the video content. See these examples below:

Expected behavior

1. Clean video free of random color artifacts.

Related forum thread where many other users have reproduced and report this issue:

https://forums.plex.tv/t/pmp-green-purple-artifacts-when-playing-x265-10bit-mkv/570323

[mseeley]

Logs and sample media clips that repro the problem are necessary please.

Tangential, what's keeping you using PMP on Windows instead of Plex for Windows? The latter has a newer mpv integrated.

[hifihedgehog]

Tangential, what's keeping you using PMP on Windows instead of Plex for Windows? The latter has a newer mpv integrated.

Thanks for the response. This is an encoding issue so it occurs on any Plex PMP (iOS, and now also macOS) and is independent of the player platform. The encoding pipeline appears to be misconfigured such that AMD's VCE/VCN produces these green/purple/green pixel clouds. This issue does not occur in other programs such as Handbrake with AMD VCE/VCN hardware encoding enabled.

[hifihedgehog]

Logs and sample media clips that repro the problem are necessary please.

Will update this post with video segments with hardware encoding enabled and disabled

[hifihedgehog]

See example case below.

> > https://gofile.io/d/KbUSp0 < <

This example above is less severe since, as a CGI animated short, it has more static or stationary frames, but it is still plainly obvious where the improper encoding occurs. This does not occur with other hardware encoding implementations I use, such as Handbrake which features VCE/VCN support. I included a log and database dump as requested. Some very obvious problem points I quickly noted in the hardware encode are ~2:14, ~2:18, and ~3:11, but there are many more problem points before and after as you may have already noticed.

[hifihedgehog]

Related forum thread where many other users have reproduced and reported this issue:

https://forums.plex.tv/t/pmp-green-purple-artifacts-when-playing-x265-10bit-mkv/570323

Pay particular note to this:

I’m having the same issue with a 5700xt. It’s when I’m hardware transcoding h265 10bit, and only about 30-40% of my collection in that format. It doesn’t matter which client is used for playback, web player, iOS, Android tv, Apple TV. If it has to transcode I will get this.

https://forums.plex.tv/t/pmp-green-purple-artifacts-when-playing-x265-10bit-mkv/570323/19

I would note, however, that in my particular case, I am using the pure MKV remuxed h.264 source bitstream of Jack Jack Attack from the original Blu-ray disc of Disney-Pixar The Incredibles. I see this artifact issue virtually with my entire Blu-ray and UHD Blu-ray collection when transcoding live and hot to any device or pre-transcoding to a flat file using the optimization option.

[hifihedgehog]

Update: Based on forum comments, this seems to be common only to the Radeon RX 5x00 series GPUs and Ryzen 4000 series APUs, which share the same updated features in their Video Core Next (VCN) codec block. See also my forum comment:

I can only guess, but the Ryzen 4000 series APUs use the same VCN 2.0 core as the Radeon RX 5x00 series GPUs. Both GPUs (integrated and discrete) share this same common architectural piece which may be the common failure point. I would bet anything that Plex does not have a fully up-to-date and compliant VCN/VCE codebase meaning no proper support for these GPUs’ codec block.

[hifihedgehog]

Pinging AMD GitHub members @gregrodgers and @kiritigowda who hopefully can tap on the right folks if this requires driver-level rectification or advanced support.

[hifihedgehog]

Can also now confirm this issue does not occur on Jellyfin though Jellyfin is lacking in a great deal many critical features when compared to Plex and Plex Pass.

[hifihedgehog]

For what it is worth, I just bought a lifetime Plex Pass this week, so I hope that raises awareness among Plex staff that paying customers like myself are looking for dependable support to patch this longstanding issue with Plex Server.

[hifihedgehog]

Downloaded the new release of Plex Media Server (1.21.0.3711), no fix yet in sight. Tried again today. Same multicolored pixel errata.

[hifihedgehog]

It's still giving me what I reported to Plex here on GitHub. The issue is server-side and has to do with their custom broken implementation of FFMPEG and VCN 2.0. Other software out there like Handbrake that use FFMPEG have no issues with AMD's VCN hardware encoding. This example I provided them is with a non-live, pre-transcoded or optimized file. They have yet to fix their issue with hardware encoding:

See example case below.

> > https://gofile.io/d/KbUSp0 < <

This example above is less severe since, as a CGI animated short, it has more static or stationary frames, but it is still plainly obvious where the improper encoding occurs. This does not occur with other hardware encoding implementations I use, such as Handbrake which features VCE/VCN support. I included a log and database dump as requested. Some very obvious problem points I quickly noted in the hardware encode are ~2:14, ~2:18, and ~3:11, but there are many more problem points before and after as you may have already noticed.