Jellyfin 10.5.5 Docker VAAPI hardware transcoding issues

[Protektor-Desura]

Jellyfin docker is not able to handle VAAPI because with Ubuntu 20.04 and Ubuntu 18.02 (LTS) the /dev/dri/renderD128 is owned by group render and the docker has no clue about this group even if in the host OS you add user 1000 to group render.

crw-rw----+  1 root render 226, 128 Jul 16 22:24 renderD128 (doesn't work)
crw-rw----+  1 root video 226, 128 Jul 16 22:24 renderD128 (works)

If you change the group to video then it work sort of. I was able to get it to transcode anything that isn't HEVC with my Radeon RX 550 card. This card does specifically support HEVC encoding and decoding.
https://www.amd.com/en/products/graphics/radeon-rx-550

I get this error.

  Stream #0:0 -> #0:0 (hevc (native) -> h264 (h264_vaapi))
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
[Parsed_scale_vaapi_2 @ 0x563e3af4e980] Failed to render parameter buffer: 6 (invalid VASurfaceID).
Error while filtering: Input/output error
Failed to inject frame into filter network: Input/output error
Error while processing the decoded data for stream #0:0
Conversion failed!```

I'm not sure why hevc_vaapi isn't listed either. I would have thought that should have been called/used if VAAPI was turned on. Maybe that is for encoding only?
https://trac.ffmpeg.org/wiki/Hardware/VAAPI

[Artiume]

Stream #0:0 -> #0:0 (hevc (native) -> h264 (h264_vaapi))

For whatever reason, Vaapi decoding says native instead of hevc_vaapi. I would have to see the full transcoding log to check the ffmpeg command to verify vaapi operation... that said, since you're having that failure, you're probably using VAAPI. I made this comment here.

I'm not sure why the render and video groups give these permission issues, i've run into it several times and not sure how to fix it for docker. If you try linuxserver/jellyfin, it may work properly

[Protektor-Desura]

I should mention that AMD AMF doesn't work for me either with HEVC.

Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> h264 (h264_amf))
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
[h264_amf @ 0x5590871d0280] DLL libamfrt64.so.1 failed to open
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
Conversion failed!

[Protektor-Desura]

[2020-07-17 16:15:10.489 -05:00] [INF] [17] MediaBrowser.MediaEncoding.Subtitles.SubtitleEncoder: "/usr/lib/jellyfin-ffmpeg/ffmpeg" "-i file:\"/data/movies/WALL·E (2008)/WALL·E (2008) [Bluray-1080p x265 AAC].mkv\" -map 0:2 -an -vn -c:s copy \"/config/data/data/subtitles/b/b76d6996-e870-4bbf-39c5-2f52b24491d4.srt\""
[2020-07-17 16:15:10.495 -05:00] [INF] [27] App: /usr/lib/jellyfin-ffmpeg/ffmpeg -hwaccel vaapi -i file:"/data/movies/WALL·E (2008)/WALL·E (2008) [Bluray-1080p x265 AAC].mkv" -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 h264_amf -pix_fmt yuv420p  -profile:v high  -g 72 -keyint_min 72 -sc_threshold 0 -start_at_zero -vsync -1 -codec:a:0 copy -copyts -avoid_negative_ts disabled -f hls -max_delay 5000000 -hls_time 3 -individual_header_trailer 0 -hls_segment_type mpegts -start_number 0 -hls_segment_filename "/config/data/transcodes/f0efa9ee3704a14bb1f85168f7c68362%d.ts" -hls_playlist_type vod -hls_list_size 0 -y "/config/data/transcodes/f0efa9ee3704a14bb1f85168f7c68362.m3u8"

[Artiume]

/usr/lib/jellyfin-ffmpeg/ffmpeg -hwaccel vaapi -i file:

The way ffmpeg works is any flags before the file will be for the decoder and anything after the file is for the encoder, so this tells me it did attempt to use vaapi for decoding.

[nyanmisaka]

[Parsed_scale_vaapi_2 @ 0x563e3af4e980] Failed to render parameter buffer: 6 (invalid VASurfaceID).
Mesa 20.1 or higher is required to enable VAAPI to transcode HEVC videos on AMD GPU.

[h264_amf @ 0x5590871d0280] DLL libamfrt64.so.1 failed to open
AMD AMF hasn't support any native hw decoder. To use AMF encoding on Ubuntu, you need to manually install amdgpu-pro and amf-amdgpu-pro.

[yesrod]

At the risk of sounding stupid:

In regards to

Mesa 20.1 or higher is required to enable VAAPI to transcode HEVC videos on AMD GPU

And with the Docker images, that Mesa version needs to be on the Docker host, and not inside the Docker container, correct?

[nyanmisaka]

And with the Docker images, that Mesa version needs to be on the Docker host, and not inside the Docker container, correct?

It depends on where Jellyfin is installed as libva cannot access to the driver outside a container. You can upgrade to Mesa 20.1 by adding a third-party repository until it is available in your distribution.

[Protektor-Desura]

What third-party repository should I add for Ubuntu 20.04? I couldn't seem to find any that would have updated drivers for my RX550 which uses Gallium drivers in combo with amdgpu-pro that I installed. I installed amdgpu-pro that I got from the AMD website. I installed "amdgpu-pro-20.20-1098277-ubuntu-20.04.tar.xz" and Ubuntu says that "amf-amdgpu-pro_20.20-1098277_amd64.deb" was installed. So I assume that I still need Mesa 20.1+ to get HEVC support?

[Protektor-Desura]

I had this fixed with a 3rd party PPA until the Jellyfin 10.6.0 upgrade. Jellyfin uses Debian and now I can't find a Debian PPA for the docker to upgrade the the Mesa 20.1 or higher libraries. I did find this one for Ubuntu for the linuxserver/jellyfin docker.

apt-get update && apt-get upgrade -y
apt-get install -y gpg
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys F63F0F2B90935439
echo "deb http://ppa.launchpad.net/kisak/kisak-mesa/ubuntu bionic main" > /etc/apt/sources.list.d/kisak-ubuntu-kisak-mesa-bionic.list
apt-get update && apt-get upgrade -y

Any ideas for a Debian PPA with updated Mesa libraries to try and upgrade the Jellyfin docker?

[stale]

This issue has gone 120 days without comment. To avoid abandoned issues, it will be closed in 21 days if there are no new comments.
If you're the original submitter of this issue, please comment confirming if this issue still affects you in the latest release or nightlies, or close the issue if it has been fixed. If you're another user also affected by this bug, please comment confirming so. Either action will remove the stale label.
This bot exists to prevent issues from becoming stale and forgotten. Jellyfin is always moving forward, and bugs are often fixed as side effects of other changes. We therefore ask that bug report authors remain vigilant about their issues to ensure they are closed if fixed, or re-confirmed - perhaps with fresh logs or reproduction examples - regularly. If you have any questions you can reach us on Matrix or Social Media.

[Protektor-Desura]

Still an issue.

[Artiume]

is this specifically a mesa issue?

[nyanmisaka]

should be resolved when bullseye is released.

[UplandsDynamic]

If you change the group to video then it work

I can confirm changing the group of /dev/dri/renderD128 from render to video does seem to resolve this issue on my system.

I'm running version 10.6.4, in docker (docker-compose), on a Ubuntu 20.04.2 host. Hardware is a Ryzen 3 3200G CPU (integrated Radeon Vega Graphics).

If the group is render, I get the errors as per OP, such as:

[Parsed_scale_vaapi_2 @ 0x55fd76758c80] Failed to render parameter buffer: 6 (invalid VASurfaceID).

The group ID of render on my system is 109. I had tried making this group available to docker through the group_add directive in docker-compose (v3.8), to no avail.

After changing the ownership of renderD128 as above, VAAPI transcoding does seem to work without issue.

Hardware acceleration settings below:

[viggy96]

Is there a reason why Jellyfin switched to Debian as the docker image base? Using Ubuntu would allow Jellyfin docker images to have more updated versions of Mesa, easing the experience for users who wish to use AMD or Intel hardware acceleration for transcoding.

I believe it was based on Ubuntu in the beginning, feel free to correct me if I'm wrong.

[Protektor-Desura]

I am not sure why they continue to use debian as the base for docker for a media server when Ubuntu which is more up to date would make way more sense and it is what Linuxserver uses for their docker as well.

[stale]

This issue has gone 120 days without comment. To avoid abandoned issues, it will be closed in 21 days if there are no new comments.
If you're the original submitter of this issue, please comment confirming if this issue still affects you in the latest release or nightlies, or close the issue if it has been fixed. If you're another user also affected by this bug, please comment confirming so. Either action will remove the stale label.
This bot exists to prevent issues from becoming stale and forgotten. Jellyfin is always moving forward, and bugs are often fixed as side effects of other changes. We therefore ask that bug report authors remain vigilant about their issues to ensure they are closed if fixed, or re-confirmed - perhaps with fresh logs or reproduction examples - regularly. If you have any questions you can reach us on Matrix or Social Media.

[cvium]

Bullseye is out now