Cannot play H265 video with hardware decoding via VAAPI

[sbstp]

Describe the bug
I have an i3 9100 with Intel UHD 630 graphics. I'm trying to get jellyfin to use the UHD 630 graphics to transcode H265 to H264. H265 will not play when hardware acceleration is enabled. The error says PlaybackErrorNoCompatibleStream. I followed the guide for Debian/Ubuntu to enable hardware acceleration.

System (please complete the following information):

• OS: Debian 10
• Browser: Firefox
• Jellyfin Version: 10.4.3
• Installed Plugins: none
• Reverse proxy: no

To Reproduce
Play a H265 video with VAAPI enabled.

Expected behavior
It should play normally using hardware acceleration.

Logs

http://192.168.0.50:8096/videos/c53e54e8-0c33-f2cd-4360-f1574f7ea6e5/hls1/main/0.ts?DeviceId=TW96aWxsYS81LjAgKFgxMTsgTGludXggeDg2XzY0OyBydjo3Mi4wKSBHZWNrby8yMDEwMDEwMSBGaXJlZm94LzcyLjB8MTU3ODg0NDM0OTY3Nw11&MediaSourceId=c53e54e80c33f2cd4360f1574f7ea6e5&VideoCodec=h264&AudioCodec=aac&AudioStreamIndex=1&VideoBitrate=139552000&AudioBitrate=448000&PlaySessionId=73d231aab075438d984a31b87582deba&api_key=1ad8358977a046a9833bdb04ea179759&SubtitleMethod=Encode&TranscodingMaxAudioChannels=2&RequireAvc=false&Tag=add0c462aed1a15327af9abf0538991a&SegmentContainer=ts&MinSegments=1&BreakOnNonKeyFrames=True&h264-profile=high,main,baseline,constrainedbaseline&h264-level=51&allowVideoStreamCopy=false&allowAudioStreamCopy=false

{"Protocol":"File","Id":"c53e54e80c33f2cd4360f1574f7ea6e5","Path":"/mnt/tank/movies/en/Monty Python's Life of Brian (1979)/Monty Python's Life of Brian (1979).mkv","Type":"Default","Container":"mkv,webm","Size":2387457792,"Name":"Monty Python's Life of Brian (1979)","IsRemote":false,"ETag":"add0c462aed1a15327af9abf0538991a","RunTimeTicks":56236478464,"ReadAtNativeFramerate":false,"IgnoreDts":false,"IgnoreIndex":false,"GenPtsInput":false,"SupportsTranscoding":true,"SupportsDirectStream":true,"SupportsDirectPlay":true,"IsInfiniteStream":false,"RequiresOpening":false,"RequiresClosing":false,"RequiresLooping":false,"SupportsProbing":true,"VideoType":"VideoFile","MediaStreams":[{"Codec":"hevc","TimeBase":"1/1000","CodecTimeBase":"1001/24000","VideoRange":"SDR","DisplayTitle":"1080P HEVC","IsInterlaced":false,"BitRate":3396311,"RefFrames":1,"IsDefault":true,"IsForced":false,"Height":1036,"Width":1920,"AverageFrameRate":23.9760246,"RealFrameRate":23.9760246,"Profile":"Main 10","Type":"Video","AspectRatio":"1.85:1","Index":0,"IsExternal":false,"IsTextSubtitleStream":false,"SupportsExternalStream":false,"PixelFormat":"yuv420p10le","Level":120},{"Codec":"ac3","Language":"eng","TimeBase":"1/1000","CodecTimeBase":"1/48000","DisplayTitle":"Eng Dolby Digital 5.1 Default","IsInterlaced":false,"ChannelLayout":"5.1","BitRate":448000,"Channels":6,"SampleRate":48000,"IsDefault":true,"IsForced":false,"Type":"Audio","Index":1,"IsExternal":false,"IsTextSubtitleStream":false,"SupportsExternalStream":false,"Level":0},{"Codec":"DVDSUB","Language":"eng","TimeBase":"1/1000","CodecTimeBase":"0/1","localizedUndefined":"Undefined","localizedDefault":"Default","localizedForced":"Forced","DisplayTitle":"Eng - Default","IsInterlaced":false,"IsDefault":true,"IsForced":false,"Type":"Subtitle","Index":2,"IsExternal":false,"IsTextSubtitleStream":false,"SupportsExternalStream":false,"Level":0}],"Formats":[],"Bitrate":3844311,"RequiredHttpHeaders":{}}

/usr/lib/jellyfin-ffmpeg/ffmpeg -hwaccel vaapi -hwaccel_output_format vaapi -vaapi_device /dev/dri/renderD128 -i file:"/mnt/tank/movies/en/Monty Python's Life of Brian (1979)/Monty Python's Life of Brian (1979).mkv" -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 h264_vaapi  -b:v 6792622 -maxrate 6792622 -bufsize 13585244 -profile:v high -level 41 -force_key_frames:0 "expr:gte(t,0+n_forced*3)" -vf "format=nv12|vaapi,hwupload" -copyts -vsync -1 -codec:a:0 aac -strict experimental -ac 2 -ab 384000 -af "volume=2" -f hls -max_delay 5000000 -avoid_negative_ts disabled -start_at_zero -hls_time 3 -individual_header_trailer 0 -hls_segment_type mpegts -start_number 0 -hls_segment_filename "/var/lib/jellyfin/transcoding-temp/transcodes/transcodes/transcodes/transcodes/transcodes/transcodes/transcodes/transcodes/b8a809e56c1793f2e236e745e89f241c%d.ts" -hls_playlist_type vod -hls_list_size 0 -y "/var/lib/jellyfin/transcoding-temp/transcodes/transcodes/transcodes/transcodes/transcodes/transcodes/transcodes/transcodes/b8a809e56c1793f2e236e745e89f241c.m3u8"


ffmpeg version 4.2.1 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 8 (Debian 8.3.0-6)
  configuration: --toolchain=hardened --prefix=/usr --target-os=linux --enable-cross-compile --extra-cflags=--static --enable-gpl --enable-static --disable-doc --disable-ffplay --disable-shared --disable-libxcb --disable-sdl2 --disable-xlib --enable-libfontconfig --enable-fontconfig --enable-gmp --enable-gnutls --enable-libass --enable-libbluray --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libwebp --enable-libx264 --enable-libx265 --enable-libzvbi --enable-omx --enable-omx-rpi --enable-version3 --enable-vaapi --enable-vdpau --arch=amd64 --enable-nvenc --enable-nvdec
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
Input #0, matroska,webm, from 'file:/mnt/tank/movies/en/Monty Python's Life of Brian (1979)/Monty Python's Life of Brian (1979).mkv':
  Metadata:
    title           : Life of Brian 1979 BluRay 10Bit 1080p DD5.1 H265-d3g
    encoder         : libebml v1.3.4 + libmatroska v1.4.5
    creation_time   : 2017-11-20T22:48:05.000000Z
  Duration: 01:33:43.65, start: 0.000000, bitrate: 3396 kb/s
    Chapter #0:0: start 0.000000, end 404.654000
    Metadata:
      title           : Logos/Three Wise Men
    Chapter #0:1: start 404.654000, end 686.811000
    Metadata:
      title           : Opening Titles Animation
    Chapter #0:2: start 686.811000, end 847.388000
    Metadata:
      title           : Judea AD 33
    Chapter #0:3: start 847.388000, end 1084.166000
    Metadata:
      title           : Alms For An Ex-leper
    Chapter #0:4: start 1084.166000, end 1424.048000
    Metadata:
      title           : "You're one of them"
    Chapter #0:5: start 1424.048000, end 1652.734000
    Metadata:
      title           : The People's Front Of Judea
    Chapter #0:6: start 1652.734000, end 1854.019000
    Metadata:
      title           : "Romans Go Home"
    Chapter #0:7: start 1854.019000, end 2292.165000
    Metadata:
      title           : Raid On Pilate's Palace
    Chapter #0:8: start 2292.165000, end 2549.547000
    Metadata:
      title           : Pilate
    Chapter #0:9: start 2549.547000, end 2892.348000
    Metadata:
      title           : Crash Landing
    Chapter #0:10: start 2892.348000, end 3356.937000
    Metadata:
      title           : Haggling
    Chapter #0:11: start 3356.937000, end 3770.767000
    Metadata:
      title           : The Shoe And The Gourd
    Chapter #0:12: start 3770.767000, end 4199.987000
    Metadata:
      title           : Hail Messiah
    Chapter #0:13: start 4199.987000, end 4839.042000
    Metadata:
      title           : "A very naughty boy"
    Chapter #0:14: start 4839.042000, end 5362.649000
    Metadata:
      title           : "Once cross each"
    Chapter #0:15: start 5362.649000, end 5623.648000
    Metadata:
      title           : Pilate's Passover Address
    Stream #0:0: Video: hevc (Main 10), yuv420p10le(tv, bt709), 1920x1036 [SAR 1:1 DAR 480:259], 23.98 fps, 23.98 tbr, 1k tbn, 23.98 tbc (default)
    Metadata:
      BPS             : 2934644
      BPS-eng         : 2934644
      DURATION        : 01:33:42.617000000
      DURATION-eng    : 01:33:42.617000000
      NUMBER_OF_FRAMES: 134808
      NUMBER_OF_FRAMES-eng: 134808
      NUMBER_OF_BYTES : 2062547962
      NUMBER_OF_BYTES-eng: 2062547962
      _STATISTICS_WRITING_APP: DVDFab 10.0.6.6
      _STATISTICS_WRITING_APP-eng: DVDFab 10.0.6.6
      _STATISTICS_WRITING_DATE_UTC: 2017-11-20 22:48:05
      _STATISTICS_WRITING_DATE_UTC-eng: 2017-11-20 22:48:05
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
    Stream #0:1(eng): Audio: ac3, 48000 Hz, 5.1(side), fltp, 448 kb/s (default)
    Metadata:
      BPS             : 448000
      BPS-eng         : 448000
      DURATION        : 01:33:43.648000000
      DURATION-eng    : 01:33:43.648000000
      NUMBER_OF_FRAMES: 175739
      NUMBER_OF_FRAMES-eng: 175739
      NUMBER_OF_BYTES : 314924288
      NUMBER_OF_BYTES-eng: 314924288
      _STATISTICS_WRITING_APP: DVDFab 10.0.6.6
      _STATISTICS_WRITING_APP-eng: DVDFab 10.0.6.6
      _STATISTICS_WRITING_DATE_UTC: 2017-11-20 22:48:05
      _STATISTICS_WRITING_DATE_UTC-eng: 2017-11-20 22:48:05
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
    Stream #0:2(eng): Subtitle: dvd_subtitle, 1920x1036 (default)
    Metadata:
      BPS             : 12471
      BPS-eng         : 12471
      DURATION        : 01:31:03.250000000
      DURATION-eng    : 01:31:03.250000000
      NUMBER_OF_FRAMES: 1367
      NUMBER_OF_FRAMES-eng: 1367
      NUMBER_OF_BYTES : 8517153
      NUMBER_OF_BYTES-eng: 8517153
      _STATISTICS_WRITING_APP: DVDFab 10.0.6.6
      _STATISTICS_WRITING_APP-eng: DVDFab 10.0.6.6
      _STATISTICS_WRITING_DATE_UTC: 2017-11-20 22:48:05
      _STATISTICS_WRITING_DATE_UTC-eng: 2017-11-20 22:48:05
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
Only '-vf format=nv12|vaapi,hwupload' read, ignoring remaining -vf options: Use ',' to separate filters
Only '-af volume=2' read, ignoring remaining -af options: Use ',' to separate filters
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> h264 (h264_vaapi))
  Stream #0:1 -> #0:1 (ac3 (native) -> aac (native))
Press [q] to stop, [?] for help
[h264_vaapi @ 0x560517c356c0] No usable encoding profile found.
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
[aac @ 0x560517c36dc0] Qavg: 59590.984
[aac @ 0x560517c36dc0] 2 frames left in the queue on closing
Conversion failed!

[sbstp]

I believe this is a dupe of #1901 . I verified and my HEVC videos are 10 bit.

[barolo]

@sbstp AFAIK the fix is already merged in the master [nightlies] that's why the dupe is closed, it's not yet available in stable [10.4.3]

[Shadowghost]

I installed the latest nightly on my Debian buster (with i965-va-driver installed) and at first it still seemed like it doesn't work. I couldn't transcode HEVC 10 bit. With the following errors:

Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> h264 (h264_vaapi))
  Stream #0:1 -> #0:1 (opus (native) -> aac (native))
Press [q] to stop, [?] for help
[Parsed_subtitles_2 @ 0x55ab1fe0dc80] Shaper: FriBidi 0.19.7 (SIMPLE) HarfBuzz-ng 2.3.1 (COMPLEX)
[Parsed_subtitles_2 @ 0x55ab1fe0dc80] Using font provider fontconfig
Impossible to convert between the formats supported by the filter 'Parsed_hwupload_1' and the filter 'auto_scaler_0'
Error reinitializing filters!
Failed to inject frame into filter network: Function not implemented
Error while processing the decoded data for stream #0:0
Conversion failed!

[h264_vaapi @ 0x557d52f17f00] No usable encoding profile found.
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
Conversion failed!

After some digging I found out that it may be caused by the not installed non-free part of intel-vaapi. I enabled non-free in /etc/apt-sources.list an installed intel-media-va-driver-non-free i965-va-driver-shaders and encoding works fine now (afaik installing only the latter one should fix it).
Seems promising: Transcoding 1080p HEVC 10 bit SDR to 1080p h264 8bit SDR takes just ~25% CPU. Transcoding 4k SDR to 1080p SDR keeps CPU at 25% but no video stutters really bad.
Maybe we should add the info about non-free software needed to the installation instructions.

Only thing I'm currently a bit startled about is that the decoding is done in software and not in hardware. Anyone having an idea why?
Relevant log lines:

 Stream #0:0 -> #0:0 (hevc (native) -> h264 (h264_vaapi))
 Stream #0:1 -> #0:1 (opus (native) -> aac (native))

[Artiume]

/usr/lib/jellyfin-ffmpeg/ffmpeg -hwaccel vaapi -hwaccel_output_format vaapi -vaapi_device /dev/dri/renderD128 -i file:"/Monty Python.mkv" -codec:v:0 h264_vaapi

Snipped for easier reading. So FFmpeg is attempting to use Hardware Acceleration on the Decode portion...

https://en.wikichip.org/wiki/intel/uhd_graphics/630
It definitely supports it so I'm not sure what the issue is.

[Artiume]

/usr/lib/jellyfin-ffmpeg/ffmpeg -hwaccel vaapi

From https://trac.ffmpeg.org/wiki/HWAccelIntro
Internal hwaccel decoders are enabled via the -hwaccel option. The software decoder starts normally, but if it detects a stream which is decodable in hardware then it will attempt to delegate all significant processing to that hardware. If the stream is not decodable in hardware (for example, it is an unsupported codec or profile) then it will still be decoded in software automatically. If the hardware requires a particular device to function (or needs to distinguish between multiple devices, say if several graphics cards are available) then one can be selected using -hwaccel_device.

So it should automatically decode the h265 if it were possible. And so I verified that your hardware does support it. I haven't checked drivers yet but it looks you solved that issue.

IIRC Nonfree needs to be added to ffmpeg at compiling. Perhaps the libs are still missing somehow. That'd be worth looking into.

ffmpeg -buildconf would show what's added at compiling

[Lilolbet]

Docker container is definitely missing some packages for decoding of H265 content when transcoding to H264. Preferably I would like to transcode H265 -> H265, but I see no way to enable this at the moment?

Anyhow, installing intel-media-va-driver-non-free i965-va-driver-shaders to the docker container, as suggested by @Shadowghost, fixes the decoding problem instantly.

[Artiume]

Why would you want to reencode to h265? That's not something you would want a media server to do on the fly.

We'll have to look at intel-media-va-driver-non-free we might not be able to integrate that into a pre-built docker image depending on licensing. I know that non-free is at least an issue for FFmpeg.

[Shadowghost]

Just a note about HEVC decoding: Internal in the ffmpeg log already implies that hwaccel is used and enabled if available. Sadly there is no seperate flag for decoding like there is h264_vaapi for vaapi encoding. I researched that yesterday but didn't note down the link to the corresponding documentation (besides the one @artium already linked above)...
If there is a licensing problem when using nonfree on a prebuilt docker-image with ffmpeg the only thing we could do about that is writing a note in the docs about it and maybe provide a Dockerfile so that users can build the container by themself.

[Lilolbet]

@Shadowghost it may imply it, but I can not transcode H265 to H264 without the additional packages. Transcoding H264 to H264 works just fine.

I therefore assume that H265 can not be decoded in the container, at least not using VAAPI.

Edit:
@Artiume, I have a very weak upload (3Mbps) and can only stream decent quality 720p content. Using H265 I could have roughly the same quality, but could raise the resolution up to 1080p.

[Artiume]

Without your logs that are from a fresh image without the updates, I couldn't give you the exact reason the decoder is failing. Feels pretty obvious that it's the missing packages though

[Artiume]

so H265 has a lot of compatibility issues and isn't supported by browsers. Jellyfin also isn't designed to encode to h265, only h264. h264 is basically the universal compatibility codec. Adding h265 encoding isn't an impossible feature, but isn't something that could be added easily and wouldn't benefit many.

[Artiume]

@Shadowghost

Sadly there is no seperate flag for decoding like there is h264_vaapi for vaapi encoding. I researched that yesterday but didn't note down the link to the corresponding documentation

I think you want avcodec_find_decoder_by_name

If you want to check the rest of my HWA/codec notes for more info, here ya go
https://github.com/Artiume/jellyfin-docs/blob/master/general/wiki/main.md

[stale]

This issue has gone 120 days without comment. To avoid abandoned issues, it will be closed in 21 days if there are no new comments.
If you're the original submitter of this issue, please comment confirming if this issue still affects you in the latest release or nightlies, or close the issue if it has been fixed. If you're another user also affected by this bug, please comment confirming so. Either action will remove the stale label.
This bot exists to prevent issues from becoming stale and forgotten. Jellyfin is always moving forward, and bugs are often fixed as side effects of other changes. We therefore ask that bug report authors remain vigilant about their issues to ensure they are closed if fixed, or re-confirmed - perhaps with fresh logs or reproduction examples - regularly. If you have any questions you can reach us on Matrix or Social Media.

[vtruong68]

I'm still experiencing this issue and am using the docker container. Currently using an AMD APU with VAAPI and any h265 playback results in the same results as mentioned by the OP.

[Artiume]

Are you basing this off master or 10.5.5?

[vtruong68]

I'm currently on 10.5.5 it seems.

[Artiume]

10.5.5 is old at this point. Test master please. And if master doesnt work, I have two PR's you can try.

[vtruong68]

Just tested the nightly release on docker. Still seems to be broken.

[Artiume]

Try artiume/jellyfin:hwaccel

[vtruong68]

That seems to have done the trick! H265 videos seem to be playing fine as far as I can tell.

[Artiume]

Check the ffmpeg logs and thoroughly test out the image to make sure everything is normal :). Best to find the bugs now

[jorritsmit]

do you have any plans to make a pull request for this?

[stale]

This issue has gone 120 days without comment. To avoid abandoned issues, it will be closed in 21 days if there are no new comments.
If you're the original submitter of this issue, please comment confirming if this issue still affects you in the latest release or nightlies, or close the issue if it has been fixed. If you're another user also affected by this bug, please comment confirming so. Either action will remove the stale label.
This bot exists to prevent issues from becoming stale and forgotten. Jellyfin is always moving forward, and bugs are often fixed as side effects of other changes. We therefore ask that bug report authors remain vigilant about their issues to ensure they are closed if fixed, or re-confirmed - perhaps with fresh logs or reproduction examples - regularly. If you have any questions you can reach us on Matrix or Social Media.