VP9 10-bit VAAPI HW decoding is broken

[smp79]

Bug report

Describe the bug

Here is a clear and concise description of what the problem is:

VP9 10-bit videos are unplayable in Linux using VAAPI HW acceleration:

error <general>: GetPicture - avcodec_receive_frame returned failure
debug <general>: CVideoPlayerVideo - video decoder returned error
error <general>: ffmpeg[0x5bc2b20]: [vp9] Not all references are available

The same videos play fine in Kodi on Windows 11 on the same Intel hardware.

To Reproduce

Steps to reproduce the behavior:

Play this sample file on any modern Intel hardware.

Debuglog

The debuglog can be found here:
https://hastebin.com/share/atigebacoh.yaml

Your Environment

Used Operating system:

• Linux

• Operating system version/name: LibreELEC

• Kodi version: current master

note: Once the issue is made we require you to update it with new information or Kodi versions should that be required.
Team Kodi will consider your problem report however, we will not make any promises the problem will be solved.

[neo1973]

Works without issues with VAAPI on an AMD Ryzen 5700U on both Wayland and GBM. (And playback is way smoother than with MPV 😉)

[smp79]

This seem to be specific to Intel. Doesn't play on my Tiger Lake Celeron 6305, also doesn't work on Alder Lake-N N100

[lrusak]

Sounds like a driver issue, can you report it to the Intel media-driver?

[smp79]

can you report it to the Intel media-driver?

I did, back in 2021.
intel/media-driver#1172

[Stereodude79]

Sounds like a driver issue, can you report it to the Intel media-driver?

When smp79 reported it 2+ years ago to Intel they clearly demonstrated that the driver works fine under Linux for 10-bit VP9 decoding. I similarly just tried decoding a 10-bit VP9 sample that doesn't work in LE with LibreELEC-Generic.x86_64-12.0-nightly-20230819-8e18456 LE nightly using FFMPEG and the VAAPI iHD driver. It decodes 10-bit VP9 using VAAPI with the iHD driver. It's pretty easy to test.

ffmpeg -hwaccel vaapi -init_hw_device vaapi=hw:/dev/dri/renderD128 -hwaccel_flags allow_profile_mismatch -filter_hw_device hw -v verbose -i "Alaska in 8K 60p [VP9 4K HDR].mp4" -pix_fmt yuv420p -f rawvideo -vsync passthrough -vframes 10 -y output.yuv

Output file #0 (output.yuv):
Output stream #0:0 (video): 10 frames encoded; 10 packets muxed (124416000 bytes);
Total: 10 packets (124416000 bytes) muxed

It seems like there is an issue with the way Kodi interfaces & interacts with the driver, not the driver itself.

[fritsch]

Decoding is only one thing, zero copy rendering the other. When the output surfaces are incompatible, you will see garbage on screen. Kodi has assumption that after decoding a 10 bit yuv surface is given. If that is not the case, Kodi won't render it.

[Stereodude79]

Decoding is only one thing, zero copy rendering the other. When the output surfaces are incompatible, you will see garbage on screen. Kodi has assumption that after decoding a 10 bit yuv surface is given. If that is not the case, Kodi won't render it.

I tried to enable a LIBVA_TRACE to get more details what the driver is doing when Kodi accesses it, but am getting stuck by the read only file system that prevents the creation of a libva.conf file in /etc/.

All I can tell you so far is what ffmpeg reports:
[graph 0 input from stream 0:0 @ 0x4b90980] w:3840 h:2160 pixfmt:p010le tb:1/16000 fr:60000/1001 sar:1/1
[auto_scaler_0 @ 0x4b9ec00] w:iw h:ih flags:'bicubic' interl:0
[format @ 0x4b912c0] auto-inserting filter 'auto_scaler_0' between the filter 'Parsed_null_0' and the filter 'format'
[auto_scaler_0 @ 0x4b9ec00] w:3840 h:2160 fmt:p010le sar:1/1 -> w:3840 h:2160 fmt:yuv420p sar:1/1 flags:0x4

[a1rwulf]

The error is probably around here:
https://github.com/xbmc/xbmc/blob/master/xbmc/cores/VideoPlayer/DVDCodecs/Video/VAAPI.cpp#L1162

I had a similar issue when doing the av1 implementation and seems like we'd need to expand this if to include VP9 10bit.
Only a guess though, I don't have time to work on this.

[smp79]

diff --git a/xbmc/cores/VideoPlayer/DVDCodecs/Video/VAAPI.cpp b/xbmc/cores/VideoPlayer/DVDCodecs/Video/VAAPI.cpp
index 856583a..c4b3e71 100644
--- a/xbmc/cores/VideoPlayer/DVDCodecs/Video/VAAPI.cpp
+++ b/xbmc/cores/VideoPlayer/DVDCodecs/Video/VAAPI.cpp
@@ -1162,6 +1162,7 @@ bool CDecoder::ConfigVAAPI()
   if ((m_vaapiConfig.profile == VAProfileHEVCMain10
 #if VA_CHECK_VERSION(1, 8, 0)
        || m_vaapiConfig.profile == VAProfileAV1Profile0
+       || m_vaapiConfig.profile == VAProfileVP9Profile2
 #endif
        ) &&
       m_vaapiConfig.bitDepth == 10)

^ This fixed it for me

[fritsch]

Please send a Pullrequest - that should get the render part correct.

[a1rwulf]

When you PR, please make sure to put your condition outside of the VA_CHECK_VERSION(1, 8, 0) block.
VP9_Profile2 is bound to #if VA_CHECK_VERSION(0, 39, 0) in ffmpeg, which is probably old enough to be ignored.