VAAPI H265 encoding

[aligirayhanozbay]

Hi, I'm trying to add support for H265 encoding to work with the VAAPI backend. I have a proof-of-concept complete, but as it stands the output is bad; there is heavy artefacting as seen here. I assume this is due to my settings, so I'd appreciate some input regarding what settings I can choose to fix it. I am running this on an AMD RDNA2 discrete GPU with kernel 5.18.12, VAAPI 1.15.0 and Mesa 22.1.3 on Debian sid.

Unfortunately, I do not have hardware to test out any other H265 encoder types like nvenc. But adding these should not be too difficult.

Additionally, I added support for some extra profiles for CQP for both the h264 and the new h265 encoders.

[GloriousEggroll]

This patch actually works great. That garbled output is because the mesa version on debian is tool old/has problems. I'm on Nobara (Fedora) with mesa git and just tested it and it works well.

Sample:

https://www.youtube.com/watch?v=DDIXR3ojpRQ

ffprobe output:

Input #0, matroska,webm, from '2022-07-31 20-42-39.mkv':
Metadata:
ENCODER : Lavf59.16.100
Duration: 00:02:29.77, start: 0.000000, bitrate: 60156 kb/s
Stream #0:0: Video: hevc (Main), yuv420p(tv), 3840x2160, 60 fps, 60 tbr, 1k tbn
Metadata:
DURATION : 00:02:29.767000000
Stream #0:1: Audio: aac (LC), 48000 Hz, stereo, fltp
Metadata:
title : Track1
DURATION : 00:02:29.760000000

[aligirayhanozbay]

@GloriousEggroll That's good to hear. I guess I can try it on arch with mesa-git from the AUR at some point.

[aligirayhanozbay]

@GloriousEggroll I confirmed on a Nobara live environment that this PR works very well. I am unsure whether using mesa-git or OBS 28 fixes it, but in either case, it seems that the PR indeed works fine.

[fzwoch]

Works fine on Debian sid on a RX 570. Thanks!

[Atemu]

@GloriousEggroll which mesa version did you test this with? I'm on 22.1.4 and I'm getting the same artifacts as @aligirayhanozbay.

I also tried a7e64ab63c5aa0b5c2079badc3ae0d5668c234f8 from today's master and mesa-22.2.0-rc1 but both resulted in h265 not showing VAAPI as an available encoder type and thus not working at all.

6800XT.

[Atemu]

Nvm, configuration error on my end: You need to explicitly enable VAAPI codecs using -Dvideo-codecs=vc1dec,h264dec,h264enc,h265dec,h265enc now.

mesa-22.2.0-rc1 works just fine!