Videos: Improve Intel QSV Hardware Support and Performance

[xrad]

1. What is not working as documented?

[DISCLAIMER: I know dev time is scarce, that this is a very specific case and that it may not even be specific to photoprism. But it happens and is reproducable in PP's docker compose container and I could not find anything about this issue anywhere else.]

Qsv accelerated ffmpeg does not work and falls back to software coding. In the docker logs this manifests as:

3.100\n  libswscale      7.  1.100 /  7.  1.100\n  libswresample   4. 10.100 /  4. 10.100\n  libpostproc    57.  1.100 / 57.  1.100\nlibva info: VA-API version 1.19.0\nlibva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so\nlibva info: Found init function __vaDriverInit_1_18\nlibva info: va_openDriver() returns 0\n[AVHWDeviceContext @ 0x562abf67fe80] Error creating a MFX session: -9.\nDevice creation failed: -1313558101.\nFailed to set value '/dev/dri/renderD128' for option 'qsv_device': Unknown error occurred\nError parsing global options: Unknown error occurred\n"
root-photoprism-1  | time="2024-01-23T13:31:48Z" level=warning msg="h264_qsv: failed to transcode 

2. How can we reproduce it?

In the container specified below, you can reproduce the issue on the command line. To further simplify, I only focus on decoding:

root@231128:/photoprism/storage$ ffmpeg -hwaccel qsv -c:v h264_qsv -i input.mp4 -c:v libx264 -f mp4 output.mp4

...
libva info: VA-API version 1.19.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_18
libva info: va_openDriver() returns 0
[AVHWDeviceContext @ 0x555f7c623cc0] Error creating a MFX session: -9.
Device creation failed: -1313558101.
No device available for decoder: device type qsv needed for codec h264_qsv.
Stream mapping:
  Stream #0:1 -> #0:0 (h264 (h264_qsv) -> h264 (libx264))
  Stream #0:0 -> #0:1 (aac (native) -> aac (native))
Device setup failed for decoder on input stream #0:1 : Unknown error occurred

Could "VA-API version 1.19.0" vs "__vaDriverInit_1_18" be a culprit?

/dev/dri is properly mapped:

root@231128:/photoprism$ ls -l /dev/dri/
total 0
crw-rw---- 1 root video 226,   0 Jan 24 10:52 card0
crw-rw---- 1 root   104 226, 128 Jan 24 10:52 renderD128

For now, I run PP as root to avoid permission issues accessing /dev/dri/renderD128

3. What behavior do you expect?

QSV working.

4. What could be the cause of your problem?

Incompatibility between Proxmox kernel/drm and PP's Ubuntu media stack?

5. Can you provide us with example files for testing, error logs, or screenshots?

Sure, let me know what would be helpful.

6. Which software versions do you use?

Proxmox: pve-manager/8.1.4/ec5affc9e41f1d79 (running kernel: 6.5.11-7-pve)
PP running in a privileged LXC container

ii  ffmpeg                               7:6.0-6ubuntu1                          amd64        Tools for transcoding, streaming and playing of multimedia files
ii  intel-media-va-driver-non-free:amd64 23.2.3+ds1-1                            amd64        VAAPI driver for the Intel GEN8+ Graphics family
ii  intel-opencl-icd                     22.43.24595.41-1                        amd64        Intel graphics compute runtime for OpenCL
ii  libdrm-intel1:amd64                  2.4.115-1                               amd64        Userspace interface to intel-specific kernel DRM services -- runtime
ii  libffmpeg-nvenc-dev                  12.0.16.0-1                             all          FFmpeg headers for interfacing with NVIDIA's codec APIs
ii  libffms2-5:amd64                     2.40+git20211209-2build2                amd64        Cross platform ffmpeg wrapper library
ii  libva-dev:amd64                      2.19.0-1                                amd64        Video Acceleration (VA) API for Linux -- development files
ii  libva-drm2:amd64                     2.19.0-1                                amd64        Video Acceleration (VA) API for Linux -- DRM runtime
ii  libva-glx2:amd64                     2.19.0-1                                amd64        Video Acceleration (VA) API for Linux -- GLX runtime
ii  libva-wayland2:amd64                 2.19.0-1                                amd64        Video Acceleration (VA) API for Linux -- Wayland runtime
ii  libva-x11-2:amd64                    2.19.0-1                                amd64        Video Acceleration (VA) API for Linux -- X11 runtime
ii  libva2:amd64                         2.19.0-1                                amd64        Video Acceleration (VA) API for Linux -- runtime
ii  vainfo                               2.12.0+ds1-1                            amd64        Video Acceleration (VA) API for Linux -- info program

(a) PhotoPrism Architecture & Build Number: AMD64, ARM64, ARMv7,...

PhotoPrism CE Build 231128-f48ff16ef AMD64

(b) Database Type & Version: MariaDB, MySQL, SQLite,...

Default mariadb from docker-compose

(c) Operating System Types & Versions: Linux, Windows, Android,...

Proxmox LXC, Debian 12

(d) Browser Types & Versions: Firefox, Chrome, Safari on iPhone,...

n/a

(e) Ad Blockers, Browser Plugins, and/or Firewall Software?

n/a

7. On what kind of device is PhotoPrism installed?

(a) Device / Processor Type: Raspberry Pi 4, Intel Core i7-3770, AMD Ryzen 7 3800X,...

Dell Wyse 5070
Intel J4005 (Gemini Lake)

(b) Physical Memory & Swap Space in GB

8GB, 4GB

(c) Storage Type: HDD, SSD, RAID, USB, Network Storage,...

SSD

(d) Anything else that might be helpful to know?

8. Do you use a Reverse Proxy, Firewall, VPN, or CDN?

No

[lastzero]

Getting this to work might not be easy in some cases and we provide the functionality as is, including the related source code in our public repository. For example, QVC support may depend on the driver versions installed on your host and in the Docker image. It could also depend on certain BIOS settings. If you have specific suggestions for improving our docs or transcoding code, we'll be happy to consider/merge them :)

[xrad]

@lastzero I know, that's why I added the disclaimer on top. I don't really expect a direct solution from you, but I felt that documenting this gives you or other users a chance to reflect on this and perhaps some more angles to investigate may boil up.

For example, since I can reproduce the issue with a pure (and simple) ffmpeg invokation seems to indicate to me that there may be an inconsistency within Ubuntu or between Ubuntu and Proxmox, so it wouldn't even be a PP issue. So it would be nice to keep this around as a place where people affected by this can exchange experiences.

[xrad]

After a ton of experiments I have a few more insights:

• VAAPI works, but is slow AF since the HW seems only be used for encoding, not decoding.
• The sample_decode from Intel Media SDK works
• At least this shows that my /dev/dri setup, Proxmox, DRM drivers etc are not the issue
• Intel Media SDK (mfx) is deprecated and oneVPL is now official.
• As far as I read some online sources, VPL should fallback to mfx where needed, but I also found this issue in FFMPEG trac - it mentions my exact issue!
• Ubuntu 23.10's ffmpeg is compiled with --enable-libvpl, so it seems like that with this setup older Intel GPUs currently have no chance of working

[lastzero]

Thanks for testing it! Would you also like to help us improve this?

[lastzero]

@xrad Note that based on my experience - without having tested all possible combinations and besides the additional constraints you mentioned - the driver version must also match. So what I was trying to say 4 days ago is that this is a known issue that we are aware of, but have no control over. Also keep in mind that encoding usually requires much more CPU time than decoding. So I can't agree that transcoding is "slow as fuck" just because we don't use the decoder. We made this choice because we support many formats that may not be supported by the GPU/CPU. With your help, we could change the implementation so that the supported decoder formats are compared to the video file format.

[xrad]

Sure, let me know how I can help.

BTW, from what I learned during the last days is that Intel GPUs generally are much more useful for decoding tasks, and just less so for encoding. With vaapi I get .5x-ish speeds, where as the SW fallback is actually doing the job quicker. Certainly I did not intend to diminish your work, sorry if I came across offensive. Most of this stuff is new to me, I would like to add.

[lastzero]

Much of the related code can be found in this directory:

• https://github.com/photoprism/photoprism/tree/develop/internal/ffmpeg

I have friends visiting at the moment, so I can't elaborate more, but if you have any questions, I'll be happy to answer them next week.

[lastzero]

@xrad Besides tweaking the command flags, e.g. depending on the input video format, another option could be to build and bundle a custom FFmpeg version that supports both the old and the new Intel driver (if this is technically possible).

[xrad]

As far as I understand the situation, a custom built ffmpeg (with --enable-libmfx) is the most promising route. With the current one in Ubuntu 23.10 I was not able to find a working command line. I'll see if I get around to doing this.

[xrad]

Ok, it seems like I took a LONG detour to find that simply installing apt install libmfx1 fixes QSV/intel support for me. Apparently the newer libvpl does contain the fallback to mfx for older-gen GPUs but fails to report that it cannot load the library. With libmfx installed I get quite decent a 2.5x rate.

My suggestion would therefore be to simply add libmfx1 to the apt-get line in install-gpu.sh.
#quick-win

I'll be playing around a bit more to see if I can come up with a call to also do the decoding on the GPU.

[xrad]

With this line, I get a whopping 7.7x rate:

/usr/bin/ffmpeg -hwaccel qsv -qsv_device /dev/dri/renderD128 -i input.mp4 -c:a aac -c:v h264_qsv -map 0:v:0 -map 0:a:0? -r 30 -b:v 12M -bitrate 12M -f mp4 -movflags +faststart -y output.avc
...
Stream mapping:
  Stream #0:1 -> #0:0 (hevc (hevc_qsv) -> h264 (h264_qsv))
  Stream #0:0 -> #0:1 (aac (native) -> aac (native))
...
frame= 2048 fps=232 q=31.0 Lsize=  101453kB time=00:01:08.20 bitrate=12186.2kbits/s dup=0 drop=3 speed=7.71x

I needed to add -hwaccel qsv and removed the filter. Supporting PHOTOPRISM_FFMPEG_SIZE would probably require the use of something like -vf "scale_qsv=w=1920:h=1080". Note this has a performance penalty, in my example the 7.7x then drops to about 4.5x.

[lastzero]

@xrad Excellent! We can add libmfx1. Any additional changes to the FFmpeg flags should be tested with a variety of formats so that they don't break anything, e.g. if the original is in a format that the hardware can't decode (or if the resolution happens to be too large for the hardware).