feat(server): transcoding hardware acceleration

[mertalev]

Description

Support for NVENC, Quick Sync and VAAPI transcoding backends. This will make transcoding much faster at the cost of lower quality at the same bitrate compared to software encoding.

Additionally, this substantially changes the server/microservices Dockerfile:

• Uses a Debian-based image
  • Alpine is incompatible with many acceleration APIs
• Builds Imagemagick and libvips from source
  • Debian's Imagemagick package has issues with RAW images

Fixes #1225, fixes #3084

Notes:

• This adds a new file hwaccel.yml that immich-microservices uses to set the appropriate settings for the docker container. Be sure to uncomment the right sections in this file to ensure hardware acceleration works.
• This uses a Debian-based image, so if running in a dev environment you should delete your current immich-microservices and immich-server containers and the dist and node_modules folders relating to them. Otherwise you might see errors stemming from it trying to load Alpine binaries.

How Has This Been Tested?

Current testing in production build on three systems, one with an Nvidia GPU, another with an older Intel CPU, and a third with a 13th gen Intel CPU:

• NVENC
  • ✅ H.264
  • ✅ HEVC
  • ❌ VP9 does not work (not supported)
• VAAPI
  • ✅ H.264
  • ✅ HEVC
  • ✅ VP9 (on a supporting Intel CPU)
• QSV
  • ✅ H.264
  • ✅ HEVC
  • ✅ VP9 (on a supporting CPU)

NOTE: Only a limited variety of image formats were tested with this, primarily HEIC and JPEG. It would be good to check other formats as well to ensure Imagemagick and libvips work as expected.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
immich	⬜️ Ignored (Inspect)	Visit Preview		Aug 2, 2023 0:34am

[martabal]

Does Dockerfile.accel support raw images ? It's just for development ?

[mertalev]

The image has to support image formats like RAW since it's responsible for thumbnail generation.

[martabal]

Does it work ? With this image we have to build imagemagick & libvips to support raw images.

[martabal]

And you can possibly have a version mismatch between libvips & sharp. Currently immich uses sharp v31.3 which requires libvips v8.13.3 and bookworm provides libvips v8.14.1-3, so it's not an issue for now

[mertalev]

It seems Imagemagick can handle raw images, but sharp throws an error. I can try compiling libvips as well to see if that helps.

[mertalev]

I can confirm RAW images work now after compiling libvips. The image is also a decent bit smaller at 1.07gb instead of 1.22gb.

[samip5]

I tried this with Quick Sync, and well, libva errors on 6th gen Intel :(

# ls -al /dev/dri/
crw-rw----    1 root     44        226,   0 Jul 17 20:22 card0
crw-rw----    1 root     109       226, 128 Jul 17 20:22 renderD128

[Nest] 7  - 07/17/2023, 10:20:31 PM     LOG [MediaService] Start encoding video 5fceb8ef-581b-43ea-9610-951b7675f0ee {"inputOptions":["-init_hw_device qsv=accel:/dev/dri/renderD128","-filter_hw_device accel"],"outputOptions":["-vcodec h264_qsv","-acodec aac","-movflags faststart","-fps_mode passthrough","-vf format=nv12,hwupload=extra_hw_frames=64","-preset 7","-global_quality 23"],"twoPass":false}
[Nest] 7  - 07/17/2023, 10:20:31 PM   ERROR [MediaRepository] ffmpeg version 6.0 Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 12.2.1 (Alpine 12.2.1_git20220924-r10) 20220924
  configuration: --prefix=/usr --disable-librtmp --disable-lzma --disable-static --disable-stripping --enable-avfilter --enable-gnutls --enable-gpl --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libdrm --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libmp3lame --enable-libopenmpt --enable-libopus --enable-libplacebo --enable-libpulse --enable-librist --enable-libsoxr --enable-libsrt --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-lto --enable-pic --enable-postproc --enable-pthreads --enable-shared --enable-vaapi --enable-vdpau --enable-vulkan --optflags=-O3 --enable-libjxl --enable-libsvtav1 --enable-libvpl
  libavutil      58.  2.100 / 58.  2.100
  libavcodec     60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter     9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample   5.  0.100 /  5.  0.100
  libpostproc    57.  1.100 / 57.  1.100
[AVHWDeviceContext @ 0x7fc4ffe2ef00] Failed to initialise VAAPI connection: -1 (unknown libva error).
Device creation failed: -5.
Failed to set value 'qsv=accel:/dev/dri/renderD128' for option 'init_hw_device': I/O error
Error parsing global options: I/O error

[Nest] 7  - 07/17/2023, 10:20:31 PM   ERROR [MediaService] Error: ffmpeg exited with code 1: Device creation failed: -5.
Failed to set value 'qsv=accel:/dev/dri/renderD128' for option 'init_hw_device': I/O error
Error parsing global options: I/O error

[mertalev]

From that log it's using Alpine binaries. For the dev environment, I think you need to set both immich-server and immich-microservices to use Dockerfile.accel and rebuild the containers.

(That should probably be the default for dev, though the bigger conversation to have is whether this image should replace the existing Alpine one or be an alternate version.)

Also double check that you have the hwaccel.yml and enabled the renderD128 device on it.

Edit: It actually does seem to be the default, so I guess it's something else after all.

[samip5]

From that log it's using Alpine binaries. For the dev environment, I think you need to set both immich-server and immich-microservices to use Dockerfile.accel and rebuild the containers.

(That should probably be the default for dev, though the bigger conversation to have is whether this image should replace the existing Alpine one or be an alternate version.)

Also double check that you have the hwaccel.yml and enabled the renderD128 device on it.

I'm using the ghcr.io/immich-app/immich-server:pr-3171 image tag though?

Can be seen here

[jrasm91]

Does the PR build push hardware accelerated enabled images?

[samip5]

Does the PR build push hardware accelerated enabled images?

The option exists in settings, but it seems to be alpine based images only.

[mertalev]

It seems GH was building with the Alpine-based Dockerfile before this. I moved Dockerfile.accel to Dockerfile for now to avoid the headache of managing both.

[jrasm91]

Code looks good!

[brighteyed]

I have not tested this PR, but LGTM!