feat(server): hardware video acceleration for Rockchip SOCs via RKMPP

[fyfrey]

This PR adds a new type of hardware acceleration for Rockchip SOCs.
The approach differs from the existing options as it requires custom/patched ffmpeg binaries+libraries from the host OS (cannot put them into an image due to licensing issues).
To enable a user must select the new hwaccel-rkmpp in docker-compose and choose RKMPP as hardware acceleration in the settings UI.

Some performance tests with an RK3588 SoC (4 "fast" ARM cores, 4 "slow" ARM cores, encoding acceleration for H265, HEVC, decoding acceleration for many formats):

• 10s HEVC 4K -> HEVC FHD: 8s with hwaccel, 60s in software
• 10s H264 FHD -> HEVC FHD: 4s with hwaccel, 50s in software
• hwaccel is between 7 and 12 times faster than software transcoding

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
immich	⬜️ Ignored (Inspect)	Visit Preview		Oct 27, 2023 9:37am

[jrasm91]

Are you planning do add any documentation for this setting?

[mertalev]

Is there a particular reason you added this instead of using getScaling?

[fyfrey]

Is there a particular reason you added this instead of using getScaling?

getScaling does not return the actual width and height. only 1 component and the other is -1 or -2 which is not understood by the RKMPP interface (needs explicit pixel values for resizing)

[mertalev]

I think I'd prefer to keep the library stuff outside of the server code if possible. But better would be to have a separate image for it. Is there a link for the licensing issue you mentioned? MPP is Apache 2.0 at least.

[fyfrey]

I think I'd prefer to keep the library stuff outside of the server code if possible. But better would be to have a separate image for it. Is there a link for the licensing issue you mentioned? MPP is Apache 2.0 at least.

ffmpeg is LGPL and I think the patched version violates that. Found a comment from the dev team mentioning a license issue:
https://forum.radxa.com/t/ffmpeg-rkmppenc-hardware-encoding-with-ffmpeg/13488/2

I'd rather not distribute any image with this included.

[mertalev]

Ohhh, it's a copyleft thing. That does make it infeasible to ship it ourselves.

[fyfrey]

Are you planning do add any documentation for this setting?

Yes. Once we settled on the final implementation / configuration (possibly in a follow-up PR)

[fyfrey]

Thank you @mertalev and @jrasm91 for the thorough review, feedback and suggestions! I'm now happy enough with the current state given the circumstances (copyleft issue, missing fluent-ffmpeg env variable support)

[mertalev]

Thanks for adding this! That speed-up is really impressive. I still hope there's a cleaner solution for this, but I think this implementation is okay given the limitations.

[Dark-Existed]

@fyfrey hi, when I want to enable hwaccel-rkmpp.yml for rk3588, I hava this problem, any idea to solve this? thx!
I have installed ffmpeg from https://ppa.launchpadcontent.net/liujianfeng1994/rockchip-multimedia/ubuntu.

Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/lib/aarch64-linux-gnu/pulseaudio/libpulsecommon-15.99.so" to rootfs at "/lib/ffmpeg-mpp/libpulsecommon-15.99.so": open /var/lib/docker/overlay2/fa7c9bd3839088343d0aa48f6bba182c04b5ada6863f668689034e309bb7d660/merged/usr/lib/ffmpeg-mpp/libpulsecommon-15.99.so: read-only file system: unknown 

ls -l /lib/aarch64-linux-gnu/pulseaudio
total 1204
-rw-r--r-- 1 root root 497776 Feb 16  2023 libpulsecommon-15.99.so
-rw-r--r-- 1 root root 686768 Feb 16  2023 libpulsecore-15.99.so
-rw-r--r-- 1 root root  42776 Feb 16  2023 libpulsedsp.so

$ uname -a
Linux rock-5b 5.10.110-rockchip-rk3588 #23.02.2 SMP Fri Feb 17 23:59:20 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

[fyfrey]

@Dark-Existed I have not encountered this error during my tests. It seems like a docker issue.
Maybe you have a symlink to this library in the main lib directory and it tries to put it twice into the container? Or try to clean the docker images and download again

[Dark-Existed]

@fyfrey

version: "3.8"

# Hardware acceleration for transcoding using RKMPP for Rockchip SOCs
# This is only needed if you want to use hardware acceleration for transcoding.
# Supported host OS is Ubuntu Jammy 22.04 with custom ffmpeg from ppa:liujianfeng1994/rockchip-multimedia

services:
  hwaccel:
    security_opt: # enables full access to /sys and /proc, still far better than privileged: true
      - systempaths=unconfined
      - apparmor=unconfined
    group_add:
      - video
    devices:
      - /dev/rga:/dev/rga
      - /dev/dri:/dev/dri
      - /dev/dma_heap:/dev/dma_heap
      - /dev/mpp_service:/dev/mpp_service
    volumes:
      - /usr/bin/ffmpeg:/usr/bin/ffmpeg_mpp:ro
      - /lib/aarch64-linux-gnu:/lib/ffmpeg-mpp:ro
      - /lib/aarch64-linux-gnu/libblas.so.3:/lib/ffmpeg-mpp/libblas.so.3:ro # symlink is resolved by mounting
      - /lib/aarch64-linux-gnu/liblapack.so.3:/lib/ffmpeg-mpp/liblapack.so.3:ro # symlink is resolved by mounting
      - /lib/aarch64-linux-gnu/pulseaudio/libpulsecommon-15.99.so:/lib/ffmpeg-mpp/libpulsecommon-15.99.so:ro

This is the lastest hwaccel-rkmpp.yml, may be this two lines cause it mount twice.

- /lib/aarch64-linux-gnu:/lib/ffmpeg-mpp:ro

- /lib/aarch64-linux-gnu/pulseaudio/libpulsecommon-15.99.so:/lib/ffmpeg-mpp/libpulsecommon-15.99.so:ro

The last line should be remove?
I delete it, the immich_microservices container still hava lib/ffmpeg-mpp/libpulsecommon-15.99.so, may work well ?

[0x0013]

Hi,

I'm trying to get this working on RK3588S nodes in a kubernetes cluster, but so far not successful.

Needed to diverge from the steps in dockerfile:

1. Had a similar error as above when starting the pod. Maybe my containerd runtime doesn't like that /lib/ffmpeg-mpp is mounted read-only and can't place another mount in there. I removed the libpulsecommon-15.99.so mount and created the corresponding symlink on hosts.
2. For testing I set the container to run in privileged mode. AFAIK Kubernetes doesn't have an option to have unconfined sysfs access, and only unmasked procfs didn't work. So far I was unable to find a more restrictive security setting which would work.
3. Relevant devices exposed with generic-device-plugin.

With that, I am able to exec into the container and successfully convert manually using a command similar to that generated by immich:

LD_LIBRARY_PATH=/lib/aarch64-linux-gnu:/lib/ffmpeg-mpp ffmpeg_mpp -i /mnt/ext/video/testvideo.mp4 -c:v hevc_rkmpp_encoder -c:a libopus -movflags faststart -fps_mode passthrough -map 0:1 -map 0:0 -g 256 -v verbose -level 153 -rc_mode 2 -quality_min 73 -quality_max 73 test.mkv

This succeeds. However, when the process is initiated by immich, it still fails:

immich-microservices-79c67f8f6c-npmz7 immich-microservices [Nest] 8  - 01/01/2024, 8:24:21 PMLOG [MediaService] Start encoding video cc39b8db-f66c-46e2-bb80-4127c8513e6c {"inputOptions":[],"outputOptions":["-c:v hevc_rkmpp_encoder","-c:a libopus","-movflags faststart","-fps_mode passthrough","-map 0:0","-map 0:1","-g 256","-v verbose","-level 153","-rc_mode 2","-quality_min 73","-quality_max 73"],"twoPass":false,"ffmpegPath":"ffmpeg_mpp","ldLibraryPath":"/lib/aarch64-linux-gnu:/lib/ffmpeg-mpp"}
immich-microservices-79c67f8f6c-npmz7 immich-microservices [Nest] 8  - 01/01/2024, 8:24:21 PM   ERROR [MediaRepository] Error: ffmpeg exited with code 127
immich-microservices-79c67f8f6c-npmz7 immich-microservices [Nest] 8  - 01/01/2024, 8:24:21 PM   ERROR [MediaService] Error: ffmpeg exited with code 127
immich-microservices-79c67f8f6c-npmz7 immich-microservices [Nest] 8  - 01/01/2024, 8:24:21 PM   ERROR [MediaService] Error occurred during transcoding. Retrying with RKMPP acceleration disabled.

The fallback software encoding then succeeds (after a long time).

Code 127 should mean "command not found", but I'm puzzled as to why it fails if it succeeds when calling it manually from shell. ffmpeg_mpp is mounted from host in /usr/bin/ and even manually calling it, I do not use the full path, so it's apparently in $PATH as it should be.
@fyfrey any ideas what could be the issue?

[fyfrey]

@0x0013 this is a tough one since it runs fine manually in the container...

can you try in docker (and not kubernetes)?

[0x0013]

@fyfrey I partially managed to get it to work. It seems the issue somehow stems from LD_LIBRARY_PATH.

I was trying to compare the environment of my manual shell command ffmpeg_mpp, and what is run by immich. But due to immediate error, I could only see the standard (integrated) ffmpeg command. I noticed this in environment variables - LD_LIBRARY_PATH=undefined.

This prompted me to try adding the variable when running the container:

    env:
    - name: LD_LIBRARY_PATH
      value: /lib/aarch64-linux-gnu:/lib/ffmpeg-mpp

and with this, it is encoding correctly with RKMPP, though I see the command is being run with double paths (second set added by immich) - LD_LIBRARY_PATH=/lib/aarch64-linux-gnu:/lib/ffmpeg-mpp:/lib/aarch64-linux-gnu:/lib/ffmpeg-mpp..

I wonder if the library paths constructed end up as something like undefined:/lib/aarch64-linux-gnu:/lib/ffmpeg-mpp and maybe that's why it would fail?

I'm not sure if adding this path manually for the whole container is a good idea - could this have bad impact elsewhere?

[mertalev]

Setting that env globally means the libraries in it take precedence over libraries elsewhere, such as /lib and /usr/lib. Libraries from those folders will still get loaded if something can't be found, but it can theoretically cause issues if it finds a library with a version or build that isn't actually compatible.

[fyfrey]

@0x0013 it seems your kube environment sets LD_LIBRARY_PATH to the string "undefined", it should be unset/undefined (i.e. echo LD_LIBRARY_PATH should produce nothing).

Don't set the library path globally, it can cause issues by overwriting some libraries in the container with libs from the host where it is not intended (e.g. libvips for image operations).

Try to fix the LD_LIBRARY_PATH issue by unsetting it or define it as a valid, but empty dir.

[0x0013]

@fyfrey Thanks for your help. I did some more testing:

I already hadn't defined LD_LIBRARY_PATH variable before. I tested again without setting it, and the first (rkmpp accelerated) process fails with the same error. Then I check the environment for the running fallback ffmpeg command, and see LD_LIBRARY_PATH=undefined. Is it possible that, when reading existing env variable

const oldLdLibraryPath = process.env.LD_LIBRARY_PATH;

code would mistakenly interpret unset value as undefined, which would then later be set as string when restoring oldLdLibraryPath? (sorry, I don't have experience with TypeScript, this is just a guess)

Then, while testing with the workaround I know is working (globally set LD_LIBRARY_PATH), I noticed something interesting - first time ffmpeg_mpp runs, I see just the library path I set globally - LD_LIBRARY_PATH=/lib/aarch64-linux-gnu:/lib/ffmpeg-mpp. On the second and following transcodings, I see double path - LD_LIBRARY_PATH=/lib/aarch64-linux-gnu:/lib/ffmpeg-mpp:/lib/aarch64-linux-gnu:/lib/ffmpeg-mpp.

This made me think - maybe the path doesn't get added on first run? I never tested this, because software encoding runs so slow, I would change settings and try something else, before it had time to retry a second time. So this time, I set the path globally to a random empty directory LD_LIBRARY_PATH=/home/node, just for testing, and reran the container. First run, ffmpeg_mpp fails just as before, and fallbacks to sofware encoding. I let it run and after a while when the first fallback process is done, the new ffmpeg_mpp processes run successfully:

immich-microservices-74685f4bf9-p8hrv immich-microservices [Nest] 7  - 01/03/2024, 2:42:37 PMLOG [MediaService] Start encoding video 78025895-9811-4f06-b121-0b1db66c4e7e {"inputOptions":[],"outputOptions":["-c:v hevc_rkmpp_encoder","-c:a libopus","-movflags faststart","-fps_mode passthrough","-map 0:0","-map 0:1","-g 256","-v verbose","-level 153","-rc_mode 2","-quality_min 73","-quality_max 73"],"twoPass":false,"ffmpegPath":"ffmpeg_mpp","ldLibraryPath":"/lib/aarch64-linux-gnu:/lib/ffmpeg-mpp"}
immich-microservices-74685f4bf9-p8hrv immich-microservices [Nest] 7  - 01/03/2024, 2:43:02 PMLOG [MediaService] Encoding success 78025895-9811-4f06-b121-0b1db66c4e7e

I didn't test it without setting anything for LD_LIBRARY_PATH and then waiting for one fallback, this was just with a random (non-library) path.

[mertalev]

Your guess is right. The issue is that reading an unset env from process.env returns undefined, but assigning undefined to process.env casts to a string.