feat(ml): ARMNN acceleration

[fyfrey]

a very long journey to find a way to use ARMNN (on ARM Mali GPUs) to offload and accelerate Immich ML models (CLIP vision, face detection & face recognition)

code overview

• new native library libann that provides c/python (ANN) access to C++ ARMNN
• libann is wrapped as a ONNX-session like object allowing it to be a drop-in-replacement for all compatible ONNX models (see the minimal code changes to clip.py, facial_recognition.py)
• modified docker image for ARM to always include ARMNN+ANN
• added a new exporter that can produce .armnn model files for CLIP vision, arcface & retinaface (cannot be integrated into existing exporter as it needs pytorch==1.x) suggested workflow: iterate over compatible models already on Immich huggignface, export armnn models, upload/add to existing HF model
• compatible armnn models need to be uploaded to existing Immich huggingface models
• ANN is automatically selected at runtime if 1. it is enabled in the config (default true), 2. the required libraries are mapped into the container and 3. the model has not only .onnx but also .armnn file

benchmarks on an Orange Pi 5 (RK3588 SOC)

"ENCODE CLIP" + "RECOGNIZE FACES" job for 3000 images, concurrency 4

ARMNN (fp16):
ram 680mb
225% CPU
total time for 3000 images: 3 minutes
SoC temp: 60.1°C

ONNX (fp32):
ram 980mb
745% CPU
total time for 3000 images: 45 minutes
SoC temp: 81.3°C

how has this been tested

• build on x86_64 and ARM64
• manually added the armnn models into the model-cache folder
• checked that ONNX models still work (and produce sensible results)
• cheked that ARMNN models work (produce sensible results)
• performed above benchmark

[cloudflare-pages]

Deploying with    Cloudflare Pages

Latest commit:	9f64cd3
Status:	✅  Deploy successful!
Preview URL:	https://f0d81e3b.immich.pages.dev
Branch Preview URL:	https://ml-armnn.immich.pages.dev

View logs

[mertalev]

This is really cool! It's good aside from some code style comments.

[fyfrey]

This is really cool! It's good aside from some code style comments.

Thanks! Also for the good review. Only remaining problem is the test / mock failure. Not sure how to make the mocking work again.

[mertalev]

LGTM