[Feature] Support Hailo for machine learning hardware acceleration

[arthurmcgregor]

I have searched the existing feature requests, both open and closed, to make sure this is not a duplicate request.

• Yes

The feature

I run Immich on a Raspberry Pi 5 and would like to connect a Raspberry Pi AI Hat to improve machine learning performance. From what I understand, this would require supporting the Hailo modules for hardware acceleration. I suspect the Raspberry Pi 5 is a common entrypoint for people starting on their self-hosting journey (like myself!) so supporting the RPi AI hardware would be valuable.

The relevant hardware is currently:

• AI HAT with "13 and 26 tera-operations per second (TOPS) variants, built around the Hailo-8L and Hailo-8 neural network inference accelerators"
• AI Kit with "13 tera-operations per second (TOPS) neural network inference accelerator built around the Hailo-8L chip."

There was a related discussion here but that discussion is closed and the Hailo part was a side-discussion, so opening a new one to see if there has been any progress since then.

Thanks, in anticipation 💚

Platform

• Server
• Web
• Mobile

[bierwald]

I would appreciate implementing the Hailo AI HAT/Kit. This would enable a lot of small installations machine learning capabilities at a moderate invest.

Implementing this seems pretty obvious to me.

[mwesten]

According to the Product sheet it should be compatible with TensorFlow, TensorFlow Lite, Keras, PyTorch and ONNX.

[mertalev]

There were some compatibility issues when compiling our model catalog to Hailo format when I last played around with this, but that was almost a year ago. They might have better OOTB support for these models these days.

[zytegalaxy]

@mertalev I am new this world but I think I can figure out the acceleration part but I know very little about what models immich use and how this can be easily integrated to run with immich machine learning docker. It would be great if you offer some pointers/insights on the work you did with hailo last time and other suggestions.

I think hailo would give ML ops a significant boost compared to Mini PCs (at least 100x speed-up). This way maybe you can crack up indexing and use it to get as much context possible from each image, without worrying much about available resources. This would allow users to do semantic search.

This way you can also do things other products can't or won't because they have to pay for cloud compute resource but since immich users run on their own devices, they don't care about their electricity bill going up which is very insignificant financially for individual users.

[mzdrahal]

@mertalev happy to take a stab at it if you can share more.

[mertalev]

The basic flow is to export the models to ONNX (repo) before exporting to other formats. The Hailo format would be an additional target format to this CLI, similar to how RKNN is supported. For specific steps on the ONNX -> HEF conversion, I would defer to their online documentation.

[gdotmueller]

@mertalev there are ready made conversions for HAILO8 and HAILO8L for similar models. its more about writing code to support inference for those. further down in the discussion someone found a repo on github who sucessfully created a ML container for immich which uses HAILO models, although the implementation of inference is partly buggy.

[mbay0r]

Hailo 8 support would be nice, using a Hailo 8 pcie Card.

[menn0]

I little bump now the raspberry AI hat v2 has been released... Any possible support for the Hailo-10H?

[webwicz]

I recently found a repo, updated it and I have it working on the raspi5 with the v1 Hailo hat with Immich 2.7.5.
immich-ml-hailo

[pmalek]

Looks really cool!

Can you share some speed comparisons between running immich native ML models on RPi 5 and those running on Hailo?

[gdotmueller]

@pmalek I benchmarked this using onnxruntime on the Pi 5 CPU vs the same models compiled to HEF running on the Hailo-8L — not the native Immich ML worker directly, but the same SCRFD + ArcFace weights so it's a fair apples-to-apples comparison of the compute.

With the best-accuracy config (scrfd_10g + arcface_r50): 669ms on CPU vs 99ms on Hailo — 6.7× faster. With the lightweight combo (scrfd_2.5g + arcface_mobilefacenet) it's 194ms vs 42ms, so around 4.6×. The bigger the model, the larger the Hailo advantage — scrfd_10g alone drops from ~410ms to ~50ms on the chip.

[gdotmueller]

@webwicz not sure if you noticed but there is a severe bug in the repo you found! the face recognition is not done correctly! i was getting really bad distance scores and dug in. since i am dealing with simialr models at work i fixed the problem and now it seems to work flawless. if i have time i may do a PR or a new fork. But dont use the repo as is, its vibe coded and there are some architectural problems. the landmarks detected are not aligned to the 112x112 image used for recognition. so no warping happening which results in completely wrong face recognition

[gdotmueller]

I only checked face detection/face recognition yet. OCR seems to work fine. clip search i have not analyzed yet to see if implementation is correct

[oksibutch]

@gdotmueller Won't you post your corrected version? I think it would be useful for the community.

[wormoworm]

Seconded, would be very useful indeed!