[Feature]: MNN for Immich ML — OCR on Rockchip first; optional general backend?

[todorangrg]

I have searched the existing feature requests, both open and closed, to make sure this is not a duplicate request.

• Yes

The feature

I've been aiming to accelerate OCR for Rockchip boards for a while. As I expanded in OCR hw acceleration for Rockchip boards discussion, this is quite cumbersome as NPU (not only rockchip but in general?) cannot really deal with dynamic inputs. Fair amount of engineering needed, and still with corner-cases and reduced quality of output. I've stumbled across "mnn" inference library recently and was intrigued by it's OpenCL / Vulkan support, maturity, activity etc, so I decided to give it a go: it works really well! It worked so well on my rk3588 board, that I started to think if this could also be the "successor" of armnn, or another useful backend even for cpu inference on arm. Below you will find an (agent) structured RFC summarizing my thoughts. Have a look at the performance in the tables below, it's looking really good! (PPOCR-v6_med 0.5s detection, 7.5s recognition on a full A4 page of text). I even went ahead and checked if mnn conversion can benefit from the work you guys were are doing here: ml-models feat/optimized-model-graphs. With a few patches (upstreamable), also attention blocks and gelu get optimized!

PoC branch with working OCR on opencl for rknn builds is here: https://github.com/todorangrg/immich/tree/mnn_ocr_support

Ask

1. Ship OCR on MNN+OpenCL in -rknn now (RKNN stays for fixed CLIP/faces)?
2. Treat MNN as a general optional backend next to ONNX — plausible ArmNN successor for Mali/Vulkan/strong ARM CPU — with users choosing engines per workload?
3. If yes to (2): later fold MNN into ml-models (fused ONNX → .mnn), not a permanent RapidOCR-only side path?

Why now

Rockchip: SigLIP/faces already use RKNN. OCR is still ORT CPU. PPOCR is dynamic; RKNN is a poor fit (multi-input explosion, padding, accuracy corners). On RK3588 Mali, MNN OpenCL ≈ 6× ORT CPU on med PPOCR.
Stance to confirm: NPUs for quasi-static vision; GPU/CPU general runtimes for highly dynamic workloads (OCR).
ArmNN: unmaintained. MNN is maintained, one API for OpenCL / Vulkan / optimized ARM CPU. Pitch is coverage and choice — not beating RKNN/CUDA where they win. (Separate from optionally shipping ArmNN libs for dry-runs.)
ml-models: fused Immich ONNX + per-target adapts (ORT rewrites, RKNN compile). Natural place for ONNX → MNN later. Fused OCR stays dynamic (rec H=48, W symbolic) — RKNN still weak there.

PoC vs optimization path

This PoC branch works as-is: RapidOCR’s official .mnn models, Immich runtime on -rknn (OpenCL default; Vulkan/CPU/off via env). No patches to MNN required. Kernel cache beside models + OpenCL numThread tuning are small and already in the branch; RapidOCR needs the upstream getNumpyData fix for usable rec speed.
Stock ONNX → MNNConvert does not fail on Immich vision/OCR graphs; it often just doesn’t fully optimize (e.g. ONNX Attention must be decomposed; MNN --transformerFuse is LLM/mask-shaped, so SigLIP/PPOCR typically keep Softmax/MatMul). Leveraging ml-models fused ONNX + Immich prepare/fuse-oriented patches is optional upside (~15–20% if fuse lands) — not a blocker for A.

Strawman

Phase	Scope
A (now)	-rknn: OCR → MNN OpenCL; CLIP/faces stay RKNN. RapidOCR .mnn OK.
B (later)	First-class MNN in ml-models from fused ONNX; optional broader -mnn / ArmNN migration.
Prefer A under consensus; design B so A doesn’t freeze RapidOCR-only forever.

Non-goals (v1)

• Replacing RKNN for SigLIP/faces on Rockchip
• Requiring Immich-owned OCR .mnn or Attention-fuse before shipping A
• Claiming current benches use fused MNN Attention

Evidence (RK3588, indicative; unfused MNN graphs)

OCR (A4 text page: ~39 boxes, ~48x1500pix boxes); RapidOCR + getNumpyData:

Model	Backend	box/s	Cold → cached warmup
v6_med	MNN OpenCL	4.6	~600s → ~0.3s
v6_med	MNN CPU (2 core)	1.9	~0
v6_med	MNN Vulkan	1.2	~6s → ~0.3s
v6_med	ONNX CPU (2 core)	0.78	~0
v6_small / tiny	MNN OpenCL	13 / 56	long → ~0.1–0.2s

Static vision (familiar RKNN/ArmNN baselines — stay on RKNN on Rockchip):

Model	Backend	Throughput	Avg inference
ViT-B-32 OpenAI (visual)	MNN OpenCL FP16	11.07 inf/s	90.3 ms
ViT-B-32 OpenAI (visual)	RKNN 1 core	4.3 inf/s	232.6 ms
ViT-B-32 OpenAI (visual)	RKNN 3 cores	13.2 inf/s	75.8 ms
ViT-B-32 OpenAI (visual)	ArmNN	11.8 inf/s	84.7 ms
ViT-B-32 OpenAI (visual)	ArmNN FP16 turbo	15.7 inf/s	63.7 ms
ViT-L-16-SigLIP2-512 WebLI (visual)	MNN OpenCL FP16	0.38 inf/s	2642 ms
ViT-L-16-SigLIP2-512 WebLI (visual)	RKNN 1 core	0.29 inf/s	3500 ms
ViT-L-16-SigLIP2-512 WebLI (visual)	RKNN 3 cores	0.67 inf/s	1500 ms
ViT-L-16-SigLIP2-512 WebLI (visual)	ArmNN	cannot convert?	—

Decisions wanted

1. A yes/no (-rknn OCR via MNN)?
2. B — general MNN backend / ArmNN successor later, or Rockchip-OCR-only?
3. OpenCL default vs CPU default + opt-in GPU (Mali ICD mounts)?
4. Process: Discussion → PR for A, B follow-up?

Related:

Platform

• Server
• Web
• Mobile

[mertalev]

It seems MNN isn't all that fast - it's slower at f16 than ANN at f32. I do actually have RKNPU working for OCR as part of the model optimization work and will have numbers for it soon. The basic idea is to have different shapes (736x736, 736x1472, 1472x736) as different models, using the best model for the input. This is actually how other backends will work as well since it turns out that they're basically all much more compute and memory efficient this way (yes, even accounting for the several models loaded).

[todorangrg]

I guess its kinda expected opencl in a framework that targets many architectures is less efficient than armnn. But armnn is dead, right? Notice even for CPU only, mnn is ~2x faster than onnx. And for OCR, mnn opencl ram usage is tiny: around 270MiB for v6 medium detector + recognizer. And this can be useful for many arm boards, not just rk*?

Looking forward to some benchmarks you got! I implemented that also in a branch via RKNN, but I ran into issues:

• detector is not rotation invariant, so one would need multiple models for landscape, portrait etc (or live with the fact that its only a square shape and having to reduce input resolution)
• for me multiple OCR model sizes were exploding in ram size. Maybe I miss something and your approach is ram efficient. My finding was that weights are a small portion of the OCR models, most of it is working memory. So even with the cpp rknn runtime that shares weights, I was getting still almost 3x ram usage when running on all 3 npu cores. I recall I was in the gigabytes range with the v5 server detector and mobile recognizer. I didn't even dream of server recognizer. For recognizer, also, padding with black at the end was reducing performance more than stretching. But with too much stretching, performance was dropping as well, so a fair amount of bins is required.

My TLDR for running OCR on RKNN via discretized resolutions :

• runs pretty fast
• crazy ram usage
• quality drop almost guaranteed. Never got to a proper quality benchmark, but I think it would be very relevant to quantify it.
• I did not load/unload multiple res models at every inference, I used the rknn "dynamic input" stuff.

Curios what you get 👍

[todorangrg]

Hi @mertalev

I did some more testing, and got inspired to try again RKNN for OCR based on your response. I managed to do some more graph surgery to the recognizer (ignore letterbox padding by feeding it a simple valid/pad mask and wiring that into CTC blanking plus the SVTR attention Softmaxes so pad columns can’t invent characters) and the results improved a fair amount using padding -- and model didn't get significantly slower. This might be useful if implementation goes the discrete inputs route. I still had to use quite a few shapes for reconginzer (17 I believe), and used 5 shapes for detector.

Here are my thoughts / conclusions after doing more through and up-to-date tests:

• all tests are with models based on the model optimization branch
• for OCR, MNN seems to be 6-10x more memory efficient (rknn is approaching 1GiB usage for v6 medium)
• RKNN is 2-3x faster than MNN
• MNN is expected to be worse; from just a raw compute perspective, the rk3588 GPU should be 3-6x slower than the NPU
• tonemapped transcode jobs using jellyfin ffmpeg take prio on GPU. 2 streams (one 4k DV 60hz 60mbit -> 8mbit 720p and one 4k HDR 50mbit -> 40mbit) stayed both in 4-4.5x RTF stable, while the mnn job slowed down appropriately. more numbers about this below.
• if we just think about "in how much time can the machine finish a large family import", MNN makes sense, as it can be thought of as a "4th core" of the compute group, which otherwise doesn't do much (tonemapping from transcodes is not filling it); if models are large, NPU still has to do smart search.
• also, "random boards that have vulkan gpus" is also a nice bonus for MNN
• MNN needs no constraints on resolution limits, bins discretization choices etc. (easy as it's fully dynamic)
• on the other hand, RKNN is just faster; and doing this discrete inputs implementation right probably means being able to support more accelerators (like Hailo which is quite popular due to RPi?)

Anyway, please find below some data:

RK3588 OCR — MNN OpenCL vs RKNN

Isolated single-model benches plus OpenCL under transcode (Immich / movie) contention.

How to read metrics

Column	Meaning
Mean ms	Serial single-job latency. Does not scale with cores.
Mpix/s	Wall pixel throughput = total Mpix finished ÷ wall time (increases with number of cores).

Do not divide mean ms by ~3 for multi-worker; that mixes latency with throughput. Use Mpix/s for scaling.

Highlights

Setup	Result
MNN det @ 736×1312	650 ms · 1.49 Mpix/s · peak 43 MB
RKNN det 1 core / 3 core	539 ms · 1.79 / 4.75 Mpix/s (~2.7×) · peak 255 / 595 MB
MNN fused rec @ W=1600	193 ms · 0.40 Mpix/s · peak 38 MB
RKNN rec 1 core / 3 core	260 ms · 0.30 / 0.74 Mpix/s (~2.5×) · peak 203 / 283 MB

Isolated detection — 736×1312 NHWC (0.966 Mpix)

Backend	Cores	Mean ms	Mpix/s	Create MB	Steady MB	Peak MB
MNN OpenCL (cache ready)	—	650	1.49	42	39	43
RKNN dynamic	1	539	1.79	146	193	255
RKNN dynamic	3	542	4.75	399	453	595

• 3-core throughput ~2.7× vs 1 core (not 3×); serial mean stays ~540 ms.
• RKNN uses roughly 6–14× more memory than MNN.

Isolated recognition — H=48, W≈1600

~0.077–0.078 Mpix/crop @ 48×1600/1632.

Backend	Cores	Mean ms	Mpix/s	Create MB	Steady MB	Peak MB
MNN fused (cache ready)	—	193	0.40	38	34	38
RKNN static 48×1600	1	258	0.30	57	74	103
RKNN static 48×1600	3	258	0.68	104	134	203
RKNN dynamic 48×1632	1	258	0.30	96	99	132
RKNN dynamic 48×1632	3	260	0.74	199	203	283

• Single-shot: MNN ~1.3× RKNN (0.40 vs 0.30 Mpix/s).
• 3-core RKNN ~2.3–2.5× vs 1 core.
• Dynamic ≈ static latency; ~1.7–2× RAM vs static.

OpenCL during (tonemapping) transcode jobs

det_medium 736×1472 + rec_medium W=2000 on Mali (MNN) while HW video ran. Solo OCR: 753.7 / 227.6 ms.

Streams (both stayed healthy — OCR took the hit, not the encode)

Stream	What	Approx bitrate path	Under OCR + video load
Immich transcode job	Dolby Vision ~60 Hz	~60 Mbit → 8 Mbit 720p	Stayed 4–4.6× realtime; no meaningful bulge
Movie playback / encode	HDR	~60 Mbit → ~40 Mbit 1080p	Same — encode held; OCR slowed

OCR slowdown

Load	det 736×1472	vs solo	rec W=2000	vs solo
Solo	753.7 ms	1.0×	227.6 ms	1.0×
det ∥ rec only	1397.7 ms	1.9×	404.4 ms	1.8×
+ Immich transcode	1778.8 ms	2.4×	522.7 ms	2.3×
+ movie + Immich	2296.2 ms	3.0×	640.5 ms	2.8×

Import-relevant: +Immich transcode ≈ 2.3–2.4× OCR slowdown. Immich transcode + movie night ≈ 2.8–3.0× on OCR while both streams kept >4× realtime.

[mertalev]

I tried to get the CTC head in as well and actually had it inside originally, but it's just too slow. Half the time went to that while it was much faster on host. The unfortunate thing is that softmax is done on host because the RKNPU softmax op is inaccurate enough to be broken and every decomposition I tried was slow and ran on CPU.

[todorangrg]

Good news — the CTC head now runs fully on NPU! (And I think it's quite accurate, no softmax op)

Recognition

Config	w160	w320	w640	w1280
feat/ocr baseline	9.83	17.66	34.83	70.35
1 core – no CTC head	7.2 / 18 → 23	10.7 / 22 → 31	22.8 / 28 → 45	47.21 / 39.5 → 73.7
1 core – graph CTC	10.7 / 19.2 → 19.5	20.0 / 21.8 → 22.1	32.2 / 25.2 → 25.6	60.08 / 32.0 → 32.6
3 core – graph CTC	11.8 / 25.3 → 26.1	21.6 / 32.5 → 33.4	37.8 / 42.6 → 43.7	74.0 / 62.8 → 64.9

All cells are ms / MiB → MiB peak, except the baseline row, which is ms only. For 3 core, time is per core (throughput is triple in that time)

While doing CTC on the host might technically be ~5ms faster for w640, the memory transfer is significantly less, there's no CPU load for that op, and RAM usage is significantly lower with the in-graph CTC!

Also, it seems the CTC head scales with classes, not width (at least across these widths) — so it's relatively speaking slower for very small widths (does that matter?) but faster for large widths.

Wanted to ask: did you time only the model, or did you time with host CTC included too?

[mertalev]

Yes, I tested with host decode time included. The decode is pretty cheap; the softmax is the only thing that costs any notable time and even that can be optimized by applying it only to the logits kept after decode.

To be honest, I think it's kind of a waste of the NPU to make recognition much slower just to get the decode into the graph. In-graph decode is a big win on the 4090 and 9070 XT I tested mainly because of the PCIe transfer, not because it's expensive compute. This doesn't apply for the NPU because everything is already DDR memory anyway aside from a tiny 6KB SRAM for individual operations.

[todorangrg]

But is it much slower? I need to test that again to be sure, but I remember w640 host decode is around 5ms. From the table above, w1280, single core (apples to apples) we have 47+10(double width)=57ms, vs 60ms in-graph. To me, the memory issue is copies: getting that output zero-copy from rknn pool all the way to the Python CTC decode requires a buffer with more slots (or just copy but that also increases ram usage). Thus, memory starts really ramping up. This is amplified with larger widths and/or 3 core usage (which unlocks the big speed gains). I will bench some bigger widths and share results, as I will look more in detail into RKNN dynamic inputs next.

[mertalev]

That's fair, I didn't compare RAM usage. If you can share your graph surgery code, I'd be interested to look at it and compare more closely.

[todorangrg]

hi @mertalev . Took me a while to come back as I went into introducing rknn dynamic input and ... this went sideways. I think now I have systematic enough data to share. Here is the branch: https://github.com/todorangrg/ml-models/tree/feat/ocr_improvements. Long story short: while the surgery that makes in-graph CTC be fast is cool, it does not work for dynamic shapes. RKNN compiler just chokes on it. There is however an improved formulation compared to the base branch, which increases performance substantially and works with dynamic shapes. How RKNN packs / reshapes is quite cryptic, and it needed a fair amount of tests to figure out optimal solution.

Detector speed is just slightly improved, but RAM scaling with dynamic shapes is much more sane. For recognizer, I think performance improvement is substantial (and RAM scaling as well). I don't think in-graph CTC is worth it just because of the static shape / RAM issue. Note that I used only few bins, but with dynamic shapes we can add significantly more without impacting inference speed nor (much) memory - so we could do 1:1, 3:2, 2:3, 4:3, 3:4, 16:9, 9:16 and speed up inference (reduce padding) without much extra memory.

Below is the (generated) report:

OCR detector on RKNPU: reference vs. optimized graph

RK3588, rknn-toolkit2 2.3.2, PP-OCRv5_mobile + PP-OCRv6_medium, 3 canvases (square, wide, tall).

TL;DR

• Memory: unambiguous win, 52–59% less resident memory, dynamic vs. static, both models, both core counts.
• Speed: dynamic wins at every canvas, both models.

Setup

• ref — first commit on branch (08f1a22), static, one binary/canvas.
• optimized — current tip (accumulated graph rewrites), one dynamic binary covering all 3 canvases.

1. Speed and 3-core scaling

Model	Canvas	ref	optimized
v5_mobile	square	83.2	78.1
v5_mobile	wide	160.3	146.1
v5_mobile	tall	153.2	148.3
v6_medium	square	257.1	253.6
v6_medium	wide	531.0	516.2
v6_medium	tall	505.9	506.9

optimized is faster than ref at every canvas, on both models. 3-core scaling: both configs, both models, 2.2–2.65x, flat across every canvas — no ceiling, since detection has zero host-side decode (unlike the recognizer's ref), so nothing sits outside the pool's parallelism.

2. Memory (MB, 1-core and 3-core — N static binaries summed vs. 1 dynamic binary peak)

Roughly equal savings across both models (unlike the recognizer, where the edge is backbone-size-dependent) — here it's just "3 binaries vs. 1," independent of backbone.

Bottom line

optimized/dynamic is an unambiguous win — memory and speed, both models, no exceptions.

OCR recognizer on RKNPU: CTC head placement

RK3588, rknn-toolkit2 2.3.2, PP-OCRv5_mobile + PP-OCRv6_medium, width ladder 320–2560px.

TL;DR

• baked (CTC decode on-NPU): fastest per-request, especially at wide crops. Static-only — no dynamic-binary option.
• no-bake+transpose: near-parity with baked on memory-constrained deployments, no bin-count memory tax. Near-parity with ref too on v6_medium through most of the ladder, decisive win at the wide end.
• 3-core pool > CTC variant choice. Get that right first (2.1–2.6x for baked/no-bake+transpose, every width). ref alone degrades at wide crops (host decode isn't parallelized).

The three variants

Same fused ONNX (backbone + SVTR head). Differ only in where CTC decode (argmax over class axis) happens:

	Decode	Backend
ref (on branch today)	host, class-major logits [B,C,S]	static
baked	on-NPU (packed MaxPool trick)	static only — bakes primary canvas's seq length into the graph, doesn't survive dynamic_input
no-bake+transpose	host, but on-NPU transpose to seq-major first [B,S,1,C]	dynamic (1 binary, whole ladder)

1. Speed and 3-core scaling (mean_ms; bar = 1-core, hatched = 3-core with its speedup labeled)

Model	Width	ref	baked	no-bake+transpose
v5_mobile	320	19.5	16.1	16.8
v5_mobile	640	30.3	33.8	29.2
v5_mobile	1280	69.6	62.4	65.6
v5_mobile	2560	206.3	120.8	152.8
v6_medium	320	27.3	29.3	26.2
v6_medium	640	51.1	49.3	53.4
v6_medium	1280	103.7	105.1	104.6
v6_medium	2560	280.5	206.6	224.8

• baked wins at wide crops (26–41% under ref), roughly ties mid-ladder.
• no-bake+transpose beats ref at every width on v5_mobile (−4% to −26%). On v6_medium: near-parity through 1280px (±4%, within run-to-run noise), decisive win at 2560 (−20%).
• 3-core scaling: baked 2.19–2.62x, no-bake+transpose 2.12–2.48x — both healthy, flat across the ladder. ref degrades at wide crops (1.41–1.64x at 2560) — its host decode (visible as the lighter segment atop each ref bar, growing with width) sits outside the pool's parallelism and increasingly dominates wall-clock.

2. Memory (MB, 1-core and 3-core; N static binaries summed vs. 1 dynamic binary peak)

Dynamic packing's edge is "shared backbone weights once, not N times" — a fixed MB saving that matters more on a bigger backbone. On v5_mobile (small backbone) it's mostly wiped out by per-worker pool overhead at 3 cores; on v6_medium it holds.

Bottom line

• Small/fixed bin set, latency-critical → baked.
• Wide bin ladder or bigger backbone, memory-constrained → no-bake+transpose — a clean call on v6_medium too.
• Get the 3-core pool running before micro-optimizing CTC placement — it's the bigger lever, and ref's degradation there is its own separate case for moving off it.

[todorangrg]

I compile the models like this:

uv run --frozen --extra export immich-model onnx export PP-OCRv5_mobile paddle --no-cache --no-bake-ctc-head

uv run --frozen --extra rknn immich-model rknn compile PP-OCRv5_mobile --dynamic --transpose-logits-output

Keep in mind the recognizer outputs in [B, S, 1, C] format, arg max should be over dim 3.

[mertalev]

Thanks for sharing! The Conv fusion change in particular is very nice since it actually improves every backend. The affine fold helps RKNN but is negligible for other backends, but it doesn't hurt either. I think both of those can just be part of the base export without making it conditional. The tiled transpose is a great find as well. It stacks nicely with switching to fp16 outputs, which is another change that benefits every backend.

One thing to note: setting more NPU cores mainly improves throughput if they're otherwise sitting idle. In reality, other tasks like facial recognition are also using the NPU. It probably still helps make utilization more consistent, but I doubt the increase is as linear as an isolated benchmark would suggest.

[todorangrg]

Glad they improve other backends as well - I honestly did zero tests outside rknn, but I assumed they should help.

Yes on the core count! For day-to-day incremental indexing it doesn't matter. But for large imports, unless you're "lucky" with the model runtimes and workload (smart search inference time, OCR detector inference time, avg. number of text boxes and their size / img), total indexing time will still be slower with single core pools. I see it analogous with core pinning on the CPU: if user knows what they are doing, its going to be effective. The "don't know, don't care I want fast" solution is run with 3 for every model. For OCR this memory scaling is by far the worst (~600 extra MB ram for 3 core for v6 medium), for the other models the memory overhead is much much smaller (working memory is negligible compared to weights - which are shared).

So, what's the next step? Anything I can do to conclude this? Should I make the rewrites active for every backend and open a PR?

PS: funny we are having this conversation under the MNN GPU discussion. After optimizing rknn, seems MNN GPU really is behaving like a "weaker NPU efficiency core" : for both detection and recognition it is somewhat slower than one npu core. I guess that kills it.

[mertalev]

I actually plucked the changes from your branch to mine as base export passes, so no change needed on your end. I'm just doing some final cross-backend testing and will have a PR up for the branch.

Yeah, I think MNN just isn't competitive unfortunately, especially with RKNN's dynamic inputs in the mix. It's a bit ironic because RKNN's dynamic shape support is actually better than several of the GPU backends, which effectively require separate models that share nothing.