Skip to content

Exp 2026 06 28 Multi Domain Residence

Jump to bottom
Michael Fischer edited this page Jun 28, 2026 · 1 revision

Multi-Domain Residence — the 4 GiB IOVA cap is PER-DOMAIN, not per-device

The headline: the RK3588 NPU's 32-bit IOVA limit is per iommu_domain_id, not per device. The rknpu driver exposes up to 8 domains, each with its own ~4 GiB IOVA space, so up to ~32 GiB of weights can be NPU-resident at once (ceiling = physical RAM, not IOVA). This is reachable from our open uABI (a field already in rknpu_ioctl.h), needs no kernel patch and no librknnrt, and is bit-exact. It overturns the prior "only ~4 GiB DMA-mappable → must stream big models" conclusion.

The finding came out of a three-step investigation (arbiter → dump → de-risk). Commits: ork-driver 8bc4004 (tools/domain_probe.c + ORK_IOMMU_DOMAIN knob), all local. Board 10.3.0.236, RK3588, rknpu v0.9.8, governors max.

Step 1 — The arbiter: does the vendor NPU beat CPU on a streamed >4 GiB model?

Converted Qwen2.5-7B-Instruct (dense) → w8a8 .rkllm = 7.65 GiB (>4 GiB → must use the 4 GiB window) on an x86_64 Rosetta-Docker container, benched on the SBC: librkllmrt (vendor closed runtime) vs llama.cpp CPU, same model.

streamed 7B w8a8, warm prefill (tok/s) decode (tok/s)
librkllmrt (NPU) 73.6 4.05
CPU (llama.cpp Q8_0, -t4) 15.7 3.42

librkllmrt-on-NPU beats CPU on a streamed >4 GiB model — prefill 4.7×, decode 1.18×. This is the existence proof: the RK3588 NPU can run a big model and beat CPU, so our open stack's prior streamed-7B loss (prefill ~2.74) was an implementation gap, not a hardware ceiling. (Decode is only 1.18× — both are ~26–31 GB/s, the LPDDR bandwidth ceiling; that's the regime model: NPU crushes compute-bound M>1 prefill, marginal on bandwidth-bound M=1 decode.)

Step 2 — The dump: how librkllmrt streams (LD_PRELOAD ioctl capture)

Captured librkllmrt's ioctl/regcmd stream during a steady-state run. The surprise: it doesn't stream at all — it resides the whole model across two IOMMU domains.

  • 8 MEM_CREATE / 8 MEM_DESTROY total for the entire run — zero per-token buffer churn. (Our open stack did ~100–170 MEM_CREATE/MEM_DESTROY per token — the MEM_DESTROY ~0.5–2 ms churn was the whole gap.)
  • 8 buffers across 2 domains: domain 0 = 3.83 GiB (weights 3.806 + regcmd + I/O), domain 1 = 2.98 GiB (weights 2.936 + regcmd + I/O) → 6.585 GiB of int8 weights resident + mapped simultaneously (> 4 GiB).
  • Weights uploaded once at load (a single TO_DEV sync of each whole weight buffer, never repeated); regcmds are static (weight base-IOVAs baked in, buffers never move) → the NPU reads weights in place from resident DMA. Per-token syncs are KB-scale on the I/O/KV buffer only.
  • 6–9 matmuls chained per SUBMIT (a fused layer block, regcfg_amount=108); 3 cores round-robin core_mask 0x1/0x2/0x4; lower layers → domain-0 weights, upper layers → domain-1.
  • All buffers plain cacheable IOMMU-mapped NON_CONTIGUOUS (0x403); no SRAM, no write-combine, no DMA32.

Step 3 — The de-risk: does multi-domain residence work from our uABI? (CONFIRMED)

domains total resident per-domain
1 (baseline) 4.23 GB then EFAULT matches dma_probe
2 8.46 GB 4.23 + 4.23
8 32.28 GB (no EFAULT) 8 × 4.03 GB, capped by the probe's 30 GB limit
  • The mechanism was already in rknpu_ioctl.h, unused: rknpu_mem_create.iommu_domain_id (places a buffer in a domain) + rknpu_submit.iommu_domain_id (selects the active domain for a submit). Setting the field is sufficient — no action needed (RKNPU_SET/GET_IOMMU_DOMAIN_ID = 25/24 exist and log switch iommu domain 0→1, but field-only works). Each domain = an independent 32-bit IOVA space (first IOVA ~0xfc000000 each).
  • Domain-1 buffers are NPU-submittable and bit-exact: the entire test_matmul suite (chained / streamed / M-tiled int8, NEON pack, NF4, imatrix, dump/load roundtrip) routed through domain 1 → all mism=0, identical to the domain-0 control.
  • 8 domains confirmed → 2 ≈ 8 GiB, 8 ≈ 32 GiB residable; ceiling is now physical RAM.
  • Caveat: a single SUBMIT reads one domain, so a layer's weights must be co-located in one domain (partition layers by domain, like librkllmrt). A single matmul spanning two domains is not proven.

Verdict

✅ The 4 GiB IOVA cap is per-domain. Models up to ~physical RAM (≈32 GiB int8, far more int4) can be NPU-resident with zero per-token churn — no streaming required. This is reachable from the open uABI (a field we'd never set), no kernel patch, no blob, bit-exact. dma_probe's EFAULT at 4.03 GiB was the per-domain cap, hit because it used one domain. The "must stream" era is over for any model that fits RAM.

Recipe (kill churn + escape the cap): per domain (≤~3.8 GiB), one resident weights buffer (0x403, packed with that domain's layers) + one static regcmd buffer (resident weight IOVAs baked in) + one I/O/KV buffer; one MEM_CREATE + one TO_DEV sync each, never freed. Split the model across 2+ iommu_domain_ids. Per token: zero MEM_CREATE/DESTROY — chained submits vs fixed regcmd + resident IOVAs, sync only the small activation/KV ranges. Chain 6–9 matmuls per submit (run_chain_i8).

Where the open stack can then BEAT the vendor: (a) int4/NF4 storage ≈ 2× decode — decode is weight-read-bandwidth-bound (~26.7 GB/s ≈ LPDDR ceiling); librkllmrt streams full int8, halving the bytes ~doubles the ceiling (needs a coherent int4 — the Hadamard track). (b) SRAM/NBUF tier — librkllmrt uses none; on-chip SRAM for hot scratch is untapped. (c) async cross-core dispatch out-pipelines its serial round-robin. (d) submit fusion (match 6–9/submit via run_chain_i8).

Provenance (clean-room, open): the multi-domain trick was learned by RE-ing librkllmrt's ioctl pattern; we call the open rknpu uABI field ourselves and synthesize our own regcmd — no proprietary code, no kernel module. Runs on the vendor BSP kernel + the (GPL, source-available) rknpu driver, same substrate the closed runtime uses.

Next (in flight): end-to-end 7B 2-domain residence in ggml-ork (target librkllmrt parity 73.6/4.05, zero churn), then the 27B scale test (RAM is now the binding constraint — int8 27B ≈ 27 GB is at the edge of 32 GB; int4 ≈ 13.5 GB fits but needs native-int4). Results to be appended.

A note on the 7B prefill "loss" that triggered this

The open stack's earlier streamed-7B numbers (prefill 2.74, decode churn) were a broken measurement, profiled here: on aarch64, llama.cpp loads Q8_0 into the CPU_REPACK buffer (is_host=0), which fails ggml's offload guard and ORK's supports_buft → the NPU got zero matmuls (the "CPU 15.7" baseline was the NPU sitting idle). GGML_NO_REPACK=1 is required to engage it. With the NPU actually running, the matmul is 0.6–6 % of the wall (~190 t/s ceiling); the loss was serial per-matmul submit + activation churn + cold-conversion tax — all implementation. Plus a cache-evict bug (erase vs unmap) was found and fixed.

Clone this wiki locally