Repository navigation
Exp 2026 06 27 Zero Copy Weight Import
Commit: ork-driver 57feca4 (v0.4.0) · API ork_dma_import/ork_dma_import_sync/ork_dma_import_free + ork_mm_load_i8_import · uABI src/rknpu_ioctl.h (DRM PRIME fd↔handle, dma-heap alloc, dma_buf_sync) · test tools/zerocopy_import_test.c. RK3588 10.3.0.236, governors pinned.
The closed RKNN runtime imports externally-allocated memory zero-copy via rknn_create_mem_from_fd/_from_phys. An LD_PRELOAD strace showed this is not a new ioctl — it's an unused input mode of the MEM_CREATE we already had: DRM_IOCTL_PRIME_FD_TO_HANDLE (generic DRM, import a dma-buf fd → GEM handle) then RKNPU_MEM_CREATE{handle≠0, flags=0, size=0} → the kernel IOMMU-maps the existing pages, no alloc, no copy. Built it into ork-driver: a dma-heap buffer filled with pre-tiled weights and imported as a matmul tensor is read directly by the NPU, bit-exact vs the alloc+copy path, make test fully green. Weight load is 2.2–4.4× faster (the host memcpy is gone), and a fixed-uniform-size-slot streaming pool cycled a 4295 MB working set (>4 GiB) through one slot, 1024 swaps, 0 fails / 0 mismatches — the earlier streaming wall was variable-size MEM_CREATE/DESTROY churn, not the swap itself. It does NOT escape the 4 GiB IOVA cap (the imported dma_addr is still a 32-bit 0xff……) and does not change M=1 decode bandwidth — the win is the eliminated copy in the M>1 streaming swap/load path.
Importing a /dev/dma_heap/system dma-buf as a matmul B (weight) tensor, the decoded syscall sequence:
PRIME_FD_TO_HANDLE in_fd=8 → handle=6 (generic DRM 0x2e: import dma-buf → GEM handle)
MEM_CREATE in{handle=6, flags=0x0, size=0} → out{handle=6, dma_addr=0xff400000}
vs an internal allocation (rknn_create_mem / ork's existing path):
MEM_CREATE in{handle=0, flags=0x403, size=4194304} → out{handle=3, dma_addr=0xffbfe000}
So RKNPU_MEM_CREATE (nr 0x42) is overloaded: handle==0 ⇒ allocate + DMA-map (what ork-driver did); handle!=0 (from PRIME_FD_TO_HANDLE) ⇒ map the pre-existing dma-buf's pages, no allocation, no copy. The prior RE recorded MEM_CREATE as alloc-only and missed the import-by-handle mode. Zero new driver-private opcodes — only the generic DRM PRIME ioctls (0x2e/0x2d) and the handle!=0 input.
RKNPU_MEM_SYNC does not flush a foreign imported cacheable dma-buf — it only covers buffers rknpu itself allocated. On the cacheable system heap that gave all-wrong output (CPU dirty lines never reached the NPU); system-uncached was correct but ~5× slower to fill. The fix is the dma-buf's own DMA_BUF_IOCTL_SYNC (START|WRITE / END|WRITE bracket around the CPU fill) — then the cacheable heap is correct and fast. This is the import analogue of the output zero-copy coherency fix (3fad74a): the NPU and CPU share the pages, so the CPU's caches must be reconciled around the device read.
Load time (alloc+copy → import), bit-exact:
| weight shape (int8) | speedup |
|---|---|
| K512 × N256 | 4.4× |
| K2048 × N512 | 2.2× |
| K4096 × N1024 | 2.6× |
Fixed-slot streaming (16 MB experts, load_import → run → free, round-robin):
| per-swap cost | ms |
|---|---|
| alloc + copy | 53.2 |
| import | 25.2 (2.1×) |
The fragmentation test (the earlier wall): cycle a 4295 MB working set (>4 GiB cumulative IOVA churn), 256 distinct experts × 256 iters = 1024 swaps through a single uniform-size slot → 0 load-fails, 0 output mismatches on both the import and alloc+copy paths. Uniform fixed-slot reuse does not fragment the IOVA — the earlier streaming / MoE-persist cost was variable-size alloc/free churn, and import additionally halves the swap cost on top.
make test stayed fully green (matmul, W4A4, layer, decode, model, speed, chain) with the new API in tree.
✅ Zero-copy weight import is real and buildable on the existing uABI (the handle!=0 MEM_CREATE mode + generic PRIME), and a uniform fixed-slot pool makes streaming crash-free past 4 GiB cumulative churn. The win is concrete where M>1 streaming spends its time: load 2.2–4.4×, swap 2.1×, no copy. dma_addr is still 32-bit IOVA (rknn_tensor_mem.phys_addr is hidden, size is uint32_t), so big models still stream; the "4 GiB accelerator, not UMA" model holds. And it does not touch M=1 decode (bandwidth-bound). NEXT: wire ork_mm_load_i8_import into the ggml-ork streaming path (the 27B/35B verify / MoE) for the end-to-end effect — does cheaper, crash-free streaming move the M>1 prefill/verify number?