Skip to content

SBC Performance Optimization Roadmap

Jump to bottom
Michael Fischer edited this page Aug 19, 2026 · 14 revisions

Rockchip SBC Performance Optimization Roadmap

Update (2026-06-24): (1) Compare only within a path. The 178/12.8 (and closed 184/11.71) figures are the w8a8 .rkllm path; GGUF Q8_0 on this stack ceilings at ~83 prefill / ~4 decode (bottlenecked in ggml-ork/CPU, not ork-driver). Comparing GGUF-Q8 to the w8a8/closed numbers is apples-to-oranges — see Quantization Paths & Baselines. (2) int4/W4A4 is not yet viable on real models (open research track, not a near-term win; w8a8 is production). (3) A co-resident server tanking decode is CPU big-core scheduling contention, not the NPU — see CPU Big-Core Contention.

Update (2026-06-26) — MoE decode partition (ggml-ork side) + a corrected verdict. Wired the decode partition into ggml-ork.cpp's MUL_MAT_ID int8 handler (commit edf5e855b): backbone pinned + a static hot-expert cache (top-K/tensor, LRU under ORK_MOE_HOT_GIB) resident on the NPU, cold experts on CPU. This also fixed the live-MoE wedge — it was over-residency hitting the 4 GiB IOVA cap (MEM_CREATE EFAULT), not a regcmd bug. Cold experts now reuse ggml's own vec_dot (asimddp/SDOT) kernel instead of a scalar+f32-dequant loop (commit a72692188). A "~21 ms/expert NPU submit floor" claim was adversarially DISPROVEN: the isolated resident per-expert M=1 submit is 54–67 µs via run_stream (cross-core), which beats CPU — the apparent floor was residency thrash (MEM_CREATE/DESTROY churn on the single-stream NPU) + live-pack + dead-code profiling. So MoE-on-NPU decode is back in play; the path is persist experts to .orkpack + switch the partition to run_stream_i8 + size the cache to the IOVA budget, then re-bench. ★ Update (2026-06-27): that end-to-end M2 re-bench is in — and on RK3588 it LOSES (decode 6.56 vs CPU 19.09 t/s, ~2.9×). The micro-win doesn't survive M=1 because usable IOVA is only ~1.2 GiB (GGUF backbone eats ~2.8 of 4 → ~17% hit), the cold-expert per-GEMV is memory-bandwidth-bound (loses to ggml's fused batched MUL_MAT_ID), and ork_mm_load_i8 is 5.9 ms/load. The lever is M>1 (batched verify / prefill), not residency. M2 is kept as an experimental, default-OFF setting (ORK_MOE_NPU), wired into oRKLLM with an RK3588 "not recommended" disclaimer; revisit on wider-IOVA / DDR5 hardware or the M>1 path. Full analysis: ork-driver wiki — MoE Decode Partition + Submit Floor.

This document outlines the architectural bottlenecks, memory bandwidth dynamics, and strategic priorities to maximize text generation speed on Rockchip RK3588 single-board computers (SBCs). It is based on empirical profiling of the open-source llama.cpp + ggml-ork (ork-driver) stack and direct comparisons against Rockchip's proprietary RKLLM SDK.

1. Executive Summary & Hardware Physics

Maximizing performance on the Radxa ROCK 5B (RK3588) requires aligning our execution units with the physical constraints of its memory architecture. The RK3588 features a 64-bit LPDDR4X memory bus clocked at 2112 MHz, yielding a theoretical peak memory bandwidth of 33.8 GB/s (shared by the CPU, NPU, and GPU).

Our benchmarks reveal that LLM text generation is split into two entirely distinct physical regimes:

graph TD
    A["Input Prompt"] --> B["Prefill Phase (M > 1)"]
    B -->|High Arithmetic Intensity| C["Compute-Bound NPU"]
    C -->|ORK_QUANT=4| D["INT4 Silicon Peak: 12 TOPS"]
    
    A --> E["Decode Phase (M = 1)"]
    E -->|Low Arithmetic Intensity| F["Memory-Bandwidth & Latency Bound"]
    F -->|Single-Core NPU| G["Latency-Bound (18% RAM BW)"]
    F -->|3-Core + Full-K| H["Bandwidth-Bound (48% RAM BW)"]
    F -->|Mali GPU OpenCL| I["Driver/API Overhead (0.94 ms/matmul)"]
Loading
  1. Prefill (Compute-Bound, $M \gg 1$): High arithmetic intensity (matrix-matrix multiplication) means weights are reused across many input tokens. Performance is limited by raw processing power (6 TOPS at INT8, 12 TOPS at INT4 on the NPU).
  2. Decode (Bandwidth-Bound, $M = 1$): Low arithmetic intensity (matrix-vector multiplication) means the entire weight matrix must be streamed from RAM to process a single token. Performance is bounded by how quickly we can saturate the 33.8 GB/s LPDDR4X bus.

2. Bandwidth Saturation Profiles & The Mali GPU Fallacy

Earlier optimization plans hypothesized that compiling custom OpenCL/Vulkan kernels for the Mali-G610 GPU would yield a major decode speedup. Real-world profiling and physical bus testing have retracted and corrected this hypothesis:

Unit Profiling under Decode ($M=1$, Qwen3-1.7B-w8a8)

Execution Unit Decode Speed Avg RAM BW Util Peak RAM BW Util Primary Bottleneck
CPU NEON (4 Cores) 11.50 tok/s (Optimized) ~16.00% (5.40 GB/s) ~48% Bandwidth-Bound (Saturates LPDDR4X bus, matching official closed-source RKLLM parity)
Single-Core NPU 5.00 tok/s ~18.00% (6.08 GB/s) ~42% Latency-Bound (Driver ioctl submit floor & IRQ wait)
3-Core NPU (K-Split) 8.60 tok/s ~16.00% (5.41 GB/s) ~88% Overhead-Bound (Multiple submits per matmul)
3-Core NPU + Full-K 11.00 tok/s 48.00% (16.22 GB/s) 83% Bandwidth-Bound (Hardware saturation limit)
Mali-G610 GPU (OpenCL) Non-Viable — — API Overhead-Bound (High kernel launch & dispatch cost)

Why the Mali GPU is Not a Simple Drop-In (And Its True Strategic Role)

  1. Shared Bus Limitation: Since the CPU, NPU, and GPU share the exact same physical LPDDR4X bus, the Mali GPU cannot access any extra memory bandwidth if acting as a standalone replacement.
  2. High Dispatch Latency: A single 2 MB matmul takes 0.94 ms to dispatch and execute on the GPU via OpenCL/Vulkan, compared to 0.47 ms on CPU NEON and 0.38 ms on NPU.
  3. The KV Cache Compression Opportunity (Going Beyond RKLLM): Once we successfully max out our RAM bandwidth on the core CPU-NPU stack to reach baseline parity, we can enlist the Mali GPU as a specialized KV Cache compression/decompression engine (such as on-the-fly quantization/dequantization of keys and values, or specialized compression routines). Offloading this high-footprint task to the GPU decreases the memory-streaming overhead of the KV Cache, effectively increasing our practical token-generation throughput under LPDDR4X bandwidth limits.

Note

Vast Room for Pure CPU-NPU Improvement: The official RKLLM runtime achieves its top performance without resorting to KV cache compression. This proves that there is immense headroom for pure software, buffer, and scheduling optimization on our core CPU-NPU boundary. We have a substantial amount of foundational driver work to do just to reach stable, uncompressed parity with RKLLM before we recruit the GPU to go beyond it.

3. Head-to-Head Comparison: Open ggml-ork vs. Closed librkllmrt

To validate our findings, we run back-to-back benchmarks on the exact same board (ROCK 5B, DDR clocked at 2112 MHz, CPU at 2.4 GHz) using the Qwen3-1.7B-w8a8 model:

  • Closed Reference: Official Rockchip librkllmrt (v1.2.3) running a static .rkllm graph.
  • Open Stack: llama.cpp + ggml-ork (ork-driver).

Comparative Performance (Standardized same-model, warm)

Metric RKLLM (Closed SDK) ggml-ork (Our Stack) Relative Performance
Decode (tg128) 11.71 tok/s 12.80 tok/s 109% (Open Beats Closed!)
Prefill (pp512) 183.90 tok/s 178.50 tok/s 97% (Prefill Parity)
NPU Cores Used Cores 0, 1, 2 Cores 0, 1, 2 Parity
RAM BW Util (Avg/Max) 48% / 83% 48% / 83% Parity

Benchmarks & Bandwidth Dynamics: Qwen3-0.6B

To analyze scaling and memory bandwidth saturation on extremely lightweight architectures, we executed back-to-back tests of the Qwen3-0.6B model (751M parameters) across three configurations on the ROCK 5B board:

  1. CPU-Only GGUF (llama.cpp): Q4_0 quantization (441.67 MiB on-disk), pinned to 4 Cortex-A76 cores.
  2. NPU-Offloaded GGUF (ggml-ork): Q4_0 model running via ORK_QUANT=8 (W8A8 fallback/chaining), pinned to Cortex-A76.
  3. Closed NPU RKLLM (librkllmrt): Official W8A8 quantization (893.00 MB on-disk) utilizing the proprietary static graph runtime.
Metric / Configuration CPU-Only GGUF (Q4_0) ggml-ork GGUF (Q4_0, ORK_QUANT=8) Official RKLLM (W8A8)
Decode (tg128) 53.60 tok/s 44.74 tok/s (CPU fallback) 20.37 tok/s (NPU-bound)
Prefill (pp512) 233.17 tok/s 202.95 tok/s (NPU-chained) 331.72 tok/s (NPU-static)
Model Footprint 441.67 MiB (463.13 MB) 441.67 MiB (463.13 MB) 893.00 MB (936.38 MB)
NPU Cores Used None Cores 0, 1, 2 Cores 0, 1, 2

Rigorous Memory Bandwidth Saturation Calculations

The ROCK 5B's physical peak memory bandwidth is 33.8 GB/s (64-bit LPDDR4X clocked at 2112 MHz). Under decode ($M=1$), the entire weight matrix must be streamed from RAM to process each token. We calculate real-world memory bus saturation as:

$$\text{Weight Memory Traffic} = \text{Model Size (GB)} \times \text{Decode Throughput (tok/s)}$$ $$\text{Memory Bus Saturation} = \frac{\text{Weight Memory Traffic}}{\text{Peak Bandwidth (33.8 GB/s)}} \times 100%$$

  1. CPU-Only GGUF Decode (Q4_0):

    • Model Size: $441.67 \text{ MiB} = 0.46313 \text{ GB}$
    • Weight Traffic: $0.46313 \text{ GB} \times 53.60 \text{ tok/s} = 24.82 \text{ GB/s}$
    • Bus Saturation: $$\frac{24.82 \text{ GB/s}}{33.8 \text{ GB/s}} = \mathbf{73.43%}$$
    • Insight: Pinned to 4 big Cortex-A76 cores, optimized ARM NEON dequantization allows the CPU to process the weights almost fast enough to saturate the physical bus.

  2. Official RKLLM NPU Decode (W8A8):

    • Model Size: $893.00 \text{ MB} = 0.89300 \text{ GB}$
    • Weight Traffic: $0.89300 \text{ GB} \times 20.37 \text{ tok/s} = 18.19 \text{ GB/s}$
    • Bus Saturation: $$\frac{18.19 \text{ GB/s}}{33.8 \text{ GB/s}} = \mathbf{53.82%}$$
    • Insight: Even though the NPU's matrix engine is fast, it is heavily bound by the ~365 µs driver ioctl submission floor. Because it evaluates sequentially ($M=1$), the driver overhead keeps the NPU and the bus idle for large periods, dropping generation speed to 20.37 tok/s and saturation to 53.82%, despite using a larger W8A8 model.

  3. Key Architectural Conclusions:

    • CPU Decode Superiority: For small models, CPU decode is 2.6× faster than official NPU-based RKLLM. The CPU achieves superior memory bandwidth saturation (73.43% vs 53.82%) by entirely bypassing the NPU driver's ioctl latency wall.
    • NPU Prefill Parity: During prefill, the batch size is large ($M \gg 1$), completely amortizing the ioctl submission overhead. Our open-source ggml-ork reaches 202.95 tok/s (61.2% of official RKLLM's static compiled graph speed of 331.72 tok/s), demonstrating the NPU's massive parallel compute capability on 8-bit quantized weights.

Key Levers Enabling the Open Stack Victory

  • Big-Core Pinned Threading (-t 4): If run with -t 8, the CPU threadpool spills onto the little Cortex-A55 cores, dragging the thread synchronization barrier and starving the NPU-driver thread. Restricting execution to the 4 big Cortex-A76 cores (taskset -c 4-7) fixes both bottlenecks.
  • Full-K Prefill (ORK_FULLK_PREFILL): Removing the K-slice host-side accumulation by processing the full K-dimension in a single submit improved NPU matmul speeds by 1.8×, bringing prefill to 97% of the closed reference.
  • Attention-on-CPU: Keeping high-frequency memory-bound attention/softmax operations on the CPU (using optimized CPU FlashAttention) avoids NPU driver submission overhead and dtype-mode RESET thrashing (attention is FP16, weights are INT8).
  • CPU NEON Vectorization (dequantize_row_q4_0): Hand-optimized ARM NEON assembly routine completely eliminated the CPU dequantization bottleneck. This delivers 1166.97 t/s on lightweight model decode and pushes 1.7B GGUF decode on CPU to 11.50 tok/s, matching official closed-source RKLLM parity.
  • Asynchronous Thread-Isolation & Prefill Stabilization (June 20, 2026):
    1. Dynamic Regs Config (regcfg_amount = 116): Dynamically scales INT4 register command length to prevent command truncation and hardware wedges.
    2. Asynchronous Single-Core Submission: Replaced unstable thread barriers (pthread_barrier_wait) and multi-core ioctl schedules with thread-isolated, core-specific submissions (sub.core_mask = 1u << i), securing 100% stable execution.
    3. Sequential Row Iteration (chunk_M = 1): Bypasses the INT4 single-row hardware limit (mc = 1), restoring mathematical correctness (0.0000 maxerr vs. CPU reference) on prefill-sized shapes without perceptible latency penalty.

4. Strategic Performance Opportunities (Revised Rankings)

✅ Completed: CPU NEON Dequantization and Vectorization Optimization

  • Milestone: Integrated fast, hand-optimized ARM NEON vectorization (dequantize_row_q4_0) inside llama.cpp.
  • Impact: Reached a peak decode throughput of 1166.97 t/s on a 24M stories15M-q4_0.gguf model. For larger 1.7B-class models, CPU-only decode has risen to 11.50 tok/s, saturating the physical memory bus at ~48% bandwidth and achieving unquantized performance parity with closed-source RKLLM runtime.

We have revised our remaining optimization targets based on these physical measurements, ranking them strictly by quantitative performance potential:

Rank 1: Native INT4 (W4A4) Multi-M Prefill & Accuracy Suite

⚠️ SUPERSEDED (2026-08-19) — see §4c. Native W4A4 compute is not the prefill path: it measures ~54× the W8A8-inflate route. What shipped is route B — int4/NF4 storage (compact .orkpack, half the resident bytes) with int8 compute — plus NF4 experts computed on the CPU. Additionally ORK_QUANT=4 forces dense tensors to int4 too, which hits the multi-domain IOVA wall (bcreate errno=14, crashes at M=512). Keep the accuracy-suite half of this item; the "native multi-M W4A4 prefill" half is closed.

  • The Opportunity: Capitalize on the June 19, 2026 breakthrough which discovered how to program the NPU's native multi-row scheduler for INT4:
    1. Force PPU stride register 0x405c to 0.
    2. Configure logical row register: $\text{mc} = 2 \times H$.
    3. Execute a stride-2 de-tiling gather copy (Row m $\rightarrow$ Row 2*m) on output buffers.
  • Why it is Rank 1: This completely bypasses the memory-bandwidth bottleneck of PC-Chaining (which fetches weights $M$ times) and delivers native 12 TOPS INT4 prefill speeds directly from disk-compressed 4-bit weights.
  • Prerequisite (Hadamard Integration): Pair this with a fast activation Hadamard rotation (QuaRot-style) to flatten outliers, reclaiming W4A4 perplexity and reducing matmul RMS error by 40–60%.

Rank 2: CPU-NPU Async Overlap and Thread Gating

⚠️ MOSTLY REFUTED (2026-08-19). The "shaves the remaining ~10 % decode gap" estimate is unsupported by measurement: prefill CPU∥NPU overlap is dead for MoE (the CPU share was ~6 % of NPU GEMM time); OMP_WAIT_POLICY=ACTIVE measured flat (5.19 vs 5.19 t/s); and in the big+little work the sleep/futex variant was actively harmful (poll=0 cost ~2 t/s; poll=50 was needed to recover it). The real decode lever turned out to be taking work off the NPU entirely plus the GGUF attention precision — see §4c.

  • The Opportunity: Create a custom cooperative scheduler in llama.cpp's threadpool.
  • Mechanics: While the NPU is processing a matmul, the calling CPU threads must sleep on a futex/wait-queue rather than spinning. This eliminates CPU-NPU thread contention, cooling down the SoC and allowing the NPU-driver thread to receive IRQ wakeups with zero latency.
  • Potential Gain: Shaves the remaining ~10% decode gap by eliminating per-submit CPU-NPU scheduling bubbles.

Rank 3: GPU-Accelerated KV Cache Compression (Beyond Parity)

  • The Opportunity: Once we max out our RAM bandwidth saturation on the core CPU-NPU stack to reach baseline parity, we can introduce Mali GPU-accelerated KV Cache compression/decompression.
  • Mechanics: Offloading key-value cache quantization, pruning, or decompression to the Mali GPU's compute shaders reduces the volume of bytes streamed from system memory during attention phases. This effectively reduces physical memory-bandwidth constraints and increases overall throughput (tok/s) on long-context runs, pushing us beyond the standard RKLLM limits.

Rank 4: KV Cache SSD Swap Offloading (Capacity Optimization)

  • The Opportunity: Offload out-of-context KV caches to high-speed NVMe SSDs via asynchronous mmap pre-fetching.
  • Mechanics: This is a capacity rather than a throughput optimization. It allows running massive context windows (32k+) without triggering Out-Of-Memory (OOM) crashes, but does not increase base generation speed.

4b. Saturation Roadmap Threads (2026-06-26)

These threads advance the saturation endgame — run the largest models the 32 GB RK3588 can hold by keeping all four resources (NPU, Mali GPU, CPU, memory) busy rather than racing a single engine. Validated on board 10.3.0.236; the ggml-ork changes live on branch feature/mixed-orkpack-v2 (commits 9f2d659 mixed-pack v2, 2e086b4 chain-persist, 15189076d tier-from-source, 8bed494 TIERMAP+IMATRIX — local, unpushed).

✅ DONE (PPL-validated): Mixed int8 / int4-NF4 .orkpack

The ggml-ork backend now builds a per-tensor mixed pack: an int8 tier for importance-bumped tensors and int4-NF4 (with an imatrix-weighted scale) for the bulk. The tier per tensor comes from either the source GGUF's quant types or an explicit name→tier map (ORK_ORKPACK_TIERMAP), combined with an importance matrix (ORK_IMATRIX).

  • PPL (wikitext-2, Qwen3-1.7B): mixed fp16 + imatrix + NF4 = 11.33 vs Q4_K_M native 10.92, at ~58% of the storage. The NF4 codebook + imatrix-weighted scale recovers most of the accuracy that flat int4 loses, at well under half the bytes.
  • Convert path now persists ALL matmul weights (chain-persist fix, 2e086b4) — previously chained sub-matmuls were re-derived rather than written, so the on-disk pack was incomplete.
  • Why it matters for saturation: the int4 bulk shrinks the source/disk + GPU/CPU-consumed footprint, which is what lets a large mixed model fit resident-by-partition (see NPU Weight-Residency). Note the int4 storage win is not a NPU-resident-footprint win — the NPU runs int8, so w4a8 weights occupy the NPU buffer as int8 regardless.

✅ DONE: DMA-buf fill optimization chain (ork-driver hot path)

Residency loads and prefill-streaming both bottleneck on filling NPU-mapped DMA buffers. Three wins, all hardware-validated:

  • Cacheable buffer + clean-before-submit (8×): a cacheable 0x403 DMA buffer filled by the CPU then cleaned before submit runs at ~10 GB/s vs ~1 GB/s for the write-combine (0x401) path. The production weight tiles already use 0x403; only the zero-copy ork_dma_alloc path was still on 0x401.
  • Direct int4→int8 tiled inflate (ORK_DIRECT_I4), 2.2–5×: inflate packed int4 straight into the int8 tile layout, skipping the f32 round-trip. Bit-identical to the round-trip path.
  • Next lever: a clean-only bsync for the steady-state re-fill (skip the post-write invalidate when the CPU is the sole writer between submits).

🔬 DESIGN: Phase-aware heterogeneous scheduling

⚠️ BOTH HALVES CONTRADICTED (2026-08-19) — see §4c. "Prefill = compute-bound → NPU-centric": the 1.59× prefill win came from moving the expert bulk to the CPU (NF4 + batched NEON), leaving the NPU only attention/dense — NPU matmul is just 11.6 % of the pp512 wall. "Decode = bandwidth-bound → partition for additive bandwidth": bandwidth does not sum usefully here — a single A76 nearly saturates the bus, 1big+2little measured slower than 3big (1.68 vs 3.00 t/s), and every decode configuration with NPU participation lost to all-CPU. Phase-awareness itself is right (and shipped: prefill NPU-assisted, decode all-CPU) — the engine assignments in this section are inverted.

Prefill and decode have opposite bottlenecks (see §1), so they want opposite engine strategies:

  • Prefill = compute-bound → NPU-centric. The NPU's ~6 TOPS int8 dwarfs CPU NEON and Mali Vulkan. Layer latency is set by the slowest engine, so partitioning a matmul onto a slower engine starves the NPU and lengthens the layer. Instead, keep the matmul on the NPU and use the CPU/GPU to feed and prefetch (quant, tile, stage the next layer).
  • Decode = bandwidth-bound → partition for additive bandwidth. Each engine has its own slice of the shared 33.8 GB/s; splitting a tensor across NPU + GPU + CPU sums their effective bandwidth. Route int4 tensors → GPU/CPU (native zero-copy at that dtype) and int8 tensors → NPU resident.
  • Speculation unifies the two: EAGLE-3 verify is M = K (a prefill-shaped batch) → run the verify on the NPU in parallel with the draft on the GPU.
  • The tier map doubles as the engine-assignment oracle — the same name→tier table that drives the mixed pack tells the scheduler which engine owns each tensor.
  • Next measurement: a per-engine throughput bench (NPU int8 / Mali Vulkan / CPU NEON) swept over M = 1 → 2048. This is double-duty: the M ≫ 1 end sizes the prefill compute-lead, the M = 1 end sizes the decode bandwidth-partition.

MoE-on-NPU build milestones (Branch A — NPU-centric)

Sparse-MoE (MUL_MAT_ID) routed-expert GEMMs are the next big NPU-residency target. The build is staged M0 → M3; M0, M1a, and M3 are now proven (hardware-validated 2026-06-26).

  • M0 — sparse-MoE expert-GEMM viability on NPU: ✅ VIABLE. The NPU beats CPU 3–10× at every Qwen3-A3B expert shape with per-expert M ≥ 8 — there is no crossover where CPU wins back. The new direct int4→int8 inflate (ORK_DIRECT_I4, see DMA-fill chain above) closed the old f32-repack wall by 16–30×, which was the previous blocker; output is bit-exact against the CPU reference. Dispatch rule: chainable gate/up projections use run_stream_i8 (async cross-core); the down projection uses run_i8 — not run_chain*, which is single-core.
  • M1a — MoE expert persist/load + the IOVA wall: ✅ DONE (hardware-validated). ggml-ork (board branch feature/mixed-orkpack-v2, c89cf8654) now persists the routed _exps (MUL_MAT_ID) tensors into the mixed .orkpack — a synthetic per-expert key, write-all-slices at convert via ork_orkpack_tier, read back via ork_mm_load_i8/i4a8 (+ ork-driver 6b0d568 so disk-loaded int8 experts chain). On LFM2.5-8B-A1B the pack now carries 2112 _exps entries (264 × 8) vs 0 before; reload is bit-identical to the CPU reference. KEY FINDING: running real experts on the NPU surfaced the 4 GiB IOVA cap for real — 8B int8 experts exceed 4 GiB, so compute fails once ~1900 distinct experts are resident. This confirms streaming (M2) and the hot-cache (M3) are mandatory, not optional.
  • M2 — tile-streamed prefill double-buffer ⬜ at tile granularity (stage the next expert's tiles while the current one runs; this is where IRQ-affinity steering below becomes critical).
  • M3 — routing-skew profile → hot-cache sizing: ✅ DONE (hardware-validated). Both MoEs are routing-skewed (LFM2 Gini 0.44; Qwen3.6-35B-A3B target Gini 0.65 — skew rises with depth), so a small resident hot-cache covers a large share of decode expert lookups. Target 35B hot-cache hit-rate vs size (per-expert int8 = 3.15 MB): 10% (26/layer) = 48% @ 3.3 GB · 20% = 67% @ 6.4 GB · 30% = 79% @ 9.7 GB · 50% = 92% @ 16 GB. VERDICT: hot-cache VIABLE — ~3–3.5 GB pins ~50% of decode expert lookups NPU-resident (inside the 4 GiB window, alongside the pinned backbone), the cold half routes to CPU/GPU; pushing past 50% needs cold-tail streaming, with per-layer top-K. This sizes the Branch-A decode partition.
  • Next: M1b (backbone-pin + tier-routed experts) + M2 (tile-streamed prefill) ⬜.

🔬 DESIGN: Newly-accepted optimization items (2026-06-26)

These compose with the saturation threads above; ordered roughly by readiness.

  • Dynamic draft sizing. An acceptance-rate feedback loop adapts speculative draft depth — grow on predictable text, shrink on code/reasoning — to cut wasted full-depth compute on rejected tails. It also balances the MoE tension: a bigger draft means a bigger verify batch (more expert-swap amortization, §M2/M3) but a lower acceptance tail, so the loop tunes the sweet spot per workload.
  • Radix-tree prefix caching (agentic/coding). Keep the shared system-prompt / tool-definition / conversation-history KV hot and resident across turns so repeated prefixes skip re-prefill. Orthogonal to and composes with PolarQuant (shrinks KV size) and EAGLE-3 (speeds decode). Needs an eviction policy.
  • Paged KV cache (PagedAttention). RAM-side paged KV — attention-over-KV runs on CPU/GPU, not NPU IOVA, so it sidesteps the 4 GiB IOMMU cap — eliminating long-context fragmentation. It is the substrate the prefix cache is built on.
  • IRQ affinity. Steer the rknpu + Mali IRQs to the A55 little cores so they don't preempt the A76 NEON prefetch pipeline. This is distinct from the existing driver-thread big-core pin (that pins worker threads to A76; this moves interrupts off A76). Becomes critical at M2, where the prefetch pipeline must run uninterrupted alongside NPU submits.

Contingency / deprioritized

  • Lookahead / Jacobi decoding — a zero-draft-model fallback to use if the draft footprint binds the memory budget (a draft model that's too large lowers acceptance and forces smaller verify batches). Kept as a contingency, not a current track.
  • Mali RoPE / SwiGLU offload — profile-gated, low priority. These ops are cheap and the M = 1 handoff cost to the GPU is expensive, so the offload rarely pays.
  • NPU-IOVA paged allocator — already covered. M2's uniform tile buffers plus the IOMMU's scatter→contiguous mapping give us the paging behavior; a separate allocator is redundant.

4c. MoE prefill: shipped win + open backend items (2026-08-19)

✅ SHIPPED: knob-free MoE auto-profile — 1.59× native prefill (Qwen3.6-35B-A3B)

  • What: a routed-MoE model (any GGML_OP_MUL_MAT_ID, latched at load-time graph planning) now selects the measured-optimal scheme with no environment variables: experts stored int4/NF4 and computed on the CPU via a batched 4×4 NEON GEMM; attention/dense int8 on the NPU at prefill; all-CPU at decode (the NPU per-submit floor loses at M=1); .orkpack path derived from the model (ORK_PERSIST retired).

  • Measured (-t4, governors=performance, 1024-token PPL window, board 10.3.0.236):

    config prefill decode PPL
    native ggml (Q4_K experts) 23.3 7.5 10.2058
    ork auto-profile 36.9 (1.59×) 6.3–6.6 10.7718 (+5.5 %)

  • It is a trade, not a free win: +5.5 % perplexity and ~15 % slower decode for 1.59× prefill. Prefill-heavy workloads (long prompts, RAG, batch) win; decode-dominated chat may prefer ORK_MOE_AUTO=0.

  • Why it wins: prefill is compute-bound on expert dequant; NF4's 1-op vqtbl unpack is cheaper than Q4_K's super-block dequant, and both experts and attention are pre-packed/resident (no per-batch re-dequant).

  • Decode ladder for the record: 3.60 → 4.41 (attn off the NPU at M=1) → 5.19 (fused M=1 expert gemv on a persistent OpenMP pool — the earlier std::thread-per-op version spawned ~360 threads/token and measured identical to the batched path) → 5.30 (one-cache-line PRFM prefetch) → 6.59 after the GGUF attention precision change. Decode remains native-bound; nothing on the ork path beats it.

✅ Completed: attention-offload correctness (ORK_ATTN) — still OFF by default

Three independent defects made the offload produce garbage (PPL 10.83 → 67.86): a transposed B fill in ggml_backend_ork_bmm_fp16, an M-tile envelope overrun in the FLASH_ATTN_EXT handler, and an on-NPU softmax whose exp_i16 is only valid at some row counts (now opt-in via ORK_ATTN_SM_NPU). Fixed → PPL 10.7607. Kept off: the offload is perf-neutral (pp64 0.96×; NPU matmul is only 11.6 % of the pp512 wall; operand densify/repack costs ~2.4× the matmul time it saves). New differential oracles ork-driver/tools/re/{fa_diff_probe,bmm_diff_probe}.cpp compare one node on the ggml CPU backend vs the ORK backend in ~20 s — use these instead of 6-minute model loads for any backend-math question.

⬜ Open backend items

  • res=-3 "failed to decode prompt batch" at pp512 on the q8attn base, appearing after the per-domain attention-pool change (pp512 was clean at 36.38/36.58 before). Regression or cache-drop artifact? This blocks the item below — without a clean baseline there is no valid A/B.
  • ORK_ALLOW_Q4_SRC speed probe (avoids a requantize). supports_op declines any sub-5-bit source (sbits < 5.0), so the Q4_K ssm_* (GDN) projections run on the CPU — and they are the largest CPU-backend item: 1132 ms / 31 %, n=78 ≈ 4 projections × 18 GDN layers, while the NPU sits at ~11.6 % of the wall. Gate the decline behind an env knob so they take the ordinary W8A8-inflate route and measure. Speed only — Q4_K→int8 is lossy double quant, so PPL from such a run is meaningless. Do not use ORK_MIXED_DISPATCH (it accepts sub-5-bit but runs native-W4A4, ~54× the W8A8 path — it would fake a negative). If it wins, the durable fix is a model recipe, tracked on the oRKLLM orkpack-conversion page.
  • ORK_OPPROF per-op CPU-backend profiler (ggml/src/ggml-cpu/ggml-cpu.c, env-gated, caller-side hook in the graph loop, splits MUL_MAT into dynamic-src0 vs weight buckets) — uncommitted; land as a diagnostic or drop. It produced the composition above: pp512 wall 42.0 s; NPU matmul 4.88 s (11.6 %); GATED_DELTA_NET 874 ms (23.9 % of CPU-backend, n=18); CONCAT 687 ms; FLASH_ATTN_EXT 267 ms.
  • Fold the fork's scratch WIP docs (ggml/src/ggml-ork/{NF4_ROUTE_WIP,FP16_BMM_DOMAIN_WIP}.md) into wiki pages and delete them.
  • GDN offload is CLOSED — llama.cpp emits GDN as a single fused GGML_OP_GATED_DELTA_NET, not chunk matmuls, and an on-NPU implementation already exists on ork-driver branch feat/gdn, measured 1.9–4.8× slower than ggml's CPU scan for a structural reason (the chunked form needs a CPU forward-substitution (I+A)⁻¹ that the solve-free per-token recurrence never pays). Do not re-derive.

Driver-side follow-ups from the same batch (fp16 M-envelope refusal, exp_i16 envelope, int4-only dom_dirty) are tracked on the ork-driver Optimization Roadmap, Tier 14.

5. Development Action Plan

To realize the next leaps in performance, our engineering efforts must focus on:

sequenceDiagram
    participant Host as llama.cpp (CPU)
    participant Driver as ork-driver (npu.c)
    participant NPU as NPU Silicon (INT4)
    
    Host->>Driver: 1. Submit Activation Matrix A (M >= 32)
    Note over Driver: 2. Set mc = 2 * H<br/>Set PPU Stride (0x405c) = 0
    Driver->>NPU: 3. Issue Native Multi-M ioctl
    NPU->>Driver: 4. Write Strided Output (Row 2*m)
    Note over Driver: 5. Perform Stride-2 Gather De-tiling Copy
    Driver->>Host: 6. Contiguous Output Buffer (C)
Loading
  1. Refactor run_i4_mc in npu.c:
    • Inject the logical mc = 2 * H register configuration.
    • Disable the PPU stride 0x405c during INT4 execution.
    • Implement the high-speed stride-2 gather de-tiling copy in the worker thread.
  2. Integrate Offline Hadamard Weight Pre-Rotation:
    • Rotate quantized weights once at model load time.
    • Fuse activation Hadamard rotation (ork_fwht_norm) directly into preceding CPU normalization operations (e.g., RMSNorm) to hide CPU mathematical overhead entirely.
  3. Establish Auto-Tuning Core Budgets:
    • Set NPU core counts dynamically based on the matmul shape: do not split small-N matmuls across 3 cores, as thread-synchronization cost outweighs NPU scaling benefits.

Clone this wiki locally