NPU Weight Residency Bypass

NPU Weight-Residency Decode Bypass Bug

Note (2026-06-24): The residual per-submit decode cost discussed here is the NPU submission round-trip, which is real. But a much larger co-resident-process decode stall has since been isolated as CPU big-core scheduling contention (an idle co-resident server's threads on the big cores preempting ork-driver's pinned submit thread), NOT an NPU single-stream limit. Rule out core affinity before attributing slow decode to the NPU — see CPU Big-Core Contention.

Executive Summary

During multi-turn conversational chat, a catastrophic performance degradation was observed on Turn 2 and subsequent turns, where token-generation (decode) speeds plummeted to $\approx 0.5\text{--}1$ tok/s (extremely slow). However, Turn 1 (the initial prefill and single-token decode) executed at high performance ($15+$ tok/s).

This document details the root cause of this performance discrepancy and the architectural fix implemented in the ggml-ork backend to restore optimal throughput.

---

Root Cause Analysis

The Weight-Residency Mechanism

To optimize tensor operations, the ork-driver backend implements a weight caching system (wcache). Weights are loaded, packed, and marked as resident in the NPU's memory:

• Turn 1 (Cold Weights): On the first turn, weights are not yet resident in the NPU's caching memory.
• Turn 2+ (Warm/Resident Weights): Once loaded, weights persist in the NPU's weight cache, and subsequent operations recognize them as "NPU resident" via is_npu_resident(src0).

The Gating Failure

In ggml-ork.cpp, the backend determines whether to route a matrix multiplication (ggml_backend_ork_device_supports_op) to the NPU using the following gating logic:

if (is_npu_resident(src0)) {
    pass_m_threshold = (M >= 1); // <-- Routing failure trigger
} else {
    pass_m_threshold = (M >= min_m || op->ne[2] > 1 || op->ne[3] > 1);
}

• On Turn 1: is_npu_resident(src0) returns false. The gating falls to the else branch, checking if the batch size $M \ge \text{min_m}$. Because single-token decode has $M=1$ (which is less than min_m), decode is correctly and efficiently routed to the CPU NEON cores (fast).
• On Turn 2+: The weights are resident, so is_npu_resident(src0) returns true. This causes supports_op to use pass_m_threshold = (M >= 1). Since $M=1$ for decode, the single-token matmuls are routed to the NPU.

The NPU Submission Tax

Although the NPU's hardware compute cores are highly efficient for large batch sizes ($M > 32$), submitting an operation to the Rockchip driver carries a fixed software submission overhead (or "tax") of $\approx 365,\mu\text{s}$ per submit. With $\approx 197$ distinct matmul submissions per token during decode, this overhead compounds: $$\text{Submission Tax per Token} = 197 \times 365,\mu\text{s} \approx 72,\text{ms}$$ This software submit tax completely dominates the actual computation time, bottlenecking throughput to $\approx 0.5\text{--}1$ tok/s.

---

The Solution

The weight-residency override was removed from the routing logic. Regardless of whether the weights are resident in the NPU cache, single-token decode operations ($M=1$) must never be scheduled on the NPU.

Gating Correction

The gating logic in ggml-ork.cpp was updated to strictly enforce the batch size threshold for all operations:

-if (is_npu_resident(src0)) {
-    pass_m_threshold = (M >= 1);
-} else {
-    pass_m_threshold = (M >= min_m || op->ne[2] > 1 || op->ne[3] > 1);
-}
+pass_m_threshold = (M >= min_m || op->ne[2] > 1 || op->ne[3] > 1);

By removing the conditional bypass, all decode operations ($M < \text{min_m}$) are consistently and stably routed to the CPU NEON threads, preserving the high-speed 15+ tok/s line-rate across all conversation turns.

---

Memory & Residency Reality (2026-06-26)

Findings from the saturation-endgame work; hardware-validated on board 10.3.0.236. These reframe what "weight residency" can and cannot buy on this SoC.

UMA: the NPU buffer is system RAM, and the NPU runs int8

The NPU's dma_buf is carved from the same 32 GB of LPDDR4X as the CPU and GPU — it is not a separate pool. Two consequences for the mixed-precision pack (see Mixed int8/int4-NF4 .orkpack):

• The int4 storage win does not shrink the NPU resident footprint. The NPU compute path is int8, so a resident w4a8 weight occupies the buffer as int8 regardless of its on-disk int4 form. The int4 savings are in the source/disk footprint and in GPU/CPU consumption (those engines read int4 natively), not in the NPU's resident bytes.
• The 4 GB IOVA cap is carved from the same 32 GB. The RK3588 NPU IOMMU is 32-bit, so only ~4 GB of weights are DMA-mappable at once — a hard ceiling on int8 NPU residency, taken out of the shared 32 GB (not additive to it).

Fit-by-partition: resident without streaming

A ~35B mixed model (~20 GB packed) can sit fully resident by partitioning across engines rather than streaming:

• NPU ≤ 4 GB — the int8 tier (hottest / most compute-heavy tensors), capped by the IOVA limit.
• CPU / GPU ~28 GB — the int4 bulk (consumed natively at int4) plus any int8 overflow.
• + KV cache — fits in the remaining headroom.

The scarce 4 GB of NPU residency is the constrained resource, so it should hold the hottest, most compute-heavy int8 tensors — exactly the tensors the tier map flags for the int8 tier. This is the residency side of the phase-aware scheduling design: prefill compute concentrates on the NPU-resident tensors, while int4 tensors live where their native-dtype engine (GPU/CPU) can stream them.

Streaming is the fallback, not the plan. Per-token decode streaming is bandwidth-walled (each token re-streams the layer), and the DMA fill only hides behind compute at large prefill M. Fit-by-partition avoids streaming entirely for models that fit the 32 GB.

What makes residency loads cheap

The fit-by-partition picture is only practical because the DMA-fill path is now fast (see the DMA-buf fill optimization chain): the cacheable 0x403 + clean-before-submit fill (~10 GB/s, 8× over write-combine) and the direct int4→int8 tiled inflate (ORK_DIRECT_I4, 2.2–5×, bit-identical) are what keep both the initial residency load and any prefill-streaming refill cheap.