Skip to content

NPU Physical Path Integration Gaps

Jump to bottom
Michael Fischer edited this page Jun 24, 2026 · 2 revisions

NPU Physical-Path Integration Gaps & Optimization Roadmap

Update (2026-06-24) — re-prioritized after real-model testing. The INT4/W4A4 "Rank 1" item below is not yet viable on real models (not a near-term win): on Qwen3-1.7B/wikitext through ggml-ork, per-channel int4 (the fast 1-submit layout) is incoherent even with Hadamard (PPL ~104 vs CPU-Q8 ~20), and grouped int4 (coherent, PPL ~24) needs ~16 submits/matmul — slower than int8, which is already coherent and fast. int4 stays an open research track (a stronger SpinQuant/learned rotation could flip it); w8a8 is the production path today. The MoE/expert-offload item is likewise not viable on RK3588 as-is (submit+repack-bound). And decode-slowdown symptoms are usually CPU big-core scheduling contention from a co-resident server, NOT NPU single-stream — see CPU Big-Core Contention and Quantization Paths & Baselines.

Executive Summary

While the ggml-ork backend in llama.cpp-rockchip delivers class-leading performance for dense INT8 models, a comparison with the raw capabilities of the underlying ork-driver reveals three critical integration gaps. These gaps prevent the software stack from fully exploiting the physical NPU silicon for sparse architectures (Mixture of Experts) and native 4-bit (INT4/W4A4) quantization.

This document details these gaps, explains their low-level mechanics, and outlines a structured roadmap to bridge them.

The Three Integration Gaps

graph TD
    A[NPU Integration Gaps] --> B[Gap 1: MoE Gating Bias]
    A --> C[Gap 2: INT4 Multi-M Scheduler]
    A --> D[Gap 3: Inline Hadamard Overhead]

    B --> B_Desc["Static M>=32 guard forces sparse experts to CPU, triggering 32ms RAM-streaming loops"]
    C --> C_Desc["Driver still uses serial CPU row loops instead of native stride-2 hardware sequencer"]
    D --> D_Desc["FWHT activation rotation runs inline on CPU, blocking NPU submission threads"]
Loading

Gap 1: MoE Expert Gating Bias (Dense-Model Policy Bias)

Low-Level Mechanic

To amortize the irreducible ~365µs NPU submission overhead (ioctl + IRQ context switch), ggml-ork.cpp enforces a static gating threshold of $M \ge 32$ (where $M$ is the batch size/token dimension). If $M < 32$, operations are routed to CPU NEON threads.

While this policy is highly effective for dense models (keeping single-token decodes on the CPU), it fails catastrophically on Mixture of Experts (MoE) architectures during prefill. Under MoE routing, token splitting distributes a batch of tokens among multiple active experts. Even with a large prefill batch (e.g., 32 tokens), individual experts may only receive 1 to 4 tokens ($M_{\text{expert}} = 4$).

The Bottleneck

  • CPU Gating Fallback: Because $M_{\text{expert}} < 32$, ggml-ork routes the expert matrix multiplications to the CPU. Since these expert weights are not resident in CPU LPDDR caches, they must be streamed over the system RAM bus (e.g., 25 MB per expert layer), imposing a severe ~32 ms memory transfer bottleneck per token.
  • The NPU Alternative: The NPU keeps expert weights fully resident in physical IOVA buffers. Evaluating the expert matmul on the NPU only pays the 365µs driver context-switch latency. Total execution takes 1.1 ms (a ~30× speedup).

The Fix

Refactor ggml_backend_ork_device_supports_op to implement a "Weight-Resident Gating Policy" that bypasses the $M \ge 32$ guard for layers containing "expert" in their name, allowing them to evaluate on the NPU down to $M \ge 1$:

bool is_expert = strstr(src0->name, "expert") != nullptr;
int actual_min_m = is_expert ? 1 : min_m;
bool pass_m_threshold = (M >= actual_min_m || op->ne[2] > 1 || op->ne[3] > 1);

Gap 2: Native INT4 Multi-M Scheduler Integration

Low-Level Mechanic

Historically, the physical INT4 (W4A4) hardware path on the RK3588 was believed to be restricted to single-row ($M=1$) submissions. To bypass this, we previously implemented PC-Chaining (ork_mm_run_chain_i4), which chained single-row tasks inside a single ioctl using sequencer registers. However, this forced the NPU to re-fetch the weight matrix from DDR for every single row, dragging down prefill throughput.

On June 19, 2026, register-level fuzzing of the physical INT4 path successfully unlocked the NPU's native multi-row scheduler:

  1. Register Trigger: Unlike INT8, the PPU output stride register 0x405c must be forced to 0 (disabled).
  2. Scheduling Formula: Setting the row-scheduler register parameter to mc = 2 * H instructs the NPU to compute exactly $H$ rows natively in a single pass.
  3. Output Mapping: The NPU writes the results using a stride-2 row layout, mapping logical row $m$ to physical output row $2 \times m$ of the DMA buffer.

The Bottleneck

The active production code in run_i4_mc (npu.c) still loops over rows on the CPU or falls back to PC-Chaining, rather than executing native multi-M submits.

The Fix

  1. Update synth_i4 in the driver to support native multi-M values and force 0x405c to 0.
  2. Refactor i4_mcworker in npu.c to submit the entire batch size $M$ to the NPU in a single pass.
  3. Implement a stride-2 de-tiling gather copy in the worker thread post-submit sync:
// Stride-2 De-tiling:
for (int m = 0; m < H; m++) {
    // Logical row m maps to physical output row 2 * m
    memcpy(contiguous_C + m * N, raw_npu_output + (2 * m) * N, N * sizeof(int16_t));
}

Gap 3: Fused Hadamard Activation Rotations

Low-Level Mechanic

To preserve high numerical accuracy and perplexity under 4-bit quantization, we employ Fast Walsh-Hadamard Transforms (FWHT) via ork_fwht_norm. This rotates the coordinate system of weights and activations, smoothing out activation outliers and dropping the W4A4 perplexity ratio from 1.69 to a stable 1.46.

The Bottleneck

Currently, the activation rotation is evaluated inline on the CPU thread during ggml_backend_ork_mul_mat_i4_hadamard execution. This introduces $O(K \log K)$ CPU mathematical overhead directly into the critical path, blocking the NPU submission threads.

The Fix

To eliminate this runtime tax:

  1. Offline Weight Rotation: Pre-rotate the weight matrices once at startup during weight packing and caching.
  2. Fused Activation Rotation: Fuse the activation rotation (ork_fwht_norm) directly into preceding CPU-side normalization layers (such as RMSNorm or LayerNorm). Since RMSNorm already reads and writes the activations buffer, incorporating the coordinate rotation during the writeback phase hides the mathematical overhead completely.

Optimization Roadmap

To implement these optimizations, we follow a coordinated, low-risk development plan:

Phase Target Backend Task Description Expected Impact
Phase 1 ggml-ork (llama.cpp) Implement Weight-Resident Gating for "expert" layers in supports_op. ~30× Speedup on MoE models, restoring NPU prefill load.
Phase 2 ork-driver Refactor run_i4_mc to support the stride-2 native Multi-M scheduler. Unlocks full INT8 prefill speed parity for native INT4 (W4A4) paths.
Phase 3 ggml-ork (llama.cpp) Fuse FWHT activation rotation into preceding RMSNorm layers. Delivers zero-overhead W4A4 perplexity recovery.

Clone this wiki locally