Skip to content

Hybrid Precision Decode Architecture

Jump to bottom
Michael Fischer edited this page Jul 16, 2026 · 6 revisions

Hybrid CPU/NPU Precision Architecture (int5 storage, sensitivity-routed)

Design captured 2026-07-15. Status: format primitive built + measured. Supersedes the "stream the model through the NPU / JIT-int8-at-swap" framing.

Update (2026-07-15, measured — tools/int5_probe.c):

Storage is SINGULAR. One ork-native format per tensor, on disk and in DRAM; both engines read the same bytes across both phases (prefill NPU M>1, decode CPU M=1). Q4_K is only the GGUF source, converted once at build — it is NOT in the runtime path (its block layout ≠ the NPU tile layout, so it fails NPU prefill). The CPU therefore reads the ork-native format too (measured memory-bound: NF4 vqtbl 22.7 GB/s, int4 nibble 23.4) — not a second Q4_K copy.

Format must be per-channel (the NPU dequant is per-channel — it can't do Q4_K's per-block scale), which sets the accuracy-per-bit ladder. Tiered per-tensor by sensitivity + memory: int8 (~0.01 rel-RMSE, better than Q4_K, native, 2× mem — sensitive tensors) · NF4 (0.109, free vqtbl inflate — bulk) · int4 (0.156, free nibble — least-sensitive).

int5 DROPPED: its 5th-bit merge is ALU-bound at M=1 (11.8 GB/s, slower than int8) — and since the CPU decode reads the same shared bytes, int5 would slow decode. NF4 gives the free-inflate accuracy step instead. Caveat: per-channel < per-block for equal bits, so the NF4/int4 bulk is coarser than the Q4_K source — recovered via int8 on sensitive tensors + tier-map tuning; needs a real-model PPL check.

The body below is the original single-int5-format framing — read it as the tiered per-channel ork-native scheme above (int8 / NF4 / int4), one orkpack, both engines/phases.

Bit-need routes the engine (2026-07-15). The NPU has exactly two native widths — int4 (W4A4) and int8 (W8A8). The CPU is flexible, so it is the only engine that can process the in-between widths at native precision — no upcast to int8 needed. So a tensor's required bit-width picks its engine. But only int4 (nibble) and int8 (byte) inflate for FREE (byte-aligned, hides behind DRAM); int5/int6/int7 are sub-byte non-nibble → cross-byte bit extraction → ALU-bound at M=1 (int5 measured 1.29× int8 time, slower than int8 despite fewer bits). So the width ladder:

width inflate engine note
int4 free (nibble / NF4 vqtbl) CPU bulk (+ NPU W4A4 prefill) vast majority of weights
int5 ALU-priced CPU-only −37.5% vs int8; memory lever (minority)
int6 ALU-priced CPU-only −25% vs int8; memory lever (minority)
int7 ALU-priced — skip only −12.5% vs int8 → just use int8
int8 free (byte) NPU-only W8A8 = the NPU's best-suited task; ~0.01 acc; keeps 8-bit bandwidth off the CPU

Engine specialization by strength: int8 is NPU-only (dense high-throughput W8A8 is what the NPU is best at, and it keeps the CPU's heaviest format off it); int4 is the vast majority, CPU bulk (free inflate, memory-bound, its bandwidth sweet spot) + NPU W4A4 at prefill. int5/int6 are CPU-only memory levers for the sensitive minority — and because int4 dominates (free), their ALU-bound inflate is a rounding error in aggregate decode time, so no upcast needed. int7 skipped.

Two coherence notes: (1) int8-NPU-only + decode M=1 → the NPU's int8 work must be swap-hidden / overlapped with the CPU int4 bulk (it loses at M=1 in isolation); the int8 tier is a sensitive minority, so its NPU share fits inside the CPU window. (2) The accuracy bet is on the int4 majority (per-channel 0.109–0.156, coarser than the Q4_K source since the NPU forces per-channel) — recovered by int8 on sensitive tensors + int5/6 in the middle; gated on a real-model PPL check.

Update (2026-07-16, measured) — NPU is a SUPPLEMENTAL path, and the CPU route must be ggml-speed.

Decode measurements forced two corrections to the runtime shape (the storage scheme above is unchanged):

1. NPU must never be on the critical path. The #14 partition routing (hot experts → NPU-only, cold → CPU-only) puts slow M=1 NPU experts on the critical path → 3× slower than CPU (LFM2.5-8B-A1B: 6.42 vs CPU-only 20.91; 35B-A3B: 6.13 vs CPU-only 7.01). MoE-on-NPU at M=1 is a net cost (submit-bound, can't be hidden if it carries whole experts). The fix is a speculative overlay: the CPU computes the complete result (the critical path, = CPU-only speed), the NPU redundantly computes the sensitive int8 subset non-blocking, combined int8-if-ready-else-int4 with a deadline, never a join — so the NPU can only add int8 quality, never drag. The win flips from a speed claim (M=1 NPU can't make it) to quality-at-CPU-speed, validated by PPL not tok/s.

2. The CPU route must match ggml, or the floor collapses. The orkpack CPU path (per-expert ork_cpu_gemv) is slower than ggml's fused batched MoE (measured 6.13 < 7.01) — the gap is fused-vs-per-expert kernel, not the int4 format. Since the CPU is the complete critical path, this must be fixed first: route the CPU-bound ops through ggml's tuned fused kernel on the ork-native weights (repack ork-native → ggml layout once at load, PATH-B cpu_repack style) so the CPU route inherits ggml-speed by construction.

3. Unified by an ork DISPATCHER (the general form). Rather than a fixed partition, an op dispatcher intercepts each fused/batched matmul and routes by (work-type M, precision/sensitivity, engine availability) against a calibrated engine-perf model: CPU-only (M=1 non-sensitive / NPU busy — the ggml-speed route), NPU-only (M>1 prefill/verify, where the NPU wins), or CPU-complete + NPU-int8 overlay (M=1 sensitive, NPU free — the supplemental route). This subsumes the partition, the M-gate, and the supplemental overlay as routing decisions.

Target = Qwen3.6-35B-A3B (35B int8 ≫ NPU ~8 GiB → NPU holds only a fraction → multi-domain mandatory; matching tiered orkpack full-nf4.orkpack 23.3 GB exists + validated resident). Baselines (CPU-only floors the dispatcher must preserve): LFM 20.91, 35B-A3B 7.01 decode. Submit-floor mechanism (multi-core NONBLOCK ork_dyn_begin_mc 2.27×/1.38×, precompiled-program cache 6.03×, doorbell/halt, multi-domain-correct) is BUILT + committed — see Exp-2026-07-16 Dynamic-Submit API. Build order now: (b) ggml-speed CPU route → dispatcher policy → supplemental overlay, validated decode ≥ CPU floor + lower PPL.

The goal is to run a >IOVA sparse-MoE model (the 35B-A3B) faster by using each engine at its precision optimum, concurrently, on one shared weight format. It is a hybrid: the CPU and NPU each do what their hardware is good at, on the same stored bytes, overlapped within a token window.

TL;DR

  • One ork-native, per-channel, tiered storage format (int8 / NF4 / int4 by sensitivity), converted from the GGUF once at build. Both engines read it, both phases — no Q4_K at runtime, one DRAM copy. NF4 inflates to int8 by a single vqtbl — measured memory-bound-free (22.7 GB/s). (int5 was proposed for Q4_K-grade accuracy; measured ALU-bound at M=1 → dropped.)
  • CPU / NEON = the flexible engine. It inflates the stored int5 to a working container for free in-register (the ALU work hides behind DRAM latency) and runs any WxAy the tier needs (W4A5, W5A8, …). It owns the precision-sensitive bulk of decode.
  • NPU = the fixed-function engine. Its datapath is W4A4 or W8A8 only. It absorbs the least-sensitive tail of the current token's active experts, concurrently, via a non-blocking submit, on weights swapped into IOVA — with the swap hidden behind CPU work.
  • Route by sensitivity, not by a fixed resident set. The CPU keeps sensitive weights; the NPU gets the least-sensitive slice (where its rigid/lossy modes don't hurt). Per-token dynamic residence, not a static hot set (which would leave the NPU idle).

Why this shape — the measurements that forced it

All on RK3588, board 10.3.0.236, this session (2026-07-15). Full data: TIERED_I4_WIP.md, Experiment Log, and the moe-expert-offload findings.

  1. The concurrent MoE split (PATH-B) loses on the 35B as-is — 20.50/3.71 tok/s vs a 22.77/6.14 baseline (0.90× prefill, 0.60× decode). Not because the split is wrong, but because the run is streaming-bound: 78 s wallsum is dominated by weight re-resolution (dequant Q4_K→f32→int8→tile, 24 s) + activation quant (25 s). The NPU MoE compute was 367 ms — noise.

  2. The resolve tax is fundamental to putting Q4_K on an int8 NPU. Q4_K's block layout ≠ the NPU tile layout, so every streamed expert must dequant + repack. The CPU pays no such tax — it computes Q4_K natively. So for streamed weights, CPU is structurally cheaper than the NPU.

  3. No int4 format is both accurate and free-to-upconvert. Weight-reconstruction rel-RMSE (Gaussian weights, tools/cpu_i4_vs_q4k.cpp):

    scheme rel-RMSE free inflate?
    int4 per-channel uniform 0.1559 ✅ nibble
    NF4 per-channel codebook (LUT) 0.1093 ✅ LUT
    int4 per-64-block 0.1076 ❌ block scale
    int5 per-channel uniform 0.0728 ✅ 5-bit expand
    Q4_K per-32-block (ggml) 0.0713 ❌ block scale

    int5 per-channel ≈ Q4_K accuracy while keeping a per-channel scale (so it inflates without a block-scale multiply). This is the key unlock. (Caveat: Gaussian synthetic; real weights have outliers where Q4_K's per-block pulls slightly ahead — validate int5 on a real tensor.)

  4. int5→int8 is a lossless widen (a 5-bit signed value fits an int8 byte exactly), so int5-storage → int8-expand → W8A8 gives Q4_K-grade weights with int8 activations (coherent) — sidestepping the W4A4 PPL-104 problem, which was the int4 activations, not the weights. So the design ships on W8A8; W4A4 is a later zero-copy optimization.

  5. The inflate is free on a memory-bound path. The int4 unpack measured 24.2 GB/s — identical to int8 (cpu_i4_vs_i8.c): the mask/shift/sign-extend ALU ops hide behind DRAM latency.

Storage format: int5 = nibble plane + MSB bit-plane

per output channel n, K weights:
  nibble[n][K/2]   low 4 bits of each 5-bit signed code   (int4-clean, existing Bi4 path)
  msb[n][K/8]      the 5th bit, packed 8/byte
  bscale[n]        per-channel dequant scale = absmax/15   (fp32)
  • Total 5K/8 B/channel = 1.25× int4 — exactly the 4→5-bit ratio; nothing redundant.
  • Nibble plane reuses the existing Bi4 layout and NEON nibble-expand (expand_chan_i4_i8).
  • The 5th bit is merged in-register during inflate (vtst/vsli spread + vorr), then the 5-bit value is sign-extended (vshl #3→vshr #3 arithmetic). The extra bit costs only the K/8 read; the base nibbles are read once, never re-read.

Inflate cost by consumer

consumer path extra DRAM
CPU compute int5 → int8/int16 in-register → sdot/matmul, consumed immediately zero beyond the K/8 plane read (never written)
NPU W8A8 feed int5 → int8 written to the DMA tile the NPU reads K-byte int8 write (the price of an int8-reading engine; hidden behind the swap)
NPU W4A4 (future) int4 nibbles used natively, zero-copy none — but needs the W4A4 activation-coherence research

Engine roles

Engine Precision Rationale
CPU / NEON flexible W4A5 / W5A8 Free in-register inflate → afford the sensitive weights + higher-precision activations at no memory cost. The general-purpose, most-of-decode engine.
NPU — least-sensitive tail W4A4 (zero-copy int4) Fixed + fast; lossy int4 activations tolerable because the slice is least-sensitive (placement, not a W4A4 fix). Future / needs W4A4 research.
NPU — accurate offload W8A8 (int5→int8 expand) Ships now; coherent int8 activations; Q4_K-grade weights; pays the int8 write.

The swap-hidden decode pipeline

Per token of the sparse MoE (illustrative "1000 layer calculations"):

  1. CPU starts immediately on the token's work in its flexible format (int5 weights, straight from DRAM), and works through the sensitive bulk.
  2. Little cores drive the domain swap — the current token's active experts' weights DMA into IOVA (in their stored quant) — concurrently with the big cores computing. Per-token dynamic residence (not a fixed hot set → the NPU is never idle on stale weights).
  3. By the time the swap lands (~CPU's 600/1000), the NPU fires a non-blocking submit (doorbell) on the least-sensitive slice of the remaining work (~100 of ~300), in W8A8 (or W4A4 later). No JIT precision conversion — W8A8 reads the losslessly-expanded int5; W4A4 reads int4 natively.
  4. CPU finishes the rest concurrently. Aggregate work-per-token-window ↑; the swap latency is masked by the CPU window.

The win is aggregate throughput (more work done in the same window), not per-op speed — a "0.8× per-op" NPU still subtracts its slice from the CPU's critical path if it lands in the window.

Enabler status

Piece Status
Non-blocking submit (doorbell) ✅ built + measured (130µs→5µs dispatch)
Multi-domain residence / swap into IOVA ✅ built (wedge-prone; validated fit)
int4 nibble store + free NEON inflate (Bi4, ORK_DIRECT_I4) ✅ exists
NF4 pack + free vqtbl inflate (int4-index + codebook) ✅ measured memory-bound-free (tools/int5_probe.c); int5 tested + dropped (ALU-bound M=1)
CPU computes the stored format natively (vs ggml Q4_K) 🔨 buildable (ggml-ork CPU-path wiring)
Sensitivity ranking (imatrix) for the tier split 🔨 buildable (imatrix already in the pack path)
NPU W4A4 native (zero-copy least-sensitive) ⚠️ deferred research (int4-activation coherence)

Non-goals / deferred

  • W4A4 is not required to ship — W8A8 covers accuracy; W4A4 is the zero-copy optimization for the least-sensitive slice, gated on the int4-activation coherence research (Hadamard-Int4-Accuracy: per-channel W4A4 = PPL 104).
  • The CPU move off Q4_K is not for speed — ork-native CPU is only ~parity with ggml Q4_K (measured 1.06–1.12×). It's mandatory for the single shared per-channel format the NPU can consume without a repack. Parity is fine; the win is one coherent format, not CPU throughput.

Clone this wiki locally