Exp 2026 08 18 NF4 CPU Prefill

Exp-2026-08-18 — NF4 experts on CPU beat native at MoE prefill (1.33×)

Repo: ggml-ork (backend) + ork-driver (ork_native_cpu.h batched kernel) Model: Qwen3.6-35B-A3B (hybrid SSM+MoE), RK3588 (10.3.0.236), -t4, governors=performance. Status: PROVEN (prefill). Correctness validated (ork_bench). Decode is out of scope (see caveat).

Hypothesis

Prefill is compute-bound on expert dequant (established: Optimization-Roadmap MoE-W4A4 wall — CPU MoE reads weights ~once/batch ≈ 1 GB/s ≈ 5 % of the bus). If so, a cheaper expert unpack on the CPU should beat native ggml Q4_K at prefill. NF4's 1-op vqtbl table-lookup unpack is cheaper than Q4_K dequant, and it needs no Hadamard rescue. Store experts as NF4-int4 resident in the orkpack; run them with a batched (M>1) NEON GEMM; keep attention on the NPU (int8).

Method

• Kernel (ork-driver/include/ork_native_cpu.h): added ork_cpu_gemm_i4 / ork_cpu_gemm_nf4 (4×4 register-blocked GEMM, weight+unpack amortized across M rows; NF4 unpack = vqtbl1q_s8(lut,·), int4 = sign-extend). Probe examples/test_i4_gemm.c validates vs the M=1 gemv.
• Backend (ggml-ork.cpp): run_cold batched cold-path (gathers each cold expert's M_e routed rows → one ork_cpu_gemm_{i4,nf4}, ofmt from the O4N1 quant_kind header); gated flag ORK_MOE_CPU (routes every expert to the CPU cold path + claims MUL_MAT_ID in supports_op).
• Pack recipe (the crux): build the orkpack from the f16 gguf with ORK_ORKPACK_I4_FFN=1 ORK_MOE_CPU=1 and no ORK_QUANT → only ffn/expert tensors go int4 (attn/dense stay int8, source-driven), and an f16 source auto-selects the NF4 codebook (nf4 = ork_src_type_bits ≥ 16, persist path). Result: qwen3.6-35b-a3b-nf4.orkpack (18 GB, 31010 weights). Load with the same flags (build-sig must match; sig=0 with no ORK_QUANT).

Results

Kernel probe (test_i4_gemm, on-board NEON): bit-exact vs the M=1 gemv; I4 2.3× / NF4 1.85× (NF4 faster in absolute time — 1-op unpack). End-to-end llama-bench, mixed.gguf base, -t4, r=3, performance governor, same board:

config	pp64 (t/s)	pp512 (t/s)
native ggml mixed (experts Q4_K-CPU, attn f16→NPU)	24.18	27.20
NF4 route (experts NF4-CPU + attn int8-NPU)	31.18	36.19
speedup	1.29×	1.33×

Correctness (ork_bench mixed.gguf "What color is the sky? Answer:" 32 12): generated "Blue" — coherent. [ork COLD-GATE] ork_native=1 ofmt=1(NF4) batched=1 confirms the path executes.

Conclusion

NF4 experts on CPU + int8 attn on NPU is a validated 1.33× prefill win over native on the 35B — above the 22 t/s task-#54 target and above the prior ~28 native ceiling. It confirms the compute-bound-dequant model: the win is the cheaper NF4 unpack + resident (no per-batch re-dequant) experts, not bandwidth.

Caveats / scope

• Prefill only. ORK_MOE_CPU forces experts to CPU at every M; at decode (M=1) it measured 3.99 t/s vs native ~6 t/s — a regression (no rows to amortize). The flag must be made phase-aware (batched-CPU only at M ≥ prefill threshold; native/NPU at decode) before it can be a default.
• False-negative avoided. An initial attempt built the pack with ORK_QUANT=4, which forces all tensors int4 incl. dense → dense int4-on-NPU hit the bcreate errno=14 multi-domain IOVA wall (crashes at M=512), dragging the contaminated pack to 6.17 t/s and a wrong "not a win" verdict. The ORK_ORKPACK_I4_FFN recipe (experts-only int4) is what surfaced the real result. (scrutinize-negative-results)
• Accuracy (PPL) not yet measured — NF4-vs-native perplexity is the open follow-up.
• Governors reset on reboot (DDR → dmc_ondemand); re-pin before timing.

Artifacts

• kernel: ork-driver/include/ork_native_cpu.h, probe examples/test_i4_gemm.c
• backend: ggml-ork.cpp run_cold batched path + ORK_MOE_CPU
• pack: qwen3.6-35b-a3b-nf4.orkpack (board)

Landed

• ork-driver main 96efa7a (batched CPU GEMM kernel + int4 nonblock fix + attest) — make test ALL PASS.
• llama.cpp-rockchip diverged-history 4cf8661d0 (ORK_MOE_CPU route + footprint residence; submodule → 96efa7a).
• Stashed sibling experiments (biglittle, coalesce, gptq, mpad/m1npu): see the linked entries; restoration patches in attachments/ + snapshot branch wip/moe-saga-2026-08-18 (both repos).