Skip to content

K Grouping Cost On RK3588

Jump to bottom
Michael Fischer edited this page Oct 8, 2026 · 3 revisions

K-axis scale granularity on RK3588 — why it costs 23.6×, and what that means

Desk research, 2026-10-08. No board time used; the two load-bearing claims were verified against source independently of the research that produced them.

Headline: the 23.6× is NOT a hardware tax. It is host-side implementation overhead, and most of it is one line. By every NPU-side metric the grouped path is cheaper than the per-channel baseline it is compared against.

1. The grouped path does LESS NPU work, not more

At M=128, K=3584, N=1024, G=128 (so Sk=28), both arms reach BCHAIN, but every BCHAIN parameter is a function of the program's K — which for the grouped arm is G=128, not 3584:

baseline (K=3584) grouped (28 × K=128)
H = orki_i4_hcap(K) rows/weight-stream 4 128
Wb = (131072/K) & ~63 64 1024 (= N)
total NPU programs 512 28
weight DMA 58.7 MB (full weight × 32 re-streams) 1.84 MB (× 1)
output cells written 0.26 MB 7.34 MB

Total weight bytes ≈ max(M·K²·N / (32768·Sk), K·N/2) — quadratic in the program's K and falling as 1/Sk. Splitting K into 28 groups cuts weight re-streaming 32×, because each group's small K buys a 32× larger row budget H.

18× fewer dispatches and 32× fewer weight bytes cannot produce a 23.6× slowdown. This conclusion is robust to every uncertainty in the NPU timing model.

2. Where it actually goes — the host side

need_o = NT·(4·H·Wmax)·64·2 = 29.36 MB of output scratch against 7.34 MB of cells the de-tile touches (a 4× stride-sparse surface). Baseline: 1.05 MB / 0.26 MB.

Per call the grouped path then performs: a GEM alloc + IOMMU map + first-touch of ~7,200 pages; a scalar SENT16 seed over 7.34 MB; two orki_bsync sweeps at the full 29.4 MB; a verify sweep; a scalar int16→int32 de-tile writing 14.68 MB into P; and then a second full pass over P for the fp32 drain. ≈110 MB of host traffic and a per-call allocator storm, vs ≈3 MB for the baseline.

The specific defect

src/npu/i4/chain.c:689, at the end of orki_i4_run_experts_bchain_db:

for(int i=0;i<nc;i++){ if(c->mcc[i].cpu){ orki_bdestroy(fd,&c->mcc[i]);
    c->mcc[i]=(struct buf){0}; c->mccsz[i]=0; c->mwarm[i]=0; } }

This is a #54 MoE multi-domain footprint fix — correct for the MoE expert-coalesce caller, which switches domains and would otherwise park ~1.6 GB of scratch. The grouped-int4 caller never switches domains and calls this in a tight loop, so it pays a full bdestroy + bscratch of ~29.4 MB on every call (line 665 reallocs, because mccsz was just zeroed). The baseline has no such free and allocates once.

Raw bytes account for maybe 2–3× of the 23.6×; the residual is page allocation, IOMMU PTE population, minor faults on 7,200 fresh pages and TLB work, all serialised in the caller.

The A/B, run 2026-10-08 — CONFIRMED, ~1.9×

Gating the free on c->dom_save (the established single-vs-multi-domain signal), ORK_I4_FREE_MCC=1 restoring the old behaviour so one binary gives both arms. M=128 K=3584 N=1024 G=128, 20 iters, both arm orders because arm position on this board is worth up to 1.126×:

arm real ork_i4_mm_run_grouped × per-channel
old (free every call), run first 74.07 ms 23.86×
old, run second 74.39 ms 24.26×
fixed, run second (disadvantaged) 37.35 ms 12.07×
fixed, run first 40.55 ms 13.15×

24× → 12×, and it wins from the disadvantaged position, so this is not position bias.

Two caveats. The probe's headline "TOTAL per-group" barely moves (28.8 → 28.1) because that figure is dominated by a different arm — the naive 28-separate-submits path — not the one fixed here; read the REAL ork_i4_mm_run_grouped line. And the ORK_I4_H=64 secondary measured 37.03 ms, identical to the fix alone: the 4× over-allocation of need_o is NOT the next binding term. Allocator churn was the whole of it — once the buffer is cached, its size stops costing anything per call. So fix #2 below can be dropped, and the remaining ~12× has to come from #3 (fuse the drain, delete P) and #4 (NEON the seed and de-tile), or from somewhere not yet itemised.

The remaining host fixes, measured 2026-10-08 — 24× → 9.7×, and host work is now ~22%

REAL ork_i4_mm_run_grouped, M=128 K=3584 N=1024 G=128, 20 iters:

ms × per-channel
original 74.07 24.26×
+ mcc fix (don't free single-domain) 37.35 12.07×
+ NEON seed / verify / de-tile 29.13 9.70×
+ batched drain, batch 4 30.92 10.24×
+ batched drain, batch 8 30.53 10.16×

Correctness: bchain worst 0 (0 bad) against the probe's exact CPU reference — bit-exact. Read the probe's "CORRECTNESS … FAIL" line carefully: it compares bchain against the ROW-DECOMPOSED path, and it is the row-decomposed path that is wrong (0.00274 worst, 380 cells, vs the exact reference).

The batched drain is a measured NEGATIVE and stays off (ORK_I4_GRP_BATCH). The extra chain launches cost more than the L3 locality buys, because the reduction it targets is only ~5% of the total — the probe says so directly: "of which reduction is 5% — DISPATCH dominates". That also retrospectively justifies NOT doing the fused-scale version of fix #3, which would have reassociated the sum and changed the expert descriptor shared with the in-use MoE path, all to attack a 5% term.

Host-side optimisation is now essentially done. Of the remaining 9.70×, the one-doorbell-submit chain alone is 7.47× (22.6 ms of 29.1 ms), so ~78% is dispatch/NPU and only ~22% is host. Further progress needs the doorbell/heterogeneous-overlap work, not more host tuning.

The A/B as originally specified

Add ORK_I4_KEEP_MCC=1 to skip that free (safe — the grouped caller is single-domain) and run the group probe both ways, both orders (see ab-arm-position-bias). Secondary: ORK_I4_H=64 halves need_o without changing the cells written, which separates the 4× over-allocation from the Sk-fold.

Fixes, in order

  1. Don't free mcc when the caller is single-domain — ~10 lines.
  2. Size and bsync the touched sub-range, not 4·H·Wmax — 4× off both sweeps and the allocation.
  3. Fuse the fp32 scale-accumulate into the de-tile and delete P — removes ~29 MB of traffic.
  4. NEON the seed and the de-tile (the seed is currently a scalar per-element store loop).

Estimated 23.6× → 3–5× with zero hardware risk.

3. Hardware mechanisms — one hard negative, one live lead

CACC partial-sum retention: architecturally absent. Do not probe it. On RK3588 the CACC CORE block has 8 registers and Rockchip removed NVDLA's CACC_D_DATAOUT_ADDR/LINE_STRIDE/ SURF_STRIDE — the accumulator has no memory port; results can only flow forward into the DPU. Neither NVDLA's compiler nor Mesa's rocket ever splits along channels; both split H only and re-stream weights under CBUF pressure.

DPU/SDP element-wise SUM: real, and Mesa uses it on this silicon. DST = out_cvt(CACC_result ALU_SUM ew_cvt(mem[EW_BASE_ADDR])), same task, flying mode: DPU_EW_CFG 0x4070 (EW_OP_TYPE=0, EW_ALU_ALGO=2 SUM, EW_OP_SRC=1 MEM), DPU_RDMA_ERDMA_CFG 0x5034, DPU_RDMA_EW_BASE_ADDR 0x5038, EW_SURF_STRIDE 0x5040. Blocked on three things: the OPS_REGISTRY row for ork_i8_mm_run_ewmul is DEAD — WEDGES, but that graft was onto an int8-out program and the sibling failure was later resolved as a dtype-path mismatch, so the verdict is precision-confounded and an int32-out graft was never tried; EDATA_SIZE=3 (4-byte operand) is untested; and it does not remove the Sk-fold DRAM traffic, only moves it off the host.

4. Algebra — what is free and what is not

  • The N-independent part of a group scale is free. s[g(k),n] = σ[g(k)] folds into A column-wise. This is exactly SmoothQuant/AWQ, and a per-K-channel σ is strictly more expressive than per-group. We already have K-axis granularity finer than g=128 — it just has to be N-independent.
  • Rank-1 factorisation of the group-scale field does NOT work. It looked like 95% of the g128 gain in weight-Frobenius terms; with a proper control that was traced to the SVD scale implicitly clipping. True contribution 0.985–0.995×, i.e. nothing.
  • Per-group zero-point is free to emulate (a rank-Sk matmul, 0.8% of the MACs) and worth 0.4% alone — LLM weight groups are near zero-mean. Q4_K's asymmetric gain is level utilisation, which is only realisable together with the group scale. The free half carries no information without the expensive half.
  • Power-of-two / MX shared exponents: rule out. Optimises host arithmetic we do not pay for, and costs ~40% relative MSE.
  • No exact low-rank equivalent of group scaling exists — q ⊙ S is a Hadamard product, not additive. The literature fits a rank-r adapter to the quantisation error instead (LQER, LQEC): ~1.6% extra MACs at r=32, but price the 2 extra programs at the M=1 regime's 9–13 µs.

5. The cost-vs-Sk knee is real, at Sk≈7

Weight traffic falls as 1/Sk; partial traffic rises linearly in Sk. Minimum at Sk* = sqrt(M·K / (16384·c')). For M=128, K=3584, N=1024 the knee is Sk≈7 (G=512), total 14.7 MB — 3.5× cheaper than the per-channel baseline's 52 MB. G=128 is 9.6 MB of NPU work. Once §2's host fixes land, G=512 should be at or below per-channel cost.

6. …but the quality case for grouping is weak here

Measured offline on real Qwen3.5-0.8B bf16 weights with LLM-like synthetic activations, metric = relative error of C = A·Wᵀ vs fp32, A per-row int8:

scheme ×per-channel cost
per-channel absmax 1.000 1.0×
per-channel + clip search 0.637 1.0× — FREE
SmoothQuant α=0.5 0.876 free
g512 0.893 7 passes
g128 0.769 28 passes
asym128 (≈Q4_K) 0.670 28 passes + zp

A free per-output-channel clipping search beats group-128, and beats Q4_K-style asym128. At true W4A4 (A per-row int4) weight granularity barely matters at all — g128 buys 0.956× because activation quantisation dominates; the only large lever is SmoothQuant migration at 0.555×.

QuaRot agrees and quantifies it: Llama-2 W4A4 with rotation+GPTQ fixed, per-channel 6.10 vs g128 5.93 (grouping worth 0.17 PPL) while RTN→GPTQ at per-channel is 8.37→6.10 (the solver worth 2.27 PPL, 13× more). Grouping and rotation/solver quality are substitutes.

The caveat that could overturn this: QuaRot Table 11 on Llama-3 has per-channel 8.16 vs g128 7.36 — 0.80 PPL, and 1.15 at 70B. The "rotation replaces grouping" result is Llama-2-shaped, and Qwen3 is the Llama-3 generation. The measurement above is one model, synthetic activations, and an error proxy rather than perplexity. Treat as a strong hypothesis, not a budget.

7. What this immediately bought

src/ork_gptq.c's clip grid stops at 0.781, which is above the per-row optimum. Verified independently (K=1024, N=64, per-row, heavy-tailed weights, structured H): H-weighted error 3796.8 at floor 0.781 vs 2526.6 at 0.531 (1.50×), with the chosen scale still sliding at floor 0.406 — the search is pinned against its boundary for essentially every row. Invisible at group=32, the only granularity test_gptq exercises. Made sweepable in ork-driver 1.0.157 (ORK_GPTQ_CLIP_N/_STEP); default unchanged pending a PPL arm.

Separately, src/npu/i4/pack.c:71 gates the existing clip-grid scale search behind if (imatrix), so without an imatrix the native int4 pack uses plain absmax — the 1.000× row above.

8. Doc debt

src/ork_regs.h:189-190 marks 0x5028/0x5034/0x5038/0x5040 "inferred, name unconfirmed". rocket_registers.h names them NRDMA_CFG, ERDMA_CFG, EW_BASE_ADDR, EW_SURF_STRIDE (plus 0x5018, 0x501c, 0x5020, 0x502c, 0x5044, 0x5068, 0x506c). Mesa's residual-add also needs both M-RDMA and E-RDMA with COMB_USE(5) and a W·H·16 inter-stream offset — undocumented and not obvious; capture it on NPU-Quirks before anyone attempts the EW-SUM graft.

Clone this wiki locally