Skip to content

Building the Regcmd Graph

Jump to bottom
Michael Fischer edited this page Jul 12, 2026 · 2 revisions

Building the Regcmd Graph — Segment / Boundary Methodology

How to turn any transformer model, on any RK-family NPU, into the fewest-submit on-NPU regcmd graph the hardware can run. This is the general technique; the Qwen3-1.7B worked example at the bottom is the current concrete target.

The prefill parity gap with librkllmrt is a submit / host-round-trip problem, not a compute problem (the NPU sits ~83% idle during prefill — Performance). rkllm wins with a precompiled static regcmd graph: the whole model's NPU work is laid out once and submitted in a handful of large, chained pieces, escaping per-node dispatch. This page is the recipe for building that graph ourselves.

The core idea: segments and boundaries

Walk the model's compute graph. It is a sequence of ops; some run on the NPU (matmuls), some must run on the CPU (softmax, norms, elementwise). Then:

  • A segment is a maximal run of NPU-eligible ops with no CPU op in between. The hardware can walk an entire segment as one submit (PC-chain, task_number > 1).
  • A boundary is the CPU op (or hard split) that ends a segment. Boundaries are what force separate submits.

A layer's submit count = (number of boundaries that can't be removed) + 1. So the whole game is: classify every boundary, then remove the ones that aren't load-bearing.

Boundary taxonomy (the column that matters)

class meaning examples action
hard no NPU handler; needs reverse-engineering to move softmax (PPU LUT/reduction) RE it, or accept the split
movable cheap CPU op that could run on the NPU/PPU, or fuse/reorder RoPE, RMSNorm, residual add, SiLU/GLU move on-NPU → boundary disappears
free view op, zero compute — only breaks contiguity/backend assignment reshape, permute, transpose reorder around; never a real cost

The insight that makes this tractable: most boundaries are movable or free. In a standard transformer layer only softmax is genuinely hard. Collapse the rest and the layer folds to (#hard) + 1 mega-segments — for a normal attention+FFN layer that's two, pivoting on softmax.

The procedure

1. Dump the real graph

Never work from memory of "how a transformer is shaped" — dump the actual node order the runtime produces (fusions, view ops, and backend splits are not obvious):

ORK_DUMP_GRAPH=1   → per-subgraph node list (op, src0/src1 names, ne[])   [ggml-ork]
ORK_ATTN_TRACE=1   → every batched attention matmul's shape + operands    [ggml-ork ORK_ATTN]

Record each MUL_MAT's K = src0->ne[0], N = op->ne[0], M = op->ne[1], and ne[2]/ne[3] (>1 ⇒ batched/dynamic-operand, i.e. attention/GDN, not a static weight).

2. Cut segments

Group consecutive, independent NPU matmuls into one segment. Independent = neither reads the other's output (Q/K/V share the norm input; gate/up share the FFN-norm input). A dependency or any CPU op ends the segment.

Watch two things the graph won't spell out:

  • Non-consecutive siblings. Q/K/V may be interleaved with RoPE/reshape in graph order, breaking a naive adjacent-only grouping window — you need scan-ahead past view/movable ops to gather them.
  • Internal K-split ≠ boundary. A matmul whose contraction K exceeds the SoC's single-submit ceiling K-splits into several programs (still NPU, no CPU in between) — it's an internal multi-submit, not a segment boundary.

3. Classify every boundary

For each CPU op between segments, label it hard / movable / free (table above). This is the map of what to attack. Only the hard ones gate the final submit count.

4. Chain each segment (the mechanics)

  • HW-chain within a segment via ork_mm_run_chain_i8(S, tasks) — one ioctl, task_number = S, the hardware walks all programs. Per-weight envelope: Sn == 1, K % 512 == 0, K ≤ 4096 (else the weight K-splits and can't single-chain). It is single-core (one PC-chain on one core).
  • Keep the whole graph in one mode. Mixing chained (DT_I8_CHAIN) and plain (DT_I8) matmuls used to force a re-warm on every transition (a per-layer thrash that made chaining a net loss). Fixed by keeping warm across DT_I8 ↔ DT_I8_CHAIN under ORK_MIXED_NOTHRASH (ork-driver run_chain_i8). Rule: route all of a layer's matmuls through one dispatch mode.
  • Kill the per-matmul bsync. One submit ≠ few ioctls — each task still cache-syncs its A and C. Put the segment's shared input in a resident DMA buffer (ork_dma_alloc) so run_chain_i8's dma_find hits (no per-task bcreate/memcpy/bsync/bdestroy) and the shared input syncs once (the bsync-dedup in run_chain_i8). This is what turns "one submit" into "few ioctls".
  • SW-chain across boundaries. Segments separated by a boundary submit sequentially, but from a precompiled regcmd (weights + activation buffers at fixed IOVA, built once, re-submitted per forward) — escaping ggml's per-node dispatch. The CPU boundary op runs between the submits.

4b. Heterogeneous chaining — matmul → SDP (activation) in one submit (validated 2026-07-12)

A "movable" boundary op (SiLU, GLU, RMSNorm) does not disappear by fusing it into the preceding matmul's output stage. It disappears by becoming its own task in the same PC-chain, right after the matmul. Three rules, all confirmed on RK3588:

  • Data flows task→task by ADDRESS ALIASING, not a forwarding datapath. Each task writes its result to a scratch IOVA (matmul/DPU output 0x1001:0x4020; CNA feature output); the next task names that same IOVA as its input (SDP 0x2001:0x5018, second operand 0x2001:0x5038; CNA feature 0x0201:0x1070). There is no "chain the output to the next input" register — the PC descriptor (0x0101:0x0010 next-addr, 0x0101:0x0014 next-reg-amount =(regcfg+3)/2) only sequences which regcmd runs next; the data moves through shared buffers in a persistent image, with no act(RESET) between tasks. It's a DAG (a task's output can feed two later tasks), not a linear pipe. This is exactly how the vendor 9-task softmax chain works (decoded: t2 CNA-matmul → t3 SDP reads t2's output IOVA). See Exp-2026-07-10 Forward Softmax.
  • A FUSED-ACTIVATION matmul output stage will NOT chain. A matmul whose output stage applies an activation LUT (set_i8_silu = int8 requant + silu LUT) wedges as any non-terminal task in a chain (and even as the terminal task in run_chain_i8). It only runs standalone (ork_mm_run_i8_silu, via submit1's warmed c->Cc). But a matmul with a plain int8 output stage (set_i8_out8, requant only, no LUT) chains fine. Rule: never fuse the activation into a chained matmul's output — emit plain int8 and put the activation in the next SDP task. (This mirrors the vendor: matmul tasks are register-config-only; activation is always a separate 0x18 SDP task. See Exp-2026-07-10 Fused-Output-Activation-No-Crossing-Chain.)
  • Build on run_chain_i8, don't hand-roll the submit. A from-scratch chain submit to fresh buffers returns rc=0 but computes nothing (documented "hand-rolled submit to a fresh buffer gives bias-only output"). run_chain_i8's warmed-buffer + warm-up-reps + all-three-subcore_task recipe is the validated path; extend it (add SDP/activation tasks) rather than reimplement. Ping-pong OFF (flags=0x1) for any chain containing an SDP/LUT task; the silu LUT is streamed to SDP SRAM once before the chain.

Validated primitive: ork_mm_run_chain_i8_sdpsilu chains [gate matmul(int8-out via set_i8_out8) → standalone silu-SDP(reads gate output, aliased)] in one submit, bit-coherent (gate int8 exact, silu int8 exact, no wedge). The int8 matmul output cube matches the SDP EWCUBE(m,n)=((n)/16)·(M·16)+(m)·16+((n)%16) input — no repack. This is the reusable pattern for every matmul→activation hop in a segment.

Dead ends ruled out: int16-output matmul (set_i16_out) is multi-register-broken with no captured reference (the only working 2-byte producer is fp16) — don't pursue it; the int8-out + standalone-SDP path is the working bridge. The production ORK_FFN_CHAIN int16-silu accuracy path requants int32→int16 on the CPU between the matmul and the silu (not a pure-NPU bridge) — confirming int16-out was never load-bearing.

5. Dispatch discipline (don't mis-route dynamic operands)

Attention matmuls (ne[2]/ne[3] > 1) have dynamic operands (KV-cache activations), not static weights. Never let them fall into the int8 weight-resolve path (it tries to pack the KV activation as a weight → slow live-convert). Claim them only for M > 1 (prefill) via the fp16 batched path; leave M == 1 (decode) on the CPU (decode is a CPU path anyway). See the dedicated-dispatch fix (supports_op gates batched matmuls on M > 1).

6. Validate every step

  • Coherence: bit-exact vs the CPU reference at the matmul level; ork_ppl at the model level. A "win" that shifts perplexity is not a win.
  • No wedge: gate on errno == 110, not dmesg RKNPU: soft reset (the deliberate ACT_RESET logs the same string — see NPU-Quirks).
  • Measure the right thing: the ORK_PROFILE submits: counter is submit-ioctls only; it does not show the bsync/mem-ioctl reduction. Judge by wall-clock and total ioctls, not submit count.

Portability — what changes per model / NPU, what doesn't

Constant (the methodology): segments, the boundary taxonomy, the "remove movable boundaries → #hard + 1 segments" principle, validate-coherence-and-errno=110. Varies, and must be re-derived:

  • Shapes — hidden K, ffn Nff, head_dim, head count / GQA ratio → which weights K-split, which siblings group.
  • SoC single-submit K ceiling — the K-split threshold (RK3588 int8: K ≤ 4096, K % 512) lives in src/soc/<chip>.c; re-measure per NPU (ksubmit_probe).
  • Which ops have NPU handlers — e.g. softmax is unhandled today; a SoC/driver with a usable PPU softmax turns that hard boundary movable.
  • Dtype modes — int8/int16/fp16 availability and their mode-transition costs (this SoC: int16 and fp16 memory outputs differ from int8; INT8→FP16 fused is not a datapath — see regcmd ISA Reference).

Worked example: Qwen3-1.7B, int8 prefill

Dumped graph (ORK_DUMP_GRAPH + ORK_ATTN_TRACE). Shapes: hidden K=2048, Nff=6144, head_dim=128, 16 heads (GQA n_kv<16). Six segments, six boundaries:

# Segment Regcmd ops Shape K→N Dispatch ↓ Boundary after class
S1 QKV projection attn_q · attn_k · attn_v 2048 → 2048 / n_kv int8, HW-chain ×3 RoPE (Q,K) movable
S2 QKᵀ scores Q·Kᵀ per head 128 → n_kv fp16 bmm (M>1) softmax hard (RE)
S3 attention·V softmax·V n_kv → 128 fp16 bmm (M>1) reshape/permute free
S4 output proj attn_output 2048 → 2048 int8 residual + RMSNorm movable
S5 gate + up ffn_gate · ffn_up 2048 → 6144 int8, HW-chain ×2 (ORK_GU_CHAIN) SiLU + glu movable → chain as separate SDP tasks (§4b)
S6 down proj ffn_down 6144 → 2048 int8, K-split (K=6144>4096) residual + RMSNorm movable

FFN inner as one chain (per §4b — activation is its own task, never fused into the matmul output): [gate matmul(int8-out) → silu-SDP → up matmul(int8-out) → glu-ewmul-SDP(silu-out, up-out) → down matmul(K-split)]. The [gate → silu-SDP] hop is validated one-submit + bit-coherent (ork_mm_run_chain_i8_sdpsilu); the rest is the same two primitives (plain int8 matmul; standalone SDP op) with data via aliased buffers.

Only softmax is hard. Rebuild target — two mega-segments pivoting on softmax:

  ┌─ MEGA-SEGMENT A (pre-softmax) ───────────────┐        ┌─ MEGA-SEGMENT B (post-softmax) ──────────────────┐
  │  QKV → [RoPE] → QKᵀ                          │  soft  │  A·V → [reshape] → O → [add+norm] →              │
  │  (RoPE moved on-NPU/PPU or fused;            │  max   │  gate·up → [SiLU on-NPU] → down(K-split) →       │
  │   QKV HW-chained; QKᵀ batched-over-heads)    │ (CPU/  │  [add+norm]  — all movable boundaries removed    │
  └──────────────────────────────────────────────┘  2b)  └───────────────────────────────────────────────────┘

Per-layer submits: 6 → 2 once the movable boundaries (RoPE, ×2 RMSNorm+residual, SiLU) move on-NPU. The single hinge is softmax (2b, PPU RE) — the only load-bearing boundary.

Status of the enablers (feature branches; main untouched)

  • run_chain_i8 keepwarm across DT_I8↔DT_I8_CHAIN — kills the chain thrash (ork-driver feat/static-graph). Keeper.
  • run_chain_i8 bsync-dedup for shared DMA buffers (ork-driver feat/static-graph).
  • ggml-ork DMA-scratch for the gate+up segment (ORK_GU_CHAIN) + dedicated attention dispatch (supports_op claims batched matmuls only for M>1) — fork feat/ork-static-graph.
  • Measured: each enabler is coherent; matmul-only chaining is neutral on its own (prefill is CPU-bound). The win requires collapsing the movable boundaries — i.e. building mega-segment B and cracking softmax for the A|B pivot.
  • Heterogeneous chaining validated (§4b, 2026-07-12): ork_mm_run_chain_i8_sdpsilu — one-submit [gate matmul(int8-out) → standalone silu-SDP], bit-coherent, on run_chain_i8's proven engine. This is the mechanism that actually collapses the SiLU/GLU/norm movable boundaries into a segment (as separate SDP tasks, not fused matmul outputs). Extends to the full FFN inner [gate → silu → up → glu → down].

Next

  1. 2b — softmax on the PPU (the hinge). Re-source a softmax .rknn capture (.239 box) to interpose; prior attempts stalled on the interposer's 4KB buffer-dump truncation (see NPU-Quirks).
  2. Move RMSNorm + RoPE on-NPU (movable boundaries → mega-segment closure).
  3. Batch the fp16 attention bmm over heads (currently one submit/head).
  4. Precompile + cache the two-segment regcmd per layer (escape per-node dispatch).

Clone this wiki locally