Exp 2026 07 14 Mode Transition Layer

Exp 2026-07-14 — Mode-transition layer (ork_npu_enter)

Status: Phase 1 landed (behavior-preserving) · Branch: feat/mode-transition-layer (off main @ 93a8595) · Board: RK3588 10.3.0.236. Related: Optimization Roadmap §Mode-transition layer · Exp-2026-07-12-NPU-Mode-Transition-Task-Descriptor · Exp-2026-07-13-Linear-Attention-Scans-on-NPU.

Motivation

The RK3588 NPU's regcmd datapath is stateful: a "mode" (precision + schedule) is tracked in software by c->last_dt, plus warm flags (warmed, per-core mwarm[]) and buffer-size caches (ccsz, per-core mccsz[]). Moving between modes can require an RKNPU_ACT_RESET ioctl (the first-int8-submit wedge guard), a cold 2-pass re-warm (fresh-output-buffer priming), and/or a buffer realloc — but not always, and the keep-warm knobs (ORK_SSM_KEEPWARM int8↔fp16, default on; ORK_MIXED_NOTHRASH int↔int, default off) suppress the churn where it's safe.

That policy had been copy-pasted inline into ~16 run/stream/chain/int4 entrypoints, each written slightly differently and quietly drifted apart. There was no single owner of "what does moving from mode X to mode Y require" — which is exactly why the int8↔fp16 keep-warm fix had to be hand-patched into 5 separate sites, and why a new interleave consumer (on-NPU softmax/attention, the chain assembler) keeps re-paying that scatter cost.

Design

One function, ork_npu_enter(c, to_marker, profile), driven by an exhaustive XSPEC[] policy table (src/npu.c). Every run path calls it first; its inline reset collapses to a single call.

Key realization that makes the NxN from→to matrix tractable: from (= c->last_dt) enters the policy ONLY through the ORK_I8_LIVE / ORK_INT_DT / ORK_KW_DT predicates. So a row is keyed by (target, caller-scope), not by an enumerated from — collapsing the matrix to one row per historical site. Modes: { COLD(-1), F16(0), I8(1), I4(2), I8_CHAIN(3), I4_CHAIN(4), I4_STREAM(5) }, plus a transient SDP (activation/ewmul, no stored marker).

A profile row is { kwp, rst, wtg, wc, stg, sc, setdt }:

• kwp — keep-warm predicate selector (KWP_MC, KWP_SC, KWP_NTI, KWP_NTL, KWP_F16, KWP_NONE).
• rst — reset condition (RC_NEVER, RC_NOTKW, RC_I8ENTRY, RC_NOTLIVE, RC_NOTLIVE_NOTKW, RC_ALWAYS).
• wtg/wc — warm-clear target (TG_SCALAR warmed / TG_PERCORE mwarm[] / TG_BOTH) and condition (WC_*).
• stg/sc — size-clear target (ccsz / mccsz[]) and condition.
• setdt — whether it writes c->last_dt.

ork_npu_enter returns 1 iff a real transition fired (0 on the from==to no-op), so the two sites that gate a caller-local warmup flag off the transition (XP_I4_INCR's warm, XP_I4_STREAM's cold) stay byte-identical. Each case in the function is a literal transcription of the predicate the original site used — so behavior is byte-identical by construction, across all knob settings, not merely at default.

The 12 profiles (16 sites consolidated)

profile	sites replaced	target	notable policy
XP_MC_MM	run_multicore	F16/I8	reset only entering I8 from non-int8-live; keep-warm MC (nothrash∨f16)
XP_SC_MM	run single-core	F16/I8	as above, f16-keep-warm only, scalar scope
XP_CHAIN_NT	run_chain_i8_impl, ork_npu_chain_progs	I8_CHAIN(3)	nothrash-only keep-warm (see drift #2)
XP_STREAM_I8	run_stream_i8	I8_CHAIN(3)	reset !I8_LIVE && !kw, clears both warm scopes
XP_STREAM_I8B	stream int8 variant	I8_CHAIN(3)	reset !kw (see drift #3)
XP_STREAM_F16	stream f16 (×2)	F16	reset !kw (stream fp16 DOES reset on entry)
XP_I4_MC	run_i4_mc, run_i4_grouped	I4	mccsz cleared on !nothrash
XP_I4_MWARM	run_i4_incr_mc, run_i4_bchain, run_i4_cbatch	I4	mwarm-only, no size clear
XP_I4_INCR	ork_mm_run_i4_incr	I4	no c-> clears (caller-local warm)
XP_I4CHAIN	ork_mm_run_chain_i4	I4_CHAIN(4)	unconditional reset + warmed + ccsz
XP_I4_STREAM	stream i4	I4_STREAM(5)	unconditional reset + both warm scopes
XP_SDP	(defined, not wired — Phase 2)	transient	unconditional reset, setdt=0

Validation (byte-identical gate)

A refactor that touches every matmul/activation path must not perturb a single output. Gate: make test on the board — exercises fp16 (test_matmul/layer/decode/model), int8, int4 (i4/test_chain_i4/perplexity_i4), stream (test_stream_interleave), int8↔fp16 SSM (test_ssd_chunk_npu), the SDP ops (test_ewmul_{i8,f16,i16}/test_silu/test_gelu/test_add), and test_mode_transition. Run once per keep-warm knob (default, ORK_SSM_KEEPWARM=0, ORK_MIXED_NOTHRASH=1). Cross-check with make mode_probe (transition matrix unchanged) and ORK_DEBUG_RESET=1 (ACT_RESET count/sites unchanged — the "28→1" keep-warm behavior intact).

Result: (filled in on landing — see the commit message / roadmap progress log.)

Phase 2 — chaining mechanism as transition state + real-pipeline findings (2026-07-14)

ork_npu_enter now takes a 4th arg chain (enum ork_chain_kind: OCK_NONE per-matmul/batch, OCK_SW run_stream_*, OCK_HW run_chain_i8, OCK_FUSED run_chain_i8_ffn static graph), recorded as c->last_chain. This makes the chaining mechanism explicit transition state so the policy can branch on it for the few handoffs where the mechanism genuinely matters — and ignore it (the common case) otherwise. Behavior-preserving (records state; no branch yet), make test byte-identical.

Real-pipeline bench via ork_bench (RK3588, governors perf, refactored ork-driver):

run	prefill tok/s	decode tok/s	XPROF	coherence
qwen3-1.7b-q8_0 default	59.99	5.89	MC_MM only	✅ coherent
qwen3-1.7b ORK_FUSE=1	63.01	5.87	MC_MM only	✅ (fusion neutral)
qwen3-1.7b ORK_FFN_CHAIN=1	68.64	0 (broke)	died before CHAIN_NT	✗ PRIME_FD domain-alloc

The sw/hw/fused chain profiles are dormant in a real dense pipeline (ork_dispatch_i8 never fires — the ggml graph interleaves rope/norm so the chain-walk never reaches ≥2 consecutive independent int8 nodes; MoE experts are CPU by default; ORK_FFN_CHAIN fails on footprint). They're exercised only by mode_probe/test_ssd_chunk_npu and narrow configs. The hunch is confirmed for the fused static graph: it chains differently (in-chain SDP/LUT, heavy footprint, fragile) and is distinguished as OCK_FUSED — not to be blindly converged with the pure-matmul hw chain. Bonus: the refactored driver ran the real model coherently end-to-end.

Phase-2 catalogue (drifted / specially-motivated sites, left for 1-by-1 evaluation)

The premise: a transition that doesn't map cleanly is probably a bug — the whole motivation for the refactor — but some drift is intentional, so each is decided individually (a one-row XSPEC edit, re-validated). Do not flip blind.

Guiding principle — which version is correct? The tests passed even with the drifted glue, so correctness cannot arbitrate. The drift itself is the bug: when two paths do the same logical transition differently, converge them to one best-performing version even if all are correct (perf + single-source-of-truth decide the survivor), whether or not the diverging path is currently exercised — after confirming they really are the same transition. Tiebreakers for the survivor, in order: (1) if candidates are otherwise byte-identical, prefer the best-performing version (bench it); (2) use the experiment logs + board-quirks data to discern/hypothesize; (3) when logs contradict, favor the most recent entry (later findings are more likely accurate); (4) when a contradiction reveals a wiki entry is wrong, fix or flag it — edit the entry to be correct (or add a dated note), keeping the wiki self-correcting.

1. SDP / activation ops — RESOLVED 2026-07-14: NOT a bug; do NOT wire XP_SDP. (Correcting this page's own earlier framing, per the self-correcting-wiki principle.) The claim that SDP leaving last_dt untouched causes the fp16-mm→SDP→fp16-mm wedge was wrong. The wedge was the c->task LUT-descriptor poisoning — a separate axis (nuance #1) — already fixed in 98c00b1; Exp-2026-07-12-NPU-Mode-Transition-Task-Descriptor states plainly "the wedge has nothing to do with last_dt," and test_mode_transition (its regression) passes. Board mode_probe (2026-07-14) confirms every SDP→matmul pair is SAFE with no reset, and that forcing one costs ~105µs/transition for zero correctness gain (SDP→int8-matmul: none ≈0.3ms vs invalidate/ reset ≈105–107ms). Flipping XP_SDP.setdt=1 would re-introduce the exact churn ORK_SSM_KEEPWARM eliminates — so don't (guiding principle #1). SDP ops correctly leave last_dt untouched; XP_SDP stays reserved/unused. Follow-up win (2026-07-14): removed the element-wise SDP entry reset. The ewmul_i8(no reset)/ewmul_f16(reset) asymmetry was investigated (ORK_SDP_NORESET gate): the fp16/int16 element-wise ops (ewmul_f16/i16, add_f16/i16) each did an unconditional ~107 ms ACT_RESET their int8 twins already skipped. Removed it — coherent (test_ewmul_f16/i16, test_add pass), wedge-free after a matmul (mode_probe SAFE), full make test green incl. in-graph SSM. Measured: test_ewmul_f16 3436→53 ms, ewmul_i16 4292→44 ms, add 1419→50 ms — a real win on the SwiGLU-ewmul / residual-add / SSM-elementwise critical path. Default on; ORK_SDP_NORESET=0 restores. (act_lut silu/gelu/exp/rsqrt already reset only conditionally.)

   SDP op →	MM_F16 (none/inval/reset ms)	MM_I8 (none/inval/reset ms)
   EXP_I16	0.2 / 0.3 / 0.3 — all ok	0.3 / 105 / 107 — all ok
   SILU_I16	0.2 / 0.3 / 0.3	0.4 / 105 / 107
   EWMUL_I16	0.2 / 0.3 / 0.3	0.5 / 107 / 107
   EWMUL_F16	0.2 / 0.3 / 0.3	0.4 / 107 / 107
   ADD_F16	0.2 / 0.3 / 0.3	0.5 / 107 / 107

   All SAFE; none (keep-warm) is fastest everywhere — bold = the chosen (current) behavior.

2. Chain profiles honor ORK_SSM_KEEPWARM — FIXED 2026-07-14 (★ ~105 ms/transition win). XP_CHAIN_NT (KWP_NTL+RC_NOTLIVE) ate a full ACT_RESET when a chain followed an fp16 op, where the stream profiles kept warm. Since the path is dormant in a dense pipeline, a manufactured fp16→int8-chain sweep (chain_xition_probe, wide K/N/M) measured it: chain(hw) reset-cost ~53,538 µs avg (individual 104–107 ms) vs stream(sw) ~0, both coherent → the reset was pure drift. Converged to KWP_MC+RC_NOTLIVE_NOTKW; re-run: chain(hw) reset-cost → −1.1 µs, all coherent, make test green.

3. XP_STREAM_I8/I8B unified — FIXED 2026-07-14. Gratuitous (non-default-knob) drift; converged to the same policy, collapsed to one profile (XP_STREAM_I8), XP_STREAM_I8B removed.

4. Diagnostic forced-fp16 resets (ork_ssd_fused, ork_ssd_probe_rawmm/fusedmm_f16) — ASSESSED: intentional unconditional reinit in board-only diagnostics, not a precision transition; left as-is.

5. Conditional prime resets (ORK_I16_RESET; fused-SiLU LUT !ORK_I8_LIVE || ORK_PROBE_RESET) — ASSESSED: correct as-is; env/warm-conditional and correctly leave last_dt (consistent with the SDP finding above). Not drift.

6. int4 clear-gating — RESOLVED: mechanism-legit, not a bug. Built tools/i4_xition_probe.c (manufactures {fp16,int8,int4-live}→int4 across a wide K/N/M sweep). All four int4 mechanisms — run_i4(XP_I4_MC), run_i4_incr(XP_I4_INCR), run_chain_i4(XP_I4CHAIN), run_stream_i4(XP_I4_STREAM) — are 36/36 coherent across every predecessor/shape. XP_I4_MC clears mccsz to resize its persistent per-core output on an int4 entry; the batch/incr paths size buffers per call — a legitimate buffer-lifecycle difference, not gratuitous drift ⇒ no convergence. (Aside: an earlier probe variant showed a repeated back-to-back ork_mm_run_chain_i4 miscompute — a possible re-warm gap on consecutive chain_i4 calls, distinct from the transition question; flagged for the experimental int4-chain.)

Test-suite + build + CI (2026-07-14)

Infrastructure that landed alongside the transition work:

• Static golden-checksum tests. Inputs are fixed-seed deterministic ⇒ the NPU output is a constant, so the reference-bound tests assert an fnv64 checksum of the NPU output vs an embedded golden instead of recomputing the O(M·N·K) CPU reference each run; the reference is kept for regen/diagnosis (ORK_REGEN=1 / ORK_FULL_REF=1). test_matmul 195→5 s, quant 39.8→1 s, test_sn3 7→1 s, model 12 3.1→0.4 s; full make test ~10-11 min → ~33 s.
• Manufactured-transition probes. chain_xition_probe (sw/hw chain) and i4_xition_probe (int4) — exercise the transition paths that are dormant in a real pipeline; chain_xition_probe is in make test.
• Build. CORE (npu.c ~9.7k lines) is compiled once into shared -fPIC objects (was recompiled ~30× inline), and CC is wrapped with ccache — full build ~5 min → ~20 s, incremental npu.c-edit rebuild ~9 s.
• CI board-validation gate. ork-driver's tests need the NPU (CI runners lack it). make test on the SBC writes tests/sbc_attest.txt = sha256 of the NPU-output-determining .c sources + golden-bearing tests; CI make check-attest (no NPU — sha256 + grep) fails if the tree hash differs, catching a commit not board-validated before push. version-bump.yml runs it before the bump (attestation precedes the version bump); .github/workflows/sbc-attest.yml gates pushes/PRs. Details: AGENTS.md §"Mode-transition layer" + §3.

How to modify / add a transition

See AGENTS.md §"Mode-transition layer". Changing a policy = editing one XSPEC[] row; adding an op/dtype = a new XP_* row + one ork_npu_enter(c, marker, XP_*, chain) call. Validate on the board (make test), then commit the refreshed tests/sbc_attest.txt so CI passes.