Exp 2026 07 12 NPU Mode Transition Task Descriptor

2026-07-12 — ★★★★ NPU mode/state-transition RE: the exp/silu→matmul wedge is a stale shared task descriptor, not a dataflow law

Commit: ork-driver feat/ssm-npu · src/npu.c (ork_npu_probe_silu_std_i16 c->task save/restore fix; mm_timeout_ms / ORK_MM_TIMEOUT; ork_npu_mode_invalidate / ork_npu_mode_reset hooks + header decls) · tools/mode_probe.c (make mode_probe) · reproducer examples/test_ssd_chunk_npu.c. Board: RK3588 (10.3.0.236). See NPU-Quirks — LUT-activation op poisons the shared c->task descriptor.

The problem

Building the whole-elementwise on-NPU SSD/Mamba-2 scan (test_ssd_chunk_npu), mode 2 (cumsum→CumBA bmm, decay exp→int16 LUT, decay multiplies→ewmul, all on NPU) reproducibly wedged with errno=110 (ETIMEDOUT; dmesg: RKNPU: job timeout … task counter: 0x0 → soft reset) on the fp16 CumBA bmm (K=64, N=16) mid-sequence. Yet:

• mode 1 (matmul spine + decay exp on NPU, same exp→bmm transition) ran clean (rel-L2 5.9e-3);
• the CumBA shape was bit-close in isolation (tools/ssd_cumba_exp.c);
• the wedge reproduced with both int16 and fp16 ewmul → not an ewmul-dtype issue.

So the wedge was contextual, and the standing hypothesis (from the earlier Activation→matmul ordering quirk) was that a standalone SDP op → separate matmul submit is unsafe because NVDLA's pipeline (MAC→SDP→…) has "no backward path." This experiment refutes that for the standalone cross-submit case and finds the real, concrete cause.

Method — a 2-op transition probe

tools/mode_probe.c runs op A then op B as separate submits on a warm context and reports whether B wedges (rc≠0 AND wall ≈ the submit timeout) — sweeping {matmul: MM_F16 64×64×16 (the CumBA shape), MM_I8 64×64×64} × {SDP: EXP_I16, SILU_I16, EWMUL_I16, EWMUL_F16, ADD_F16}, both directions. Board safety was the gating constraint (this task induces wedges): a new ORK_MM_TIMEOUT knob makes every matmul-path submit fail fast (~2 s) instead of the 60 s job timeout, each test runs as its own process under timeout (SIGTERM only), and the board self-heals on process teardown between tests. A one-shot mode_probe A B fix mode runs a single (pair, fix) in a fresh process — the only correct way to test whether an interposed action prevents the wedge (the multi-fix loop contaminates: once none wedges, the state is unrecoverable in-process).

Results — the transition matrix (separate submits, warm)

first op (A)	second op (B)	before fix
matmul (fp16 or int8)	any SDP (exp/silu/ewmul/add)	SAFE
ewmul_i16 / ewmul_f16 / add_f16 (non-LUT SDP)	matmul (single- or multi-core)	SAFE
exp_i16 / silu_i16 (LUT activation)	multi-core matmul (N ≳ 64)	SAFE
exp_i16 / silu_i16 (LUT activation)	single-core matmul (small N, e.g. N=16)	WEDGE errno=110

Interposed-action tests on the wedging pair (EXP_I16 → MM_F16), each a fresh process:

interposed action between exp and matmul	result
none (baseline)	WEDGE (exp ok 0.8 ms; matmul times out)
ORK_NO_PINGPONG=1 (ping-pong off)	WEDGE — not a LUT/ping-pong SRAM race
ork_npu_mode_reset (explicit HW ACT_RESET + state invalidate)	WEDGE — a HW reset does not clear it
ork_npu_mode_invalidate (clear cached last_dt/warmed)	WEDGE

That ACT_RESET doesn't help — while a mere descriptor restore does (below) — is the proof it is software state, not hardware pipeline state.

Root cause

The LUT-activation ops (ork_npu_exp_i16/silu_i16/gelu/rsqrt → act_lut_i16 → ork_npu_probe_silu_std_i16) memset and reprogram the shared context task descriptor c->task to their own SDP program (regcfg_amount 1097 for the LUT-load task, then 69 for the activation task; enable_mask = 0x18) and left it that way. The single-core matmul path (run() → submit1) does not rebuild c->task — it relies on the value set once in ork_npu_init (regcfg_amount = 108, enable_mask = 0xd, regcmd_addr = c->regcmd.dma) persisting. So the next single-core matmul writes its 108-word matmul regcmd but submits it under the stale 69-reg / 0x18 SDP descriptor → the NPU reads a truncated / mis-typed program, dispatches no task (task counter 0x0) → job timeout. ACT_RESET can't fix a poisoned descriptor sitting in the task buffer.

This reconciles every observation:

• ewmul/add → matmul SAFE: those SDP ops already save and restore regcfg_amount/enable_mask around their submit, so they leave c->task as they found it.
• multi-core matmul after exp SAFE: the multi-core path uses per-core c->mtk[] descriptors that it rebuilds every call — it never reads the poisoned c->task.
• mode 1 survived / mode 2 wedged: mode 1's matmuls are all N≥64 → multi-core; mode 2 adds the CumBA (N=16), the only single-core matmul, which is the one that reads c->task.
• delayed & "mid-sequence": c->task stays poisoned across intervening multi-core matmuls and ewmuls (which faithfully preserve the poisoned 69) until a single-core matmul finally trips — so it doesn't look like a clean adjacent A→B pair.

Fix

Make the LUT op save c->task's regcfg_amount/enable_mask/regcmd_addr on entry and restore them on every exit (success and the LUT-load error path) — exactly the save/restore the ewmul/add ops already do (ork_npu_probe_silu_std_i16, src/npu.c).

Validated on-board (RK3588):

• All previously-wedging transitions now SAFE with no interposed action (mode_probe 2 0 0, 3 0 0, 2 1 0): matmul completes in 0.2–0.4 ms instead of the timeout.
• examples/test_ssd_chunk_npu PASS end-to-end: matmul-G1 3.6e-4, exp-G1 5.9e-3, full-G1 5.8e-3, full-G2 5.9e-3 (mode 2 was a hard wedge before). Board healthy throughout, zero real wedges.

The rule

The minimal safe-sequencing rule for composing heterogeneous per-op NPU sequences across submit boundaries is: every op must leave the shared c->task descriptor as it found it. A fused single-submit PC-chain is not required for per-op correctness (it remains preferable for the submit-floor and the LUT-load ping-pong reasons in the Ping-pong quirk — but that is a performance/independent-chain concern, not this correctness bug). The chosen fix is category (a) — a driver-side descriptor discipline — but specifically a task-descriptor save/restore in the SDP LUT op, not a last_dt/precision-aware reset: the wedge has nothing to do with last_dt, precision, or op-class direction.

Artifacts

• tools/mode_probe.c (make mode_probe) — the transition probe; mode_probe [A [B [fix]]].
• ORK_MM_TIMEOUT — matmul-path submit timeout (ms), mirrors ORK_EW_TIMEOUT; default 60000. Set low for wedge-search so a hang self-terminates.
• ork_npu_mode_invalidate(ctx) / ork_npu_mode_reset(ctx) — interposed-fix hooks (used to prove ACT_RESET does not help; kept as RE/diagnostic hooks).
• Reproducer: examples/test_ssd_chunk_npu.c mode 2.