Kernel Modifications

Kernel Modifications (RK3588 board, 10.3.0.236)

The board does NOT run a stock kernel. Two changes are ours. Both were undocumented until 2026-08-24, which made them a hidden variable in every board measurement taken before that date — including a multi-day IOVA-leak investigation that concluded "kernel-side, not ours" while running a patched driver.

uname -r                     6.1.115-vendor-rk35xx-sram
built by                     root@7235a93dce00   <- docker container "kbuild" on the .239 colima VM
source tree                  kbuild:/root/linux-rockchip   (changes are UNCOMMITTED working-tree edits)
revert                       /boot/RESTORE_STOCK_KERNEL.sh  (two symlink flips, idempotent)

Where the source lives

The build ran inside a stopped-but-intact docker container on the Colima VM at 10.3.0.239:

ssh 10.3.0.239 'export PATH=$PATH:/opt/homebrew/bin
  docker start kbuild
  docker exec kbuild sh -c "cd /root/linux-rockchip && git status --short && git diff"'

Both modifications are uncommitted working-tree edits — there is no commit to point at, which is why they were invisible. /root/kernel is a second, unmodified tree.

Change 1 — NPU SRAM enabler (`arch/arm64/boot/dts/rockchip/rk3588s.dtsi`)

Gives the NPU the on-chip SRAM that the video decoders normally own, and merges the two decoder regions into one 956 KB block (0xef000).

 	rknpu: npu@fdab0000 {
 		compatible = "rockchip,rk3588-rknpu";
+		rockchip,sram = <&rkvdec0_sram>;

 	rkvdec0: rkvdec-core@fdc38000 {
-		rockchip,sram = <&rkvdec0_sram>;
 	rkvdec1: rkvdec-core@fdc48000 {
-		rockchip,sram = <&rkvdec1_sram>;

 		rkvdec0_sram: rkvdec-sram@0 {
-			reg = <0x0 0x78000>;
-		};
-		rkvdec1_sram: rkvdec-sram@78000 {
-			reg = <0x78000 0x77000>;
+			reg = <0x0 0xef000>;
 		};

Paired with CONFIG_ROCKCHIP_RKNPU_SRAM=y — the only config line differing from stock (# CONFIG_ROCKCHIP_RKNPU_SRAM is not set).

Why: the NPU's on-chip SRAM reported 0 available; this hands it the decoders' 956 KB. Cost: rkvdec0 and rkvdec1 lose their SRAM — video decode is degraded/untested on this board. Verify live: /proc/device-tree/npu@fdab0000/rockchip,sram exists.

Change 2 — RKNPU domain-refcount leak on job timeout (`drivers/rknpu/rknpu_job.c`)

 static void rknpu_job_timeout_clean(struct rknpu_device *rknpu_dev, ...)
 			spin_unlock_irqrestore(&rknpu_dev->irq_lock, flags);
 
+			rknpu_iommu_domain_put(rknpu_dev); /* #patch: reap the domain ref this timed-out job
+			                                      leaked (job_done/job_abort put; timeout_clean
+			                                      forgot) -> fixes multi-domain switch wedge */
 			do {
 				schedule_work(&job->cleanup_work);

Why: rknpu_job_done and rknpu_job_abort release the IOMMU domain reference; rknpu_job_timeout_clean did not, so every timed-out job leaked one — and a held reference blocks the next domain switch, which is the multi-domain wedge.

⚠ THIS PATCH IS A SUSPECT IN THE CURRENT LEAK (Tier 17). It is an added put. If a path reaches timeout_clean after the reference was already released, this is a double-put — the refcount underflows, the domain is freed early, and a later unmap hits a stale domain, producing exactly the observed WARN_ON(unmapped != size) in __iommu_dma_unmap and the never-reclaimed IOVA. The symptom it was written to fix (switch iommu domain time out) is also the symptom still being observed, so it is either incomplete or overcorrecting.

Change 3 — RKNPU records no IOMMU domain for dma-buf imports (`drivers/rknpu/rknpu_gem.c`)

This is the multi-domain IOVA leak. Diff: 03-rknpu-import-domain-id.patch. Deployed 2026-08-24; backup of the pre-patch image is /boot/vmlinuz-6.1.115-vendor-rk35xx-sram.pre03-bak.

 	rknpu_obj->dma_addr = sg_dma_address(sgt->sgl);
+	/* #patch: the sg above was mapped against the CURRENTLY attached domain.
+	 * Record it, or destroy unmaps from domain 0 and strands this IOVA. */
+	rknpu_obj->iommu_domain_id = rknpu_dev->iommu_domain_id;

Three driver facts combine into the bug:

rknpu_gem_prime_import_sg_table() never sets iommu_domain_id, and rknpu_gem_init() uses kzalloc — so every imported object is filed under domain 0.
Its sg table was mapped by dma_buf_map_attachment() during PRIME_FD_TO_HANDLE, against whichever domain happened to be attached at that moment.
rknpu_gem_create_ioctl() does rknpu_gem_object_find(args->handle) and returns early on a hit. args->iommu_domain_id is therefore silently ignored for imports — userspace passes the domain and the kernel drops it. There is no uABI way to make it stick.

rknpu_gem_object_destroy() then faithfully switches to the recorded id (0) and unmaps from the wrong page table:

drm_release -> drm_gem_release -> idr_for_each -> rknpu_gem_object_destroy
  -> drm_prime_gem_destroy -> dma_buf_unmap_attachment -> __iommu_dma_unmap
WARNING: drivers/iommu/dma-iommu.c:691 __iommu_dma_unmap+0xb8/0x13c

The unmap silently does nothing and the IOVA in the real domain is stranded for the life of the boot. Compare the freshly-allocated path, rknpu_gem_object_create(): it switches to the domain first and then stamps iommu_domain_id. Correct. Only the import path has the gap.

Why it looked like a multi-domain-only bug: a single-domain process has the attached domain at 0, matching the recorded 0, so the unmap hits the right table. The handles leak identically — they just cost nothing.

Measured, 0.8B, `ORK_DOMAINS_FORCE`, same binary either side

	before (patch 01+02)	after (patch 03)
`WARN_ON` per multi-domain run	30, invariant across dtype / domain count / weight count	0
consecutive multi-domain runs before failure	2 (3rd: `MEM_CREATE EFAULT` at 176 MB of 3900)	5+, no degradation
int8, 250 imports under non-zero domains	30	0
PPL	36.3973 int4 / 17.5502 int8	identical

Userspace prediction check before patching: 26 imports performed under a non-zero domain vs 30 observed WARNs — "requested domain" is only a proxy for "domain attached at PRIME_FD time", which is what actually decides where the mapping lands.

Worth reporting upstream against both the vendor driver and drivers/accel/rocket.

Change 4 — signal the fence when a job is torn down without completing (`drivers/rknpu/rknpu_job.c`)

dma_fence_signal(job->fence) occurs at exactly ONE site: the RKNPU_JOB_DONE completion path. rknpu_job_abort() and rknpu_job_timeout_clean() tear a job down without signalling, and rknpu_job_free() only does dma_fence_put() — a reference drop, not a signal. So a stalled job never wakes its waiter, which defeats the point of waiting on a fence to detect a stall.

Fix: in rknpu_job_free(), if the fence is not yet signalled, dma_fence_set_error(-ETIMEDOUT) then dma_fence_signal() before the put. No-op on the success path (already signalled). Diff: 04-rknpu-fence-signal-on-abort.patch.

Note dma_fence is a generic Linux primitive (drivers/dma-buf), not an rknpu or NVDLA concept; the vendor driver implements the whole FENCE_IN/FENCE_OUT path and simply left it compiled out.

Change 5 — per-core progress watchdog (`drivers/rknpu/rknpu_drv.c`)

This is what root-caused the int4 doorbell "miss". rknpu_job_timeout_clean is not a watchdog: it runs only when ANOTHER submit arrives on that core, and it measures TOTAL JOB AGE, which is meaningless for a PC-chained job whose task list is extensible at runtime. Change 5 samples the PC completed-task counter (config->pc_task_status_offset & config->pc_task_number_mask) per core at 1 kHz while a job is outstanding, and reports "no task progress in N samples". Detector only — it logs, it does not reap.

Result: on every stall the counter reads 0x0 with the job >100 ms old on all three cores — the job is accepted (rc=0) and never dispatches. See NPU-Quirks for the full write-up and control.

Params under /sys/module/rknpu/parameters/: wd_period_us (0 disables), wd_samples; counters wd_ticks, wd_seen_job, wd_pm_skip, wd_stalls. Diff: 05-rknpu-progress-watchdog.patch.

⚠ POWER SAFETY — read before touching this code

Two earlier versions were wrong, and the first hard-hung the board (no ping, power cycle required):

v1 read NPU MMIO directly from the hrtimer callback. A register read on a powered-down block is an external abort. It hung on its first NPU workload.
v2 guarded with pm_runtime_get_if_in_use(rknpu_dev->dev). Still wrong on RK3588: this driver keeps its OWN refcount (power_refcount / power_lock) and rknpu_power_off() puts the three PER-CORE genpds (genpd_dev_npu0/1/2) directly, so a reference on the parent does not hold them up. It survived only because the deferred power-off keeps the NPU up during a busy run — luck, not correctness. RETRACTION (2026-08-26): the watchdog did NOT cause the board resets. They were blamed on it from a single A/B (n=1 each way), then the same workload with the watchdog ON survived. netconsole later caught the real fault — an unguarded MMIO read in the VENDOR rknpu_irq_handler (see Change 6). Only the v1 hard-hang below is genuinely attributable to this watchdog. It ships default-off anyway, because it is a diagnostic, not something a production run should carry.
v3 (shipped) samples in PROCESS context (hrtimer → queue_work) holding power_lock, which rknpu_power_off() must also take, so the genpds cannot drop mid-read. It deliberately does not call rknpu_power_get() — that powers the NPU ON just to read a counter and would pin it up forever; it checks power_refcount > 0 and skips otherwise. mutex_trylock so it can never delay a submit. Measured wd_pm_skip = 332 / ~35000 samples (0.9%), so coverage is essentially unaffected.

Change 6 — RKNPU IRQ handler touches MMIO while the NPU is powered down (`drivers/rknpu/rknpu_job.c`)

★ This is very likely the long-standing "hard wedge" that has cost this board power cycles and at least one physical SPI reflash.

rknpu_irq_handler() accesses registers unconditionally on BOTH paths — the no-job path does REG_WRITE(RKNPU_INT_CLEAR), the normal path does REG_READ(RKNPU_OFFSET_INT_STATUS) — with no power check. Power-off runs from a DEFERRED work item (rknpu_power_off_delay_work), so a late or spurious interrupt can land after the block has lost power. The register access then takes an external abort and the machine dies instantly: no console, no ping, power cycle required.

Captured via netconsole (nothing else on this board can retain it — see below):

pc : readl+0x4/0x20
lr : rknpu_irq_handler.isra.0+0x94/0x2f0
Call trace:
  readl+0x4/0x20
  rknpu_core0_irq_handler+0x20/0x30
  __handle_irq_event_percpu+0xbc/0x1d4
  ... el1_interrupt / cpuidle_enter / do_idle

CPU 0 was idle — i.e. the NPU had finished, powered down, and the IRQ arrived afterwards.

Corroboration that late IRQs are real on this platform: the neighbouring rk_iommu_irq() (drivers/iommu/rockchip-iommu.c:762) already guards this exact race with pm_runtime_get_if_in_use() + WARN_ON_ONCE, and that WARNING fires on this board under the same load. The IOMMU driver survives with a warning where rknpu died — because rknpu had no guard.

Fix: return IRQ_NONE without touching MMIO when power_refcount <= 0. power_refcount is an atomic, so unlike power_lock (a mutex) it is safe from hard-IRQ context. If the block is unpowered it cannot be asserting, so IRQ_NONE is correct and cannot cause a level-IRQ storm. Diff: 06-rknpu-irq-power-guard.patch.

Validated: 6 passes x 3 shapes x 2000 reps (36,000 int4 doorbell ops) with the Change-5 watchdog deliberately ENABLED — 0 panics, board healthy. The identical workload previously reset the board.

Why this only became visible now

It is not new. With the stock CONFIG_PANIC_ON_OOPS_VALUE=0, this fault had no clean path: the machine died silently and unrecoverably. Enabling panic_on_oops=1 + panic=10 (see the fence-kernel section) converts it into panic → backtrace → auto-reboot in 10 s, which is the ONLY reason the trace exists. The historical hard wedges recorded elsewhere in this wiki have the same fingerprint.

Log capture on this board — what actually works

netconsole: works (this trace came from it). Persisted via /etc/systemd/system/netconsole.service (ordered After=network-online.target; modules-load.d is too early to bind eth0) + /usr/local/sbin/netconsole-up.sh. netpoll needs a destination MAC — the gateway's MAC is the one that reaches the capture host here.
⚠ console_loglevel defaults to 4, so only messages BELOW level 4 reach the console and hence netconsole. <4>/KERN_WARNING test messages are silently dropped — this cost an hour of wrongly concluding netconsole was broken. kernel.printk = 8 4 1 7 is persisted in /etc/sysctl.d/.
pstore/ramoops: DOES NOT WORK on RK3588. The regions are reserved and re-registered every boot, but /sys/fs/pstore comes back empty even after a warm panic-reboot — the Rockchip restart re-initialises the DDR controller and wipes the region. Do not rely on it.
/var/log is a 50 MB tmpfs (DietPi RAMlog, AUTO_SETUP_LOGGING_INDEX=-1) and journald retains no prior boots (journalctl -b -1 → "No journal boot entry found") despite Storage=persistent.
A durable fallback runs on the NVMe: kmsglog.py → /home/michael/ork-outbox/kern.log, fsync per message, outside RAMlog.

Change 7 — NULL IOMMU domain dereferenced in the GEM DMA paths (`drivers/rknpu/rknpu_iommu.c`)

rknpu_iommu_dma_map_sg(), rknpu_iommu_dma_unmap_sg() and rknpu_iommu_dma_alloc_iova() each dereference iommu_get_domain_for_dev() (or the domain handed to them) with no NULL check, one line in:

struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
struct rknpu_iommu_dma_cookie *cookie = (void *)domain->iova_cookie;   /* boom */

And the device really can end up attached to no domain at all: rknpu_iommu_switch_domain() does iommu_detach_device(src) and then, if attaching dst fails, tries to re-attach src — if that also fails it returns with the device detached. This is reachable in normal operation, because a stalled job cannot be drained, so the switch hits its 6 s timeout (switch iommu domain time out) and the abort path runs.

Captured (netconsole):

Unable to handle kernel NULL pointer dereference at virtual address 0000000000000040
pc : rknpu_iommu_dma_map_sg+0x48    Comm: i4_widek_stall_
  rknpu_gem_object_create / rknpu_gem_create_ioctl / drm_ioctl

and, after guarding only map_sg, the identical shape at rknpu_iommu_dma_unmap_sg+0x3c on the next run.

Fix: a rknpu_iommu_cookie() helper that returns NULL with a ratelimited error instead of faulting (callers turn it into an ordinary failure), plus a switch failure path that attaches any still-valid domain rather than returning detached. Diff: 07-rknpu-null-iommu-domain-guards.patch — applies standalone to the base commit.

This is not theoretical: the guard fired 12 times in a single run. The detached state happens.

Change 8 — `rknpu_job_abort` publishes a freed job pointer (`drivers/rknpu/rknpu_job.c`)

if (job == subcore_data->job && !job->irq_entry[i]) {
        subcore_data->job = NULL;
        subcore_data->task_num -= rknpu_get_task_number(job, i);
}
...
rknpu_job_cleanup(job);          /* frees the job EITHER WAY */

The clear is gated on !job->irq_entry[i], but the free is not. Whenever irq_entry[i] is set, the job is freed while subcore_data->job still points at it. The IRQ handler, rknpu_job_next() and the progress watchdog all read that field, so the next reader touches freed slab.

Because the corruption and the crash are far apart, this never looked like an NPU bug. Observed panic sites, all downstream of the same root:

BUG: Bad page cache ... "still mapped when deleted"   (shmem_evict_inode)
__memcg_kmem_charge_page  <- copy_process <- fork, from dropbear
sg_free_table / __free_pages  <- rknpu_gem_object_destroy
cfs_rq_clock_pelt <- attach_entity_cfs_rq

Fix: clear subcore_data->job unconditionally; keep the task_num adjustment gated exactly as before so the accounting is unchanged. Diff: 08-rknpu-abort-published-freed-job.patch — applies standalone to the base commit.

Consequence worth knowing: with the dangling pointer gone, the Change-9 stall kick correctly finds NULL and declines — and measured recovery drops from "3/3" to 0/5. See the retraction below.

Change 9 — doorbell-stall diagnostics and per-core reset (DIAGNOSTIC, default-off)

A bundle built to root-cause the int4 NONBLOCK dispatch stall. Not intended to ship: RKNPU_ACT_RESET_CORE uABI (0x100, outside the vendor 0-25 range) + rknpu_soft_reset_core(); rknpu_job_kick_stalled() with wd_kick modes; KICKLAT timing; commit-register snapshot and full PC register-block dump; IOMMU attach/switch counters; MMU register dump at stall; a settle-only kick mode.

It also fixes a real vendor wart found on the way: rknpu_soft_reset() returned 0 (success) on a failed mutex_trylock without resetting. Harmless until something else took reset_lock — once the kick did, userspace ACT_RESET became a silent no-op and throughput fell ~10x (2.7 vs 34 commits/s).

What it measured — four hypotheses refuted with counters

hypothesis	instrument	result
deferred attach leaves a stale page table	`rk_dbg_attach_deferred`	refuted — 0 deferred vs 1017 programmed
the switch races an in-flight job	`rknpu_dbg_switch_inflight`	refuted — 0 in-flight vs 1012 idle
the MMU itself wedges	`RK_MMU_STATUS` at stall	refuted — all 4 banks `0x19` = paging, idle, replay-empty
recovery needs post-reset settle time	settle 2/10/40 ms	refuted — 0 recovered of 20

An idle MMU at stall time is the sharpest clue on record: the NPU core never issued a memory request, so the failure is upstream of the IOMMU entirely.

⚠ RETRACTION — "the IOMMU re-attach is what makes recovery work"

An earlier result here claimed the IOMMU detach/attach was the essential recovery ingredient (3/3 with it, 0/11 without). That is retracted. Once Change 8 stopped abort publishing a freed job, the kick recovers 0/5. The original measurement was almost certainly re-committing a job that had already been freed — consistent with its suspiciously identical ~4.47 ms completions (4466/4471/4473/4478 us). Do not build on it, and do not cite the 3/3-vs-0/11 split.

The kick also had a defect of its own: its iommu_attach_device() return was unchecked, so a failed re-attach left the device detached. One occurrence produced 153 switch timeouts — 933 s of a 955 s run. Now checked with a fallback, but not fully contained.

Leave wd_kick=0 and wd_period_us=0 (both defaults). Post-retraction the kick has no demonstrated benefit, and it can wedge the driver into 6-second-timeout purgatory.

Watchdog cadence — a 100x throughput trap

Detection latency is wd_samples * wd_period_us. wd_period_us=1000 (1 kHz) queues work every millisecond and contends on the power/reset locks: 2.8 commits/s vs 283-490/s warm. Use wd_period_us=10000 with wd_samples=12 — the same ~120 ms detection at a tenth the sampling cost.

Complete current state

The board is running kernel #12 = base commit + all of changes 1-9. The exact cumulative diff is 99-ALL-current-state.patch (794 insertions, 11 files) — apply it to 1cd878b3ac49 to reproduce the running kernel. Changes 7 and 8 are additionally provided standalone because they are real fixes worth carrying without the diagnostics.

Change 10 — `rknpu_gem_destroy_ioctl` holds an unreferenced GEM pointer across an ~18 s wait (`drivers/rknpu/rknpu_gem.c`)

rknpu_gem_object_find() (include/rknpu_gem.h) is the lookup-then-put anti-pattern:

obj = drm_gem_object_lookup(filp, handle);   /* takes a reference */
if (!obj) return NULL;
rknpu_gem_object_put(obj);                   /* ...and drops it immediately */
return to_rknpu_obj(obj);                    /* caller gets an UNREFERENCED pointer */

Its kerneldoc claimed "gem object reference count would be increased" — that comment was false, and it is how this went unnoticed. Change 10 corrects the comment as well as the caller.

An unreferenced pointer is survivable for a short ioctl. rknpu_gem_destroy_ioctl() is not short: it then blocks in rknpu_iommu_domain_get_and_switch() for up to 6 s per attempt, three attempts (~18 s), and dereferences rknpu_obj->iommu_domain_id inside the retry loop. If the owning process dies in that window — SIGTERM -> DRM file close -> every handle released -> objects freed — the pointer dangles.

The window is wide open in practice, not theoretical: a stalled NPU job is exactly what makes the domain switch time out, and a stuck run is exactly when the process gets killed.

Why this is the suspected source of the page-cache corruption. rknpu maps the GEM's shmem pages into userspace itself with vm_insert_page() (rknpu_gem.c:983,1122) and clears VM_PFNMAP (:1148), so they are ordinary mapped pages that raise page->_mapcount. But vma->vm_file remains the DRM node — there is no get_file(obj->filp) anywhere. So the shmem inode's i_mmap tree has no record of these VMAs and truncate/evict cannot unmap them; it can only report:

BUG: Bad page cache in process i4_widek_stall_   pfn:128a06
page dumped because: still mapped when deleted     (mapcount:1)
  filemap_unaccount_folio / shmem_undo_range / shmem_evict_inode

The GEM refcount is therefore the only thing preventing a premature free — which makes an unreferenced pointer held across a multi-second sleep the prime suspect.

Fix: take a real drm_gem_object_lookup() reference for the whole ioctl and put it on every exit path. The put is placed before rknpu_iommu_domain_put() so that, if ours is the last reference, the object's teardown still runs with the domain held — the same ordering the original had, where handle_destroy could free it at that exact point. Diff: 10-rknpu-gem-destroy-unreferenced-pointer.patch — applies standalone to the base commit.

Status: built and running as kernel #13; make test ALL TESTS PASSED. NOT yet proven to fix the corruption — that needs a run that reproduces the old crash (stall probe + SIGTERM mid-stall). The other two rknpu_gem_object_find() callers (rknpu_gem_create_ioctl, the mmap-offset path) have the same latent pattern but no blocking wait, so they were left alone deliberately.

Correction to Change 8's risk assessment

While landing this, Change 8 (clearing subcore_data->job unconditionally in rknpu_job_abort) was briefly suspected of stalling job dispatch. It does not, and the opposite is true. rknpu_job_next() advances only when the slot is empty:

if (subcore_data->job || list_empty(&subcore_data->todo_list)) { spin_unlock; return; }

A non-NULL entry blocks that core. So upstream's behaviour — leaving a pointer to a freed job published — would block the core permanently, and clearing it can only help progress.

Change 11 — `rknpu_gem_create_ioctl` dereferences an unreferenced GEM pointer (`drivers/rknpu/rknpu_gem.c`)

The second live caller of the lookup-then-put rknpu_gem_object_find() (see Change 10). It reads the object four times after the lookup:

args->size      = rknpu_obj->size;
args->sram_size = rknpu_obj->sram_size;
args->obj_addr  = (__u64)(uintptr_t)rknpu_obj;
args->dma_addr  = rknpu_obj->dma_addr;

with nothing but the handle table keeping it alive, so a concurrent close of that handle frees it underneath.

This is hardening, not a suspected crash source — unlike Change 10 nothing sleeps between the lookup and the last use, so the window is short. But it is a real window, and correctness should not rest on "probably not preempted there". (The vendor left a // rknpu_gem_object_get(&rknpu_obj->base); commented out right in the middle of it, so the gap was noticed at some point and then abandoned.)

Fix: drm_gem_object_lookup() on the existing-handle branch, drm_gem_object_get() after rknpu_gem_handle_create() on the create branch — so both leave exactly one reference owned by the ioctl — and one put before returning. Net refcount across the ioctl is unchanged: the handle keeps the object alive afterwards exactly as before; only the in-ioctl window is closed. Diff: 11-rknpu-gem-create-unreferenced-pointer.patch — applies standalone to the base commit.

The third caller was deliberately NOT touched. rknpu_gem_dumb_map_offset() has the same pattern and is the worst of the three on paper — it calls drm_gem_create_mmap_offset(), which allocates and can sleep, on an unreferenced object. But it sits inside #if KERNEL_VERSION(4, 19, 0) > LINUX_VERSION_CODE, so on 6.1.115 it is not compiled at all; the live path is DRM core's drm_gem_dumb_map_offset(), which refcounts correctly. Confirmed against the built rknpu_gem.o, not just by reading the #if. Fixing dead code would be churn with no upside — but if this driver is ever backported below 4.19, fix it there first.

Status: built and running as kernel #14; make test ALL TESTS PASSED.

Change 12 — the progress watchdog is ON by default, and `wd_kick` is deleted

12a. `rknpu_wd_arm()` never fired for blocking submits (the bug that made the watchdog useless)

The arm site lived inside if (args->flags & RKNPU_JOB_NONBLOCK) in rknpu_submit(), so a blocking submit was never sampled. That is not a corner case: ork_dyn_colsplit() issues blocking submits by default ("NO barrier + BLOCKING submit — EXACTLY the mcworker", src/npu/core/colsplit.c:48) and only goes NONBLOCK under ORK_F16_SENTINEL. Its ork_dyn_ prefix makes it look like the doorbell path; it is not.

Found the hard way by a second agent debugging a 1.5B-Q8 stall: with wd_period_us=1000 set by hand, the run stalled 8 times and recorded wd_ticks=0 — the hrtimer never started once, so the instrumentation built to catch these stalls could not observe the very path that stalls. Meanwhile a stalled blocking submit sits in rknpu_job_wait() for its whole timeout; 60.6 s was measured.

Fix: arm on every submit. Verified — a blocking-submit workload now yields wd_ticks=66, wd_seen_job=78 where it previously gave 0/0.

12b. Detection is now default-ON (`wd_period_us=10000`, `wd_samples=12`, ~120 ms)

Both original reasons for opt-in have expired: the "it caused board resets" attribution was retracted (that was the change-6 vendor IRQ bug), and the throughput objection was a cadence problem, not an inherent one. Measured A/B on this board (2000-rep int4 probe, ndom=1, interleaved x3, warm-up discarded):

	r1	r2	r3	mean
OFF	133/s	133/s	133/s	133.0/s
ON (10 ms x 12)	131/s	133/s	129/s	131.0/s

~1.5% — small but real, and not described as free. Paid on every submit now that 12a landed. Judged worth it: the stalls it exposes cost ~23% of 27B prefill and were previously invisible. Validated immediately — make test on kernel #16 passed while reporting wd_stalls=2, two stalls that until now were silent.

Do NOT lower wd_period_us to 1000 without lowering wd_samples too: 1 kHz sampling costs ~100x throughput (2.8 vs 283-490 commits/s).

12c. `wd_kick` deleted

The auto-recovery kick is removed entirely: the wd_kick parameter, rknpu_job_kick_stalled(), wd_settle_us, the KICKLAT timing and the MMU register dump that only it called.

Basis for deletion — it has no demonstrated benefit and a demonstrated harm:

Its entire justification, "the IOMMU re-attach recovers 3/3", was retracted: that measurement was re-committing a job rknpu_job_abort() had already freed (change 8). Post-fix it recovers 0/5.
Across every mode tried post-fix — reset+recommit, reset+settle at 2/10/40 ms, reset+IOMMU — it recovered 0 jobs out of roughly two dozen observed stalls.
It caused harm: an unchecked iommu_attach_device() return left the device detached and produced 153 domain-switch timeouts, 933 s of a 955 s run.

Honest limitation: the intended bar was a single clean trial of 20+ stalls, and that was never achieved — the stall-heavy workload wedges the driver (switch-timeout purgatory) before enough stalls accumulate, and three attempts died that way. The deletion rests on the totality above, not on one decisive trial.

KEPT: rknpu_soft_reset_core() and the RKNPU_ACT_RESET_CORE uABI (0x100). Those are a deliberate per-core counterpart to the full reset and are reachable from the ioctl path independently of the kick.

Not separately cherry-pickable (it removes code added by changes 5/9), so the authority is 99-ALL-current-state.patch — now changes 1-12, the exact content of kernel #16 (make test ALL TESTS PASSED).

Change 13 — fast-abort a stalled BLOCKING submit (`wd_abort_ms`, OPT-IN, partial win)

A stalled blocking submit sits in rknpu_job_wait() for args->timeout x3 — 61 s measured — before returning -ETIMEDOUT and letting userspace self-heal. Change 13 lets the watchdog wake that waiter early: a new internal RKNPU_JOB_STALLED flag (bit 8 of job->flags, which is never assigned from the uABI args->flags) is added to the wait condition, and setting it routes into the existing ret <= 0 path, so the outcome is byte-for-byte the same -ETIMEDOUT, just sooner.

The abort bound is deliberately NOT the logging threshold. The PC counter advances per COMPLETED TASK, so one long-running task is indistinguishable from a stall by this signal; aborting at the ~120 ms log threshold would kill slow-but-healthy jobs. wd_abort_ms is a separate, much larger bound (2000 ms tested).

Measured on the 1.5B-Q8 repro (`ork_bench ... 256 32`), two boots

	control (`wd_abort_ms=0`)	fast-abort, first cut	fast-abort + per-core reset
submit failures	14	2	2
longest in-kernel wait	61.0 s	184.3 s	61.6 s
run completes	no	no	no

A regression found and fixed en route. The first cut tripled the worst-case wait, because a fast-aborted job still reaches rknpu_job_abort() with -ETIMEDOUT, which runs a device-wide rknpu_soft_reset() — now firing ~30x more often and disrupting healthy jobs on the other cores. Fixed by resetting only the cores the job owns (rknpu_soft_reset_core(), kept from change 9 for exactly this sort of use). The 184 s tail went straight back to 61 s.

Verdict: keep OPT-IN. It is a partial win, and it is detection-limited.

Submit failures drop 14 -> 2, reproducibly (14 in both control runs, 2 in both treatment runs, separate boots). But the run still does not complete and the worst-case wait is unchanged, because only 2 of 14 failures are ever detected. The abort mechanism is not the limiter; coverage is.

Likely cause of the coverage gap (next thing to fix): the sampler only inspects subcore_data->job — the committed job. A submit sitting in todo_list behind a stuck job is never sampled, so its waiter burns the full timeout invisibly. rknpu_job_next() will not promote it while the slot is occupied. Covering queued jobs is the obvious next step.

Corrections to earlier characterizations (from a second agent's independent run)

Two claims elsewhere in this wiki and in DOORBELL_STALL_WIP.md do not survive their data:

"All THREE cores report simultaneously." Not universal — an episode named cores 1 and 2 only. THIS CORRECTION IS ITSELF WITHDRAWN (see change 14). The missing core-0 report was an artifact of the watchdog's own under-counting: wd_flat was not reset across a job boundary, so only the FIRST stall in a run of them was ever reported and core 0's had already been absorbed. With that fixed (#patch44) the same workload reports core 0 x3, core 1 x3, core 2 x3 — exactly symmetric. The original "all three cores" observation stands. A good reminder that a correction drawn from an instrument with a known coverage bug inherits that bug.
wd_stalls is NOT a miss count. It is a lower bound. 11-14 userspace submit failures produced only 2-3 watchdog episodes. Contributing factors: the sampler only sees committed jobs (above), and the "fire ONCE per episode" equality test (++wd_flat == wd_samples) collapses a persistent stall into a single report. Do not treat wd_stalls as the number of misses.

Their run also confirmed the two instruments now corroborate: tasks=0/4 done ... dom=1 in the kernel line matches the userspace ORK_KMSG_SUBMIT stamp's domain=1 task_number=4 for the same event.

Change 14 — attribute a stall run to the JOB, not to a counter value

wd_flat (consecutive no-advance samples) only reset when the PC counter CHANGED. But consecutive jobs on one core each start their counter at 0, so across a job boundary the counter looks unchanged and wd_flat keeps climbing. Since both the stall report and the fast-abort fire on equality (== wd_samples, == an), only the FIRST stall in a run of them was ever seen; every later one was silently absorbed. Fixed by resetting the run whenever subcore_data->job identity changes.

Measured, 1.5B-Q8 repro, counted WITHOUT circularity

Coverage must not use the watchdog for both numerator and denominator. rknpu_job_wait() logs "failed to wait job ... elapsed time" on every timed-out wait, independently of the watchdog, so wd_stalls / failed-to-wait-lines is a sound ratio. (Read it from /var/log/ork-kmsg.log, not dmesg — the kernel ring OVERFLOWS on a bad run and silently truncates the denominator, which invalidated one measurement here.)

	before #patch44	after #patch44
timed-out waits (independent)	10	14
detected + fast-aborted	5	9
coverage	50%	64%
wait durations	2.0s x5, ~61s x5	2.0s (many), ~61s x10
per-core stalls	cores 1,2 only	core 0 x3, core 1 x3, core 2 x3

Two things this settles:

The fast-abort is sound: everything it detects ends at ~2.0 s instead of ~61 s, sharply bimodal.
Stalls hit all three cores symmetrically. The earlier "not all three cores" report was the under-counting bug, not a per-core-loss signature.

Still open: ~36% of timed-out waits are still not detected, and the run does not complete. The todo_list-queued-jobs theory was tested and REFUTED (queued-aborts=0 at every stall — the queue was always empty), so the remaining misses are something else.

Change 15 — the domain wedge: a reset/switch race, plus a refcount clamp

Two distinct wedges were conflated under "domain wedge". One is fixed, one is a downstream symptom, and a third theory was tested and refuted.

15a. REFUTED: refcount underflow

rknpu_iommu_domain_put() did a bare atomic_dec() while get_and_switch() proceeds only when the count reads exactly 0 — so one unbalanced put would wedge the device permanently, and our own change 02 added a put (a documented double-put suspect). Plausible, and wrong: instrumented with a clamp and an underflow counter, dbg_domain_underflow stayed 0 across a full failing run. The refcount is balanced.

The clamp is kept anyway — turning a hypothetical over-put into a counted anomaly rather than a dead NPU is cheap insurance, and dbg_domain_underflow != 0 is now a live tripwire if a real imbalance appears.

15b. FIXED: `rknpu_soft_reset` raced domain switching

rknpu_soft_reset() holds reset_lock and does iommu_detach_device() + iommu_attach_device(). rknpu_iommu_domain_get_and_switch() holds domain_lock. Different locks — so a switch could read iommu_get_domain_for_dev() while the reset had the device detached, find it inconsistent with rknpu_dev->iommu_domains[], and bail with "mismatch domain get from iommu_get_domain_for_dev".

Always latent; invisible while resets were rare. The change-13 fast-abort raised the reset rate ~30x and made it dominant: 1600 mismatch lines in one 300 s run. Fixed by taking domain_lock around the reset's detach/attach (ordering reset_lock -> domain_lock; safe because no path takes reset_lock while holding domain_lock).

Measured across the two fixes, same 1.5B-Q8 repro

	before	after
`mismatch domain` lines	1600	0
`rknpu_gem_object_create error` (the allocation wedge)	many	0
timed-out waits in 300 s	68	9
detection coverage	64%	89% (8/9)
`switch iommu domain time out`	1	83

15c. NOT a bug: the remaining switch timeouts

Mismatches converted into switch timeouts because the mismatch was previously failing fast and hiding them. get_and_switch() waits for iommu_domain_refcount == 0, and an in-flight job holds a domain reference for its whole life — so while a job is stalled, every other thread's domain switch blocks and eventually times out. These are downstream of the stall, not an independent defect. Fixing them means fixing the stall (or not holding a domain reference across a stalled job).

Measurement caveat for anyone repeating this: the count of timed-out waits is bounded by time budget / wait duration. At 61 s per wait only ~5 fit in 300 s; at 2 s many more do. Counts are NOT comparable across wd_abort_ms settings — compare rates.

Change 16 — ★ opt-in timing profiler, and the strongest correlation found so far

prof_mask (0 = off) turns on a per-job timing profile. Everything it reports is derived from timestamps the driver already takes (job->timestamp, job->hw_commit_time) plus one new per-core "previous commit" time, so it is nearly free and defaults off.

Two quantities, recorded for every job and accumulated separately for healthy vs stalled: gap_since_prev_commit (this core's previous commit -> this one) and queue (submit -> commit). Exposed as prof_{ok,bad}_{n,gap_us_sum,queue_us_sum} under /sys/module/rknpu/parameters/.

★ Result: stalls correlate with BACK-TO-BACK commits on the same core

1.5B-Q8 repro, one 300 s run, 100% detection coverage (86 stalls / 86 independent timed-out waits):

	n	mean gap since prev commit	mean queue
healthy	1069	207,356 us	0 us
stalled	86	492 us	0 us

Individual stalls, always in core-0/1/2 triplets:

PROF stalled core 0: gap=321us   core 1: gap=259us   core 2: gap=240us
PROF stalled core 0: gap=863us   core 1: gap=839us   core 2: gap=821us
PROF stalled core 0: gap=415us   core 1: gap=434us

Every stalled job was committed 0.24-0.9 ms after the previous commit on that core. Queue delay is ~0 for both populations, so it is not a queueing effect. This is the first positive correlation found after every software stage was individually exonerated (commit registers byte-identical, regcmd bytes verified in DRAM, MMU idle, zero IOMMU faults, four hypotheses refuted with counters).

What this does NOT yet establish

The healthy mean is the wrong statistic — 207 ms is inflated by idle periods between bursts. A short gap is clearly NECESSARY for a stall; whether it is SUFFICIENT needs the healthy distribution (what fraction of healthy commits also fall under 1 ms). prof_mask bit 1 is documented as "also log healthy commits" but that path is NOT implemented yet — only the accumulator is.
Correlation, not causation. The decisive test is to enforce a minimum inter-commit spacing and see whether stalls disappear. Note rknpu_job_next() can run from the completion IRQ, so any enforced spacing there must be udelay(), not a sleeping wait.

Side effect worth recording: the repro now COMPLETES

With changes 12-16 the 1.5B-Q8 run finishes instead of timing out: prefill 7.27 tok/s, decode 11.49 tok/s (rc=0, previously rc=124 every time). Still ~30x below the ~220 tok/s healthy baseline because 86 stalls x ~2 s of fast-abort dominate the time, but it is no longer an unrecoverable hang.

Change 17 — `RKNPU_ACT_REINIT`: clear all IOMMU domain state without rebooting

rknpu_job_abort() releases the IOMMU domain reference unconditionally, so with several cores busy it drives iommu_domain_refcount to zero while other cores are still executing. rknpu_iommu_domain_get_and_switch() reads a zero count as "safe to switch" and switches domains underneath live work. That accumulates across runs until every switch fails (mismatch domain get from iommu_get_domain_for_dev) and then every allocation fails (rknpu_gem_get_pages: dma map ... fail). Filed upstream as rockchip-linux/kernel#387.

Until that is fixed a reboot was the only way to clear it — ~90 s each, and it destroys any in-RAM diagnostics. RKNPU_ACT_REINIT (0x101, outside the vendor action range) is the shortcut: abort in-flight jobs, tear down and rebuild every domain, zero the refcount and the diagnostic counters.

Driven from userspace by tools/re/npu_reinit.c in ork-driver.

It must ABORT, not refuse — this was got wrong first

The first cut returned -EBUSY whenever any core had a job. That made it useless for its only purpose: a device in the bad state almost always HAS a stuck job — that is what being stuck means — so it refused at exactly the moment it was needed. Measured: idle device -> ok; degraded device -> -EBUSY. The fix reaps first (rknpu_reap_all_cores(), the driver's own timeout_clean path, which aborts the jobs and releases their references), then re-checks; -EBUSY now only comes back if a job SURVIVES the reap, which is a truthful answer rather than an unhelpful one.

Measured

step	result
REINIT on an idle device	`ok`, counters zeroed
accumulation over 3 runs	`premature_zero` 29 -> 57 -> 96
REINIT on the degraded device	`ok` (was `-EBUSY` before the fix)
state afterwards	`premature_zero=0 underflow=0 map_n=0`
next run, no reboot	`rc=0`, prefill 7.18 tok/s, `createfail=0`

The counters then accumulate again from zero on the next run (premature_zero=29, map_n=71), confirming real state was reset rather than the display being cleared.

Caller contract

The reinit changes IOVAs, so every existing buffer AND every regcmd built against those IOVAs is invalid afterwards. Run it between processes, not inside one. It is manual and explicit — nothing fires it automatically.

This is a workaround, not the fix. The real repair is to stop the count reaching zero under live work: gate get_and_switch() on refcount == 0 and no core having a job, or make the domain reference per-job. See #387.

Not separately cherry-pickable (it uses the reap helper introduced alongside change 9), so the authority is 99-ALL-current-state.patch.

The fence kernel (`6.1.115-vendor-rk35xx-fence`, 2026-08-25)

Built ON THE SBC (~/kbuild/linux-rockchip), armbian/linux-rockchip @ 1cd878b3ac49, changes 1+2+3+4+5, plus CONFIG_ROCKCHIP_RKNPU_FENCE=y (the only config delta besides LOCALVERSION).

CONFIG_LOCALVERSION="-vendor-rk35xx-fence" is deliberate: the running kernel was built with gcc 13.3.0, the board has gcc 14.2, and CONFIG_MODVERSIONS=y — a same-version rebuild could break module loading. A distinct version gives it its own /lib/modules tree and leaves the -sram kernel completely untouched.

Boot safety was checked first: CONFIG_NVME_CORE/BLK_DEV_NVME/EXT4_FS=y and STMMAC_ETH/DWMAC_ROCKCHIP=y are all BUILT-IN, so the board boots and keeps SSH even if no module loads.

Select / revert = one symlink flip (DietPi boots via boot.scr + the Image/dtb symlinks):

sudo ln -sfn vmlinuz-6.1.115-vendor-rk35xx-fence /boot/Image   # or ...-sram to revert
sudo ln -sfn dtb-6.1.115-vendor-rk35xx-fence     /boot/dtb
sudo reboot

Patch 03 will not apply with git apply or patch — its hunk header is a bare @@ with no line numbers. Apply it manually; the script used is ~/kbuild/p03.py on the board.

The three kernels on the board — a ready-made bisect

image	SRAM	job.c patch	date
`vmlinuz-6.1.115-vendor-rk35xx`	✗	✗	stock
`vmlinuz-6.1.115-vendor-rk35xx-sram.prepatch-bak`	✓	✗	Jul 8
`vmlinuz-6.1.115-vendor-rk35xx-sram.pre03-bak`	✓	✓	Aug 11
`vmlinuz-6.1.115-vendor-rk35xx-sram` (+change 3)	✓	✓	Aug 24
`vmlinuz-6.1.115-vendor-rk35xx-fence` (running, +changes 4,5,6, `RKNPU_FENCE=y`, lockup detectors)	✓	✓	Aug 26

cmp confirms the Aug 11 image differs from the Jul 8 backup, so .prepatch-bak is a genuine SRAM-without-the-patch kernel. Booting all three against the same 27B run isolates which change (if either) causes the leak — see Tier 17.

Reverting

sudo sh /boot/RESTORE_STOCK_KERNEL.sh repoints /boot/Image and /boot/dtb at the stock names. Nothing was overwritten; stock kernel, initrd, dtb dir and modules are untouched. Idempotent. If an -sram kernel fails to boot, the script documents a microSD rescue path (mount /dev/nvme0n1p1, chroot, run).

Rules going forward

Every kernel change gets an entry here, with its diff, before it is deployed. Both of these ran for weeks as invisible variables.
Commit them in the build tree. They are uncommitted working-tree edits in a stopped container — one docker rm from being lost permanently, with no diff recoverable anywhere else.
A board measurement is only comparable to another on the same kernel. Record uname -r alongside any result that matters.

Change 18 — light IOMMU domain switch (drop the `default_domain` lie and the force-reset)

Files: drivers/iommu/rockchip-iommu.c, drivers/rknpu/rknpu_iommu.c Patch: attachments/12-rk-iommu-light-domain-switch.patch Kernel build: #46 (2026-08-27). Validated: board make test ALL TESTS PASSED, including orkd_dom_api (domains 1 and 2, bit-exact).

The defect

dma-iommu.c resolves every DMA-API mapping through

static struct iommu_domain *iommu_get_dma_domain(struct device *dev)
{ return dev->iommu_group->default_domain; }

so the DMA API is single-domain by construction. To make a mapping land in domain N, the vendor rknpu driver rewrote the core's notion of the default:

rknpu_dev->iommu_group->default_domain = dst_domain;

reaching the field through a private copy of struct iommu_group pasted into the driver. That one lie is the origin of mismatch domain get from iommu_get_domain_for_dev and of the core reattaching a domain the driver may already have freed.

It was also load-bearing as written: iommu_detach_device() → __iommu_group_set_core_domain() reattaches group->default_domain, and __iommu_attach_group() returns -EBUSY unless group->domain == group->default_domain. Deleting the line alone breaks switching outright.

Worse, rk_iommu_attach_device() → rk_iommu_enable() performs a full rk_iommu_force_reset() of every MMU bank. Without the lie, detach(D1) bounces the device onto the true default before attach(D2) — two force-resets per switch, in exactly the window that races in-flight NPU jobs.

The change

rk_iommu_switch_domain(dev, domain) (new, exported) reprograms the page table in place:

enable_stall -> write RK_MMU_DTE_ADDR on every bank -> ZAP_CACHE -> disable_stall

paging left ON, no force-reset, and no IOMMU-core involvement — so no detach/attach pair, no -EBUSY dance, no transient domain, and nothing to lie about. It also moves the rk_iommu between the two domains' iommus lists, which the TLB-flush paths walk.

rknpu then:

switches with the new primitive instead of iommu_detach_device/iommu_attach_device;
initialises its own IOVA allocator per domain (init_iova_domain(&ck->iovad, SZ_4K, 1) + iova_domain_init_rcaches()), replacing the type |= __IOMMU_DOMAIN_DMA_API + iommu_setup_dma_ops() route whose only real effect was that same iovad init;
resolves the live domain from its own bookkeeping (rknpu_live_domain()) instead of iommu_get_domain_for_dev(), which no longer tracks our switches;
drops the mismatch domain check — it could now only fire spuriously;
deletes the default_domain overwrite, and releases each domain's iovad (put_iova_domain) at teardown.

Safe for our workload because ork-driver allocates NON_CONTIGUOUS | IOMMU_LIMIT_IOVA_ALIGNMENT (0x403/0x40b), which already took the explicit iommu_map_sg()/iommu_unmap() path against a named domain and never used dma_map_sg().

Measured

	kernel #45	kernel #46 (this change)
`mismatch domain`	present	0
`switch iommu domain time out`, multi-domain bench	12	0
pinned prefill / decode tok/s	217.03 / 11.72	217.19, 216.61 / 11.67, 11.67
`make test`	pass	pass (incl. multi-domain bit-exact)

No performance cost, no correctness cost — ZAP_CACHE alone is sufficient, so force_reset was not doing necessary work on this path.

What it does NOT fix

The dispatch stall (job committed, task counter: 0, no interrupt) is untouched and is now the sole blocker: a multi-domain ork_bench run still ends rc=124, blocked in rknpu_submit_ioctl with job iommu domain id: 1, dev iommu domain id: 1 agreeing — i.e. the domain machinery is out of the picture and the hardware simply never completes the task.

The refcount wait in rknpu_iommu_domain_get_and_switch() is also unchanged, so switch iommu domain time out can still occur when a job holds a domain reference (6 seen during make test, non-fatal — all tests passed). Reducing that is separate work.

Change 19 — reclaim-and-retry on a domain-switch timeout (a stall stops being fatal)

File: drivers/rknpu/rknpu_iommu.c (+ dbg_healthy_ctrl in rknpu_job.c) Patch: attachments/13-domain-reclaim-and-retry.patch Kernel build: #48 (2026-08-27). Param: dom_reclaim (default 1).

The problem

A dispatch-stalled job (committed, PC task counter stuck at 0x0, no interrupt) holds its IOMMU domain reference forever. rknpu_iommu_domain_get_and_switch() only proceeds when iommu_domain_refcount reads 0, so from the first stall onward every switch burns its full 6 s and fails — and because rknpu_gem_object_create() switches domains, every later allocation fails -EINVAL (MEM_CREATE errno=22 at zero IOVA used, so not exhaustion). One stall killed the whole run, and the process ended unkillable in D state.

#patch54 already reaped on timeout, but it reaped and then returned -EINVAL anyway, so the caller that triggered the reap still failed; only some later call could benefit. It also required switch_escalate consecutive timeouts (3 × 6 s = 18 s), with any success resetting the counter. Measured: the reproducer still died with switch_escalate=3.

The change

On the first switch timeout: reap the stuck jobs, confirm no core still reports a job, and retry the switch instead of failing the caller. If the reap cleared every in-flight job but the refcount is still non-zero, that count is by definition leaked — no job owns it — so force it to 0. Once only; a second timeout after a successful reclaim returns the error.

The reap runs only after the full 6 s wait, so anything still in flight has been running ≥ 6 s. A real NPU job here is microseconds to milliseconds, so that set is exactly the stuck jobs.

Measured (deterministic reproducer, `ORK_MM_TIMEOUT=3000 ./orkd_dom_api`)

	#47 (Change 18 only)	#48 (this change)
`rc`	124 — hung, unkillable D state	1 — clean exit
`MEM_CREATE errno=22`	1	0
`switch iommu domain time out`	4	1 (the one that triggers reclaim)
`dbg_reclaim_forced`	—	0

reclaim_forced=0 is the important one: the reap released the reference on its own, so the refcount protocol is sound once the stuck job is actually reaped — the force path is insurance, not the mechanism.

A stall no longer hangs the process or kills a domain. Three consecutive runs were identical (stalls=2, create22=0, swtmo=1), so runs no longer poison each other — same-boot A/B testing is valid again, which it was not before (an earlier switch_escalate A/B was uninterpretable because each failure left the next run crashing early at rc=139).

What it does NOT fix

The stall itself, and the individual stalled op still returns an error to userspace — ork-driver does not retry it, so orkd_dom_api reports FAIL rather than PASS. The failure is now contained to one op instead of cascading into a dead device.

Related negative result — the "post-swap danger zone" is not supported

ORK_DOM_WASH=<n> (ork-driver ef354d4) runs n sacrificial tiny ops in the new domain right after a switch. Interleaved same-boot on the reproducer: n=0 → stalls 2,2,1,2; n=1 → 1,2,2; n=3 → 2,2,2,2. No effect. The wash ops are themselves commits into the freshly switched domain, land inside the ~1 ms window, and never stall — while the real op behind them still does. This points away from a positional cause and toward the stall tracking the specific op (ork_dyn_colsplit K=1024 N=128, the same shape every time).

Change 20 — soft reset must reprogram the LIVE domain (fixes the silently-dropped op)

Files: drivers/iommu/rockchip-iommu.c, drivers/rknpu/rknpu_reset.c Patch: attachments/14-soft-reset-live-domain.patch Kernel build: #51 (2026-08-27).

The bug — a regression introduced by Change 18

rknpu_soft_reset() ended with:

if (rknpu_dev->iommu_en)
        domain = iommu_get_domain_for_dev(rknpu_dev->dev);
if (domain) {
        iommu_detach_device(domain, rknpu_dev->dev);
        iommu_attach_device(domain, rknpu_dev->dev);   /* re-establish the MMU after the reset */
}

A hardware soft reset wipes the MMU, so the page table must be reprogrammed afterwards. That code was only ever correct because the OLD multi-domain path overwrote iommu_group->default_domain on every switch, so iommu_get_domain_for_dev() happened to return the live domain.

Change 18 deliberately took the IOMMU core out of the switch path. iommu_get_domain_for_dev() therefore reports the group's default domain (domain 0), and this sequence re-attached domain 0 while rknpu_dev->iommu_domain_id still said domain N. Every IOVA the next job used then resolved against the wrong page table, so the job was committed and never completed — no interrupt, no error, an output buffer left holding its seed.

Symptom: the first int8 op after a mode-transition ACT_RESET was silently dropped. It was invisible for a long time because it only bites when a non-zero domain is live — single-domain runs (ORK_PIN_DOM=0) were always clean, and make test passed because ~40 earlier tests had already made the transition.

The fix

New exported rk_iommu_reprogram(dev): unconditionally re-enable the MMU from iommu->domain, which the light switch keeps pointed at the live domain, so the correct page table is restored. Unconditional matters — rk_iommu_switch_domain() short-circuits when the domain is unchanged, and here the domain has NOT changed; the hardware has been wiped. rknpu_soft_reset() calls it instead of the iommu_get_domain_for_dev() detach/attach pair.

Measured

	before (#50)	after (#51)
standalone `orkd_dom_api` (default)	MISMATCH, 5/5	PASS bit-exact, 3/3
reproducer at `ORK_MM_TIMEOUT=3000`	FAIL 5/5, 2 stalls + 1 swtmo each	PASS 3/3, stalls=0, swtmo=0
board `make test`	pass, 1 sentinel miss	pass, 0 sentinel misses

Zero sentinel misses is the strongest signal: the drop is eliminated, not merely reported. ORK_XSPEC_NORESET=1 is no longer needed and its arm still passes.

Cross-reference: mainline `accel/rocket`

Worth recording, since it argues the vendor design is the outlier. In mainline, rocket_core_reset() performs NO IOMMU operations — it only asserts/deasserts the reset lines. rknpu's soft reset additionally entangles an IOMMU detach/attach with the hardware reset, and that extra step is exactly what broke. Rocket also has no stall detection at all: no task-counter monitoring, no missing-interrupt logic, just a 500 ms drm_sched timeout (rocket_job_timedout() -> rocket_reset() -> restart the scheduler). So mainline offers no evidence that the RK3588 dispatch stall is a known silicon behaviour — it simply treats "unfinished in 500 ms" as generic failure.

Change 21 — audit: every remaining `iommu_get_domain_for_dev()` caller (4 more stale-domain bugs)

Files: drivers/rknpu/{rknpu_gem.c,rknpu_reset.c,rknpu_iommu.c,include/rknpu_iommu.h} Patch: attachments/15-live-domain-audit.patch Kernel build: #52 (2026-08-27). Validated: board make test ALL TESTS PASSED, 0 sentinel misses.

Change 18 made iommu_get_domain_for_dev() a known-bad accessor in this driver: with the IOMMU core out of the switch path it reports the group's DEFAULT domain (domain 0), never the live one. Change 20 fixed one caller. This is the exhaustive sweep of the rest.

site	what it did	verdict
`rknpu_iommu.c` `init_domain()`	captures domain 0 at init	correct — at init the core's domain IS domain 0
`rknpu_gem.c:75`	`iommu_flush_iotlb_all()` after a map	WRONG — flushed domain 0's IOTLB, leaving stale translations for pages just mapped into domain N
`rknpu_gem.c:504`	`iommu_map()` target, SRAM/NBUF cache buffers	WRONG — mapped the cache buffer into domain 0
`rknpu_gem.c:657`	matching `iommu_unmap()`	WRONG — unmapped from domain 0
`rknpu_reset.c` per-core reset	detach/attach re-establish	WRONG — the twin of the Change 20 bug, which that change missed

The per-core one matters most: it is the path RKNPU_ACT_RESET_CORE and the progress watchdog drive, so it fires during exactly the multi-domain runs under investigation. Change 20 fixed the device-wide rknpu_soft_reset() and left its per-core counterpart carrying the identical bug.

Fix: rknpu_live_domain() is exported as rknpu_iommu_live_domain() (declared in include/rknpu_iommu.h with a comment stating why the core accessor must not be used), the three gem sites use it, and the per-core reset uses rk_iommu_reprogram() exactly as Change 20 does.

These are real bugs, but they are NOT the multi-domain stall

Benched on #52, 1.5B Q8_0, same configuration as #51:

mode	rc	stalls	errno=110	perf
multi	124	21	20	—
multi	124	21	20	—
pinned	0	0	0	219.94 tok/s

Byte-identical to #51. So the "another live-domain divergence" hypothesis is weakened, not confirmed: four genuine instances of that exact class were found and fixed, and the stall did not move. Recorded so the next investigation does not re-run this reasoning.

Change 22 — imports must map into the LIVE domain (★ this is the multi-domain dispatch stall)

Files: drivers/rknpu/{rknpu_gem.c,rknpu_iommu.c,rknpu_drv.c,include/*}, drivers/iommu/rockchip-iommu.c Patch: attachments/16-import-domain-fix.patch Kernel build: #54 (2026-08-27). Pairs with ork-driver 4f8db7f.

The bug

The "dispatch stall" — job committed, PC task counter 0x0, no interrupt, no error — was never silicon. A dma-buf import's sg is mapped in two places that both got the domain wrong:

DRM core, at attach time. dma_buf_map_attachment() runs inside DRM_IOCTL_PRIME_FD_TO_HANDLE, before any rknpu hook, and maps through the DMA API — which always targets iommu_group->default_domain. Change 18 removed the vendor's default_domain overwrite, so from then on every import mapped into domain 0 while being recorded as domain N.
Ordering. Even with (1) fixed, PRIME_FD_TO_HANDLE carries no domain; the target is named only by the later MEM_CREATE, which does not re-map an already-imported handle. So the mapping lands wherever the device happened to be.

Native packs were never affected: their MEM_CREATE does the switch and the mapping together.

The fix

22a (kernel): rknpu_iommu_default_swap() — scope an override of the core's default domain to the attachment map, in the .gem_prime_import wrapper, and to the matching unmap at destroy. Deliberately not the vendor's overwrite: that one was permanent, so any core path reattaching "the default" used a domain the driver might free. This lives for one call, under domain_lock, and is restored immediately.
22b (kernel + userspace): RKNPU_ACT_SET_DOMAIN (0x102), a switch-only action with no allocation. ork-driver issues it before the PRIME ioctl in both import paths, so the mapping lands in the domain the weight will actually be used in.

Measured — 1.5B Q8_0 multi-domain `ork_bench` (failing since the start of the investigation)

	rc	stalls	errno=110	prefill
every kernel #45–#53	124	21	20	—
#54	0	0	0	219.09 / 217.88 tok/s
pinned (reference)	0	0	0	218.30 tok/s

Multi-domain now matches single-domain performance exactly, with coherent output. Board make test: ALL TESTS PASSED, 0 sentinel misses.

How it was isolated

tools/dom_scale_probe.c made the weight SOURCE the only variable — native packs passed 600 cycles bit-exact while dma-buf imports failed at cycle 0, and imports passed when pinned to domain 0 (live == default). That predicted-then-confirmed pair named the mapping domain.

Note on the tooling

vm_kbuild.sh used to install the PREVIOUS image after a failed build, so the board booted a stale kernel while the log said "installed". A compile error in this very fix was briefly scored as "the fix does not work". It now aborts instead (ork-driver 4f8db7f).

Change 23 — the progress watchdog is a DEVELOPER tool: default off

File: drivers/rknpu/rknpu_drv.c · Patch: attachments/17-watchdog-default-off.patch Kernel build: #55 (2026-08-27).

rknpu_wd_period_us defaulted to 10000, so stall detection re-armed on every boot. It was written to chase "job committed, PC task counter 0x0, no interrupt" — which turned out to be our own multi-domain page-table bugs (changes 20–22), not hardware. Mainline accel/rocket carries no stall detection at all and does not need any.

With those fixed the watchdog should never fire, so leaving it armed only buys a 100 Hz timer and the risk of misreading a slow-but-healthy job: the threshold is wd_samples * wd_period_us ≈ 120 ms, and a cold weight-DMA can exceed that and still complete bit-exact (measured — several detections per suite run with every output bit-exact).

It stays valuable for developers, because a wedge-prone shape can still hang the NPU and a hang costs a reboot — or, for wide 27B prefill, a physical SPI reflash. So it is opt-in, in two stages:

echo 10000 > /sys/module/rknpu/parameters/wd_period_us   # detect + log
echo 2000  > /sys/module/rknpu/parameters/wd_abort_ms    # AND fail the submit (opt-in on top)

wd_abort_ms was already 0 by default — abort is the dangerous half and stays strictly opt-in on top of detection. The counters (cnt_commit/cnt_done/cnt_irq) remain always-on and free; they are what actually localised the bug this was built for.

Recovery mechanisms stay armed (dom_reclaim=1, switch_escalate=3) — unlike detection, they only act on a failure that has already happened.

Validated: kernel #55, board make test ALL TESTS PASSED, 0 sentinel misses. Note a dmesg "0 stalls" is now trivially true with detection off; the evidence that nothing was dropped is the 0 sentinel misses, a userspace guard that fires regardless of the kernel watchdog.

Where the kernel work lives

what	where	upstream
The five reference-counting / teardown fixes	`oRKLLM/rk3588-kernel` branch `rknpu-iommu-fixes`	PR #388 — closed, superseded
Those five plus the multi-domain rework (changes 18–22)	same repo, branch `rknpu-iommu-rework`	PR #390 — open, closes #387
Diagnostics (watchdog, counters, probes — change 23 and friends)	the SBC working tree + attachment 18 below	not proposed upstream

Both branches are forks of the vendor tree (rockchip-linux/kernel, develop-6.1), not the armbian distro copy, so they carry correct history and can serve PRs.

#388 was closed in favour of #390 because the five fixes alone are not sufficient for multi-domain use — see the A/B in the Experiment-Log entry for 2026-08-28. #390 contains those five commits unchanged plus five more that take rk_iommu_force_reset() out of the switch path.

Build status of #390

★ 2026-10-09 — #390 HAS NOW BEEN BOOTED, AND IT IS NOT SUFFICIENT. Built on a second board (10.3.0.235) as armbian + exactly these ten commits and nothing else, orkd_dom_api fails 3/3 with six switch iommu domain time out + rknpu_gem_object_create error. The missing piece is Change 19 / #patchB5 below: #390 clamps the domain refcount at zero but cannot recover a count that is already too high, so one leaked reference wedges every later switch and every later allocation — the failure mode #390's own rknpu_iommu_domain_put() comment describes. Porting B5 onto the series takes it to PASS 3/3 with the reclaim observed firing. The runtime testing recorded below was done on the armbian tree carrying the full board state, which silently included B5 — which is why this never showed up before. Details, minimal 98-line patch and the two honest limits: Exp-2026-10-09.

drivers/iommu/ and drivers/rknpu/ compile against develop-6.1 with no errors and no warnings, and a full Image links with rk_iommu_switch_domain / rk_iommu_reprogram correctly exported. It has not been booted on develop-6.1 itself — the runtime testing was on the armbian 6.1 tree, whose rknpu driver is byte-identical (0/0/0 differing lines in rknpu_job.c, rknpu_gem.c, rknpu_reset.c; 2 in rknpu_iommu.c), though its rockchip-iommu.c differs by 61 lines.

Two pre-existing build failures in that tree with an armbian-derived config, neither caused by the series: drivers/input/touchscreen/focaltech_touch (mm_segment_t/get_fs) and a modules_check module name conflict.

Still only on the SBC

The diagnostics and the rk3588s.dtsi SRAM enabler are not in either branch. The dtsi is excluded deliberately: it is unrelated to the domain work, and vendor vs armbian differ by 1442 lines there, so it must be re-derived rather than copied. attachments/18-full-working-tree-2026-08-27.patch remains the one authoritative snapshot of the complete SBC state including the dtsi.

The SRAM enabler — OPTIONAL, and not intended for mainstream

This is an opt-in experiment, not part of the NPU fixes. It is unrelated to the multi-domain work, it is deliberately excluded from PR #390, and it should not be proposed upstream: it is a resource-allocation policy choice, not a defect fix — it takes on-chip SRAM away from the hardware video decoders. On a board doing inference that is a reasonable trade; as a general default it is not.

Its value is also unproven for our workload. ORK_SRAM_REGCMD was A/B'd on 2026-08-26 and showed no measurable effect; an initial apparent win was a false positive caught by the A-B-A control.

Packaging caveat (open): as carried on the SBC it is a patched rk3588s.dtsi, i.e. compiled into the kernel image — so it is NOT optional for anyone running that kernel. The correct form for an optional feature is a device tree overlay (.dtbo) enabled per board through overlays= in armbianEnv.txt: opt-in, reversible by removing one line, and it survives kernel package upgrades without carrying a patch. If this is ever shipped, it should be converted to an overlay first.

What it does — gives the NPU the 956 KB rkvdec0_sram block, which is otherwise unused by the NPU:

rknpu: add rockchip,sram = <&rkvdec0_sram>
rkvdec0 / rkvdec1: drop their rockchip,sram references
syssram: merge rkvdec0_sram (0x0 0x78000) and rkvdec1_sram (0x78000 0x77000) into a single rkvdec0_sram of 0x0 0xef000

Pairs with CONFIG_ROCKCHIP_RKNPU_SRAM=y. (0x78000 = 480 KiB and 0x77000 = 476 KiB merge into 0xef000 = 956 KiB.) Reversible, but only by rebuilding the kernel while it lives in the dtsi.

What the board is actually running — `#59` (2026-09-01)

Record this with any measurement. The board runs a locally-built kernel that is ahead of every merged branch, and that has been a hidden variable in this project before. As of 2026-09-01:

6.1.115-vendor-rk35xx-fence #59 SMP Tue Sep  1 02:29:04 UTC 2026

It is NOT any PR branch

The build source is the board's own tree (~/kbuild/linux-rockchip), which tools/util/vm_kbuild.sh syncs from the board into the build VM. That tree is the diagnostic tree: ~49 numbered hunks (#patch29–#patch70, #patchB1–#patchB12), most of which exist in neither develop-6.1 nor the clean fork branches. oRKLLM/rk3588-kernel's branches carry only the upstreamable subset.

So: a result measured on #59 is not a result about any PR. Two consequences worth stating plainly:

Nothing here is merged to develop-6.1. rknpu-job-list-uaf (PR #2, issue #1) and rknpu-null-domain-guard (PR #4, issue #3) are both open.
The reverse also holds: the fork branches contain a cleaner form of some fixes than the board. The null-domain guard is the example — the board reached it as #patch33/34, and the fork branch reworks it into a single rknpu_iommu_cookie() accessor.

The delta that matters for #59 vs #58

hunk	what	where it lives
`#patch67`	unlink an aborted job from the core `todo_list` before freeing it	PR #2 (as a rewritten commit)
`#patch68`	NULL-guard `iommu_flush_iotlb_all()` in `rknpu_gem_create()`	PR #4
`#patch33/34`	NULL-check the IOMMU cookie (map/unmap)	PR #4, reworked into one accessor
`#patch69/70`	GEM-destroy bail counter + deferred reclaim of objects whose destroy could not switch domains	neither — another agent's in-progress work, present in the board tree

#patch69/70 are not mine and are not in either PR. They were already in the board tree when #59 was built, and because the build syncs from the board, #59 carries them. I did not verify whether #58 also had them, so #58 → #59 should not be treated as a clean single-variable step.

The trap this documents

vm_kbuild.sh builds from the board's tree, not from a git branch. So any edit another agent leaves in ~/kbuild/linux-rockchip is silently included in the next kernel anyone builds, and attributed to whatever that person thought they were testing. Before treating a kernel build as validating a specific change, diff the board tree against the branch you think you are testing — the patch markers (grep -ohE "#patch[0-9A-Za-z]+" drivers/rknpu/*.c drivers/iommu/rockchip-iommu.c | sort -u) are the cheapest inventory.

Validation status of `#59`

ork-driver make test ALL TESTS PASSED, 0 oops, 0 null-domain warnings. That covers the composite tree, not the PRs in isolation. Neither reported crash was reproduced — both need a submit or a teardown racing a domain switch, which the suite does not construct.

ramoops/pstore is now registered on this board (pstore: Registered ramoops as persistent store backend), so the next panic should survive the reboot rather than needing netconsole to catch it.

A76 clusters unlocked to 2400 MHz (2026-09-01)

Board modification, not a kernel-repo branch. Rockchip's binning gates are deliberate, so removing them is not upstreamable; this lives here and in the board tree only.

What was changed

arch/arm64/boot/dts/rockchip/rk3588s.dtsi, both cluster OPP tables — 8 edits, 4 OPPs × 2 clusters:

opp-2256000000:  opp-supported-hw = <0xf9 0x13>  ->  <0xf9 0xffff>
opp-2304000000:  opp-supported-hw = <0xf9 0x24>  ->  <0xf9 0xffff>
opp-2352000000:  opp-supported-hw = <0xf9 0x48>  ->  <0xf9 0xffff>
opp-2400000000:  opp-supported-hw = <0xf9 0x80>  ->  <0xf9 0xffff>

0xffff is not invented — it is the "all bins" mask this same file already uses for 2208 MHz and below, so this makes the top of the ladder follow the file's own existing convention.

Why this is not an overclock

Each top OPP was gated by a mutually exclusive opp-supported-hw bin mask, selected at boot by a live PVTM (ring-oscillator) measurement per cluster — which is why the two A76 clusters landed on different maxima (2304 and 2352) from identical device-tree tables. The decisive detail is the voltage: 2304, 2352 and 2400 all specify the same 1.00 V (opp-microvolt = 0xf4240), and the rail already sits at 1.00 V with a 1.05 V ceiling. So enabling 2400 asks for no voltage change and no out-of-spec operating point — RK3588's A76 is specified at 2.4 GHz, and the part was factory-conservative, not incapable.

Result and validation

before   policy4 cpuinfo_max=2304000    policy6 cpuinfo_max=2352000
after    policy4 cpuinfo_max=2400000    policy6 cpuinfo_max=2400000

ork-driver make test — ALL TESTS PASSED, 0 kernel errors
120 s sustained load on all four A76s: held 2400000 on both clusters throughout, 46 °C -> 49 °C, no throttling, no Internal error / hardware error / thermal events
cpu_capacity is now uniform across the big cluster (cpu4-7 = 1024; little cores renormalise 405 -> 397)

The residual risk was never voltage but timing closure at the higher clock on this particular die, which is exactly what the sustained-load run tests. It passed.

Side effect worth knowing

This removes the asymmetry behind ork-driver#6. That issue's real finding — a cpu_capacity == max test selecting only cpu6,7 and costing 1.42x — is now masked on this board, because all four big cores report 1024. The underlying bug is not fixed by this. Any consumer still using == max will break again on any part whose clusters bin differently. ork-driver uses a largest-relative-gap split (ork_big_core_set, commit 4052202), which is correct either way.

Reverting

DTB and dtsi backups are on the board:

~/ork-outbox/rk3588-rock-5b.dtb.pre-opp2400     # the DTB that was installed before
~/ork-outbox/rk3588s.dtsi.pre-opp2400           # the source before the edit
~/ork-outbox/rk3588s.dtsi.a76-2400              # the source after (re-apply after a tree refresh)

sudo cp ~/ork-outbox/rk3588-rock-5b.dtb.pre-opp2400 \
        /boot/dtb-6.1.115-vendor-rk35xx-fence/rockchip/rk3588-rock-5b.dtb   # then reboot

The board tree is armbian-derived and the Rockchip vendor tree differs here — armbian uses <0xf9 ...> for the SoC-version cell, the vendor tree <0xe9 ...>. A sed written for one silently matches nothing in the other, so check which tree you are editing. The edit is also not carried by vm_kbuild.sh, which syncs only drivers/rknpu/**, drivers/iommu/rockchip-iommu.c and .config — a kernel rebuild will not lose it (the DTB is separate), but a fresh kernel tree will not have it either.

#patch73 — an async job must own its own power reference (kernel #62 onward)

Fixes rk3588-kernel#5 · PR rk3588-kernel#6 · branches rknpu-nonblock-power (fix) and rknpu-job-counters (fix + instrumentation).

The bug

RKNPU_IOCTL() brackets every ioctl in rknpu_power_get() / rknpu_power_put_delay(), so a submit's power reference is scoped to the ioctl call:

#define RKNPU_IOCTL(func) ... {
        rknpu_power_get(rknpu_dev);
        ret = func(dev, data, file_priv);
        rknpu_power_put_delay(rknpu_dev);
        return ret; }

For a blocking submit that is exactly right — the ioctl does not return until rknpu_job_wait() has seen the job finish, so the reference covers the whole job. For a NONBLOCK submit it is wrong by construction: the ioctl returns the moment the job is committed, the refcount falls to zero, rknpu_power_off_delay_work() fires, and the block is powered down underneath a live job.

The completion interrupt then arrives at an unpowered block. Two endings, and this project hit both:

Unguarded — the machine dies. The handler's first readl() of the core's register window takes an external abort, no console output, no recovery but a power cycle (pc : readl+0x4/0x20, lr : rknpu_irq_handler). Fixed 2026-08-27 by a power_refcount <= 0 bail.
Guarded — a core wedges forever. The interrupt is simply dropped. interrupt_count is decremented in exactly one place, rknpu_job_done(), reached from exactly one place, the IRQ handler. So the completion is never accounted: subcore_data->job stays set, that core is owned forever, and every later submit lands in todo_list behind a job that will never finish.

So the 2026-08-27 guard converted a fatal oops into a silent wedge. Both are one root cause. That commit called the interrupt "late or spurious"; it was neither — it was a real completion for a job still running, and the reference should never have been dropped. Corrected here.

Evidence

core 0: NOT dispatching -- still owned by job (flags 0x2, age 60313754us, int_cnt 1, run_cnt 0)

flags 0x2 = RKNPU_JOB_NONBLOCK, run_cnt 0 = dispatched, int_cnt 1 = never completed. And cnt_unpow = 3441 completion interrupts dropped in one session.

The fix

Give the job its own reference — the same shape as #patch64's dom_held for IOMMU domains. A pwr_held bit, taken at the NONBLOCK branch, released in rknpu_job_free().

Three placement details that are easy to get wrong, and I got two of them wrong first:

Not rknpu_job_done() — rknpu_job_timeout_clean() tears down a stale async job by scheduling cleanup_work directly, reaching neither job_done() nor job_abort(). job_free() is the one point every job passes through exactly once.
Context — rknpu_power_put_delay() takes power_lock, a mutex, and job_done() runs in hard IRQ. My first version released there: sleep-in-atomic.
Conditional — the early-error paths in rknpu_job_submit() call job_free() before the reference is ever taken, so test_and_clear_bit is doing real work, not decoration.

The IRQ guard stays as belt-and-braces for a genuinely spurious interrupt.

Result

	before	after
dropped completion interrupts (`cnt_unpow`)	3441	0
interrupts / completions	—	109035 / 109035
stuck-owner reports	present	0
`test_f16colsplit` in the suite	wedged ~1 run in 2	4/4 bit-exact

test_f16colsplit is un-quarantined in ork-driver as of this patch and now runs in make test. On a kernel without #patch73 it fails roughly every other run.

⚠ This is a counterexample to a standing project rule

Vendor-Kernel-Behaviour says: "job committed, PC task counter 0x0, no interrupt, no error" has NEVER once been silicon; every instance was a page-table mismatch. At this stall the domains matched (submit_dom=0 weight_dom=0) and it was not a page-table problem at all. The rule is still a good first bet, but it is no longer exhaustive — check cnt_unpow and the stuck-owner log before assuming domains.

#patch71 / #patch72 — the instrumentation that found it

Both were board-only for a while; #patch71 is now committed as rknpu-job-counters.

#patch71 — job counters and the stuck-owner detector

Seven counters that separate the three ways a submit can produce no output: never committed / committed, no interrupt / completed, wrote elsewhere. Plus a report naming a job that owns a core it will never release (flags, age, interrupt_count, run_count, capped at 16 lines).

Cost, and how it is gated. commit / irq / done / next_blocked are per-job — ~85k increments per suite pass across three cores — so they are off by default behind a static key: when off the call sites are patched to a nop, not a branch on a flag. Verified: counters stay flat at 0 through a full workload with the gate off.

unpow / nojob / blocked_slow are never gated. Their paths are not taken on a healthy driver (all read 0 across a full suite), so the increment is free, and each names a specific defect. cnt_unpow is the standing regression canary for #patch73 — if it is ever nonzero again, that bug is back.

Control uses the driver's own debugfs facility, live, no reboot and no module reload:

echo on    > /sys/kernel/debug/rknpu/counters   # start counting
cat          /sys/kernel/debug/rknpu/counters   # labelled dump + gate state
echo reset > /sys/kernel/debug/rknpu/counters   # zero them

Raw values stay at /sys/module/rknpu/parameters/cnt_* for programmatic readers, with dbg_on alongside saying whether collection is on. That flag is not optional: ork-driver's doorbell probe classifies a stall from these counters, and without it a gated-off zero reads as "never committed" — a confident wrong answer at exactly the moment someone is root-causing a wedge.

The other debug surface already in the driver: LOG_DEBUG is pr_devel, and with CONFIG_DYNAMIC_DEBUG=y its 4 call sites are switchable at zero cost with echo 'module rknpu +p' > /sys/kernel/debug/dynamic_debug/control.

#patch72 — re-drive the queue after a soft reset

Fixes rk3588-kernel#7 · PR rk3588-kernel#8 · branch rknpu-redrive-after-reset. Independent of #patch73.

Not the fix for the stall above, and it should not be recorded as one. It closes two real but separate holes around soft_reseting:

The window. rknpu_job_next() returns immediately for every core while soft_reseting is set, and its only callers are the completion path, rknpu_job_schedule() and rknpu_job_timeout_clean(). A job declined inside that window is never re-driven — the reset clears the flag, but the core is idle, so no completion interrupt is coming to call job_next() again, and the job sits on todo_list forever. The realistic way in is rknpu_job_timeout_clean(), which calls rknpu_soft_reset() from inside the submit path: a late completion for the timed-out job arrives during the reset, clears subcore_data->job, is declined, and strands whatever is queued behind it.
The failure path. soft_reseting is left true when reset assert/deassert fails, so dispatch is disabled for the whole device permanently. A failed reset does not degrade the NPU, it silently stops it. This one needs no race.

Placement matters (the board's first version had it wrong on both counts): re-drive after state_init(), because committing a job before it programs a half-initialised block, and after dropping reset_lock, because a commit can lead back into rknpu_soft_reset(), whose mutex_trylock() would then fail and silently skip a reset that was needed.

What the measurement actually says

56 soft resets across a full suite pass, cnt_stranded = 0 and cnt_reset_declined = 0. The race window does not open in our workload. Hole 1 is clear in the code but has never been observed biting us — do not cite it as the cause of a stall you are chasing. Hole 2 needs no window and is plain from the code.

Two always-on counters make it visible if it ever does open: cnt_stranded (jobs found queued on an idle core when dispatch came back — each would have hung) and cnt_reset_declined (dispatch attempts refused because a reset was running). Both on the reset path, so free.

⚠ Do not try to force the window with tiny submit timeouts. ORK_MM_TIMEOUT=5 on test_matmul wedged the board twice through reset thrash, and it did not open the window either — both counters were still 0. This is the reset-thrash hazard Vendor-Kernel-Behaviour already warns about; it cost two recoveries here for no information.

Dead ends, recorded so they are not re-tried

Prior fault state — falsified. It reproduced in a boot with job faults = 0.
A bad mode left by a predecessor test — falsified. A 36-test prefix (the exact suite predecessors) survived at every N.
Domain placement — falsified. ORK_PIN_DOM=0, 0 switches, and matching domains at the stall.

#patch74 — the abort-path double `list_del` (the `.235` panic of 2026-10-11)

Fixes the kernel Oops at rknpu_job_abort+0x160, dead000000000108. Upstream oRKLLM/rk3588-kernel branch rknpu-iommu-rework, commits f64e73a + d147723, pushed to rockchip-linux/kernel#390.

★ This is a regression introduced by PR #390 itself, by its own commit afc9c0412e10 ("unlink an aborted job from the core todo_lists before freeing it", our #patch67). Before that commit the overlapping case was a latent, timing-dependent UAF; after it, it is a deterministic panic for any blocking submit that times out while queued. The two must land together.

The mechanism

There are two unlinks of the same job->head[i], back to back on one thread:

#	site	call
1	`rknpu_job_wait()`, the `!last_task` / `"job commit failed"` branch	`list_del()` — poisons the node
2	`rknpu_job_abort()` (added by `#patch67`)	`list_del_init()` — faults on the poison

rknpu_job_wait() has one caller, and rknpu_submit() calls rknpu_job_abort() on every error return from it, so #2 always follows #1. list_del_init() is idempotent against another list_del_init(), not against list_del() — both reach __list_del(prev, next), whose first store is next->prev = prev, i.e. a write to LIST_POISON1 + 8 = 0xdead000000000108 on arm64.

list_del() in rknpu_job_wait() was the only poisoning list deletion in drivers/rknpu/.

Preconditions

Not a race. One thread, one submit, two statements:

a blocking submit — RKNPU_JOB_PC set and RKNPU_JOB_NONBLOCK clear, so rknpu_job_wait() is entered at all (see Vendor-Kernel-Behaviour §7);
the job times out while still queued behind a job the hardware never completed, so job->last_task is still NULL.

In ork-driver the trigger is ORK_I4_BLOCKVERIFY=1 (src/npu/i4/chain.c:419), which is the only int4 mode that takes the blocking submit; the default NONBLOCK doorbell never calls rknpu_job_wait(). That is why this never appeared before and why it is not an ORK_I4_BLOCKVERIFY bug — that knob only reaches a kernel path the async path does not.

`d147723` — a second defect found in the same loop

rknpu_job_wait()'s cleanup iterated for (i = 0; i < job->use_core_num; i++) while head[] and subcore_datas[] are indexed by core index. A single-core submit on core 1 or core 2 therefore inspected list 0 and unlinked nothing. Masks dense from core 0 are unaffected, which is why it survived. Since #patch67 the stale link is cleaned up by the abort path, so it is no longer a UAF on its own — fixed anyway rather than leaving correctness resting on that.

Evidence

Full pstore capture (console + dmesg ramoops, provenance, md5s): ~/ork-board-evidence/235-rknpu-oops-2026-10-11/ off-board, ~/work/crash/ and ~/ork-outbox/235-rknpu-oops-2026-10-11/ on .235.

Three things pin it to that pair of statements rather than to a generic UAF:

x0/x1 are LIST_POISON2/LIST_POISON1 — the literal arguments of __list_del(). A UAF through reused slab gives arbitrary values.
WnR = 1 — a write, matching next->prev = prev.
"job commit failed" at [763.827425], Oops at [763.933142] — 106 ms, which is rknpu_job_abort()'s own msleep(100) sitting between the two unlinks.

A local (not upstreamed) refcount diagnostic also fired five times that boot: four at 1 job(s) IN FLIGHT, all survivable, and the machine died on the first at 2 job(s) IN FLIGHT. That matches the mechanism — a second job in flight is what keeps subcore_data->job occupied so the first never completes and the second is still queued with last_task == NULL. With one job in flight the branch is unreachable.

The non-destructive reproduction did NOT fire — and what that is worth

With CONFIG_DEBUG_LIST=y the fault becomes a WARN from __list_del_entry_valid() naming LIST_POISON1 instead of a panic (CONFIG_BUG_ON_DATA_CORRUPTION=n; validators confirmed live in /proc/kallsyms). The intent was a non-destructive before/after A/B. It did not reproduce. ~1 h of the triggering workload on the unfixed -predbg kernel:

arm	`job commit failed`	`list_del` WARNs
3 cores, `ORK_MM_TIMEOUT` 60 s (default)	5	0
`ORK_NPU_MC=2`, 3 s	1	0
3 cores, 3 s	3	0
total	9	0

Blocking drops were distributed 2 / 4 / 6 across cores 0 / 1 / 2, so core 0 does fail — but no core-0 failure ever coincided with the soft-reset window that produces the commit-failed path. The panic needs that conjunction, which is why the original took 763 s of a long session to happen once.

Two explanations survive, and this data cannot separate them. (a) the core asymmetry — only a core-0 job is poisoned, because rknpu_job_wait()'s loop indexes by use_core_num and a single-core job on core 1/2 searches list 0 and never finds itself; or (b) rknpu_job_next() had already dequeued the job with list_del_init() before the reset, leaving nothing to poison. The per-core numbers above are aggregate drop counts, not per-event attribution, so they are consistent with (a) without establishing it. Anyone resuming this should add the core mask to the "job commit failed" log line first — that one change distinguishes them.

The panic trace remains the evidence of record, and it does not need the WARN: register-level confirmation of the LIST_POISON arithmetic plus the 106 ms msleep(100) interval tying both unlinks to one submit.

★ Positive result extracted from the failed arm: the race needs a SECOND submit stream

ORK_NPU_MC=1 produced 86 soft resets and ZERO job commit failed. Pinning to one core removes the bug's trigger entirely — with a single submit stream there is nothing to land on a core while another core's abort is soft-resetting the device. That is a structural statement about the mechanism, and it also means ORK_NPU_MC=1 is a genuine mitigation on an unfixed kernel.

⚠ `.236` was never immune

board-snapshot-59 (.236's #59) carries #patch67 and the poisoning list_del — the same pair, verified in the tree. .236 simply does not enter the permanent int4 doorbell-miss regime that makes last_task == NULL common, and the project's own earlier instrumentation measured queued-aborts=0 there (Change 14). Do not read .236's clean history as protection.

Kernels on `.235` after this work (2026-10-12)

Armbian boots whatever /boot/{Image,uInitrd,dtb} point at, so switching is three symlink flips and a reboot — no files are written to /, so the switch itself carries no brick risk. /boot/SELECT_KERNEL.sh does all three together and refuses an incomplete release; run it with no argument to list what is installed and what is selected.

release	contents	use
`6.1.115-vendor-rk35xx`	stock Armbian	always-safe fallback (`/boot/RESTORE_STOCK_KERNEL.sh`)
`6.1.115-vendor-rk35xx-npu`	`board-snapshot-2026-10-09` + domain-ref-leak fix	the kernel that panicked. Kept only as the pre-fix control
`6.1.115-vendor-rk35xx-fix`	^ + `#patch74`	PRODUCTION
`6.1.115-vendor-rk35xx-fixdbg`	^ + `CONFIG_DEBUG_LIST=y`	diagnostic only — do not benchmark on it
`6.1.115-vendor-rk35xx-predbg`	unfixed + `CONFIG_DEBUG_LIST=y`	reproduces the bug as a non-fatal WARN; delete once that A/B is no longer wanted

★★ `.236`'s A76 2400 MHz ungating lives ONLY in its installed DTB — ANY rebuild loses it

This closes the open item this page previously left as ".236's own pvtm-volt-sel is unread … worth capturing at its next natural reboot". Captured 2026-10-12. The answer is worse than "unread":

.236 was running at cpu4/cpu6 2400 MHz because its installed dtb-…-fence has opp-supported-hw = <0xf9 0xffff> on the top four A76 OPPs in both clusters — the PVTM speed-bin gate lifted. That edit exists in no committed tree. Rebuilding the DTB from board-snapshot-2026-10-09 / #72 reproduces the stock masks:

 -fence     (installed, known-good)   opp-supported-hw = <0xf9 0xffff>   x4
 -fencefix  (rebuilt from the tree)   opp-supported-hw = <0xf9 0x13>, <0xf9 0x24>,
                                                        <0xf9 0x48>, <0xf9 0x80>

and the board silently comes up at cpu4 2304 / cpu6 2352 instead of 2400/2400 — a ~4% big-core clock loss, on the measurement board, invisible unless you read scaling_cur_freq. That is exactly the shape of hidden variable this page exists to prevent, and it bit the #patch74 install until the post-reboot check caught it before any benchmark was taken.

Unlike .235, .236 cannot fix this with an overlay: its dietpiEnv.txt has user_overlays= empty and there is no rk3588-a76-2400.dtbo on the board.

Rule: when installing a rebuilt kernel on .236, do NOT ship the rebuilt board DTB. Copy the known-good one forward so the device tree is a non-variable and only the kernel changes:
sudo cp -f /boot/dtb-6.1.115-vendor-rk35xx-fence/rockchip/rk3588-rock-5b.dtb \
           /boot/dtb-<new-release>/rockchip/rk3588-rock-5b.dtb
Verify after every reboot: cat /sys/devices/system/cpu/cpu{4,6}/cpufreq/scaling_cur_freq must read 2400000. Do not assume it; the DTB is the only thing carrying it.

The durable fix is to get the ungating into the board tree (or an overlay) so a rebuild keeps it.

⚠ `.236` and `.235` DO NOT share a boot contract

	`.235` (ROCK 5B+)	`.236` (ROCK 5B)
distro	Armbian	DietPi
env file	`armbianEnv.txt`	`dietpiEnv.txt`
initrd	per-kernel `uInitrd-<rel>` (u-boot wrapped)	one shared raw `initrd.img` → the STOCK initrd
symlinks that select the kernel	three: `Image`, `uInitrd`, `dtb`	two: `Image`, `dtb`
board dtb	`rk3588-rock-5b-plus.dtb`	`rk3588-rock-5b.dtb`
A76 2400	`user_overlays=rk3588-a76-2400`	in the installed DTB only (above)

.236's boot.cmd does load … ${prefix}initrd.img + booti — a raw initrd, no uInitrd. Running update-initramfs/mkimage there, as the Armbian installer does, is wrong for this board. An installer written for one of these boards must not be pointed at the other unexamined.

⚠ macOS in the transfer path silently DELETES kernel modules

Staging a built kernel through a Mac (docker cp → APFS → rsync → board) loses every file whose name differs from another only by case. APFS is case-insensitive, so xt_DSCP.ko and xt_dscp.ko collapse to one. Measured: 4 modules lost per kernel — xt_DSCP, xt_HL, xt_RATEEST, xt_TCPMSS — on all four kernels installed on 2026-10-12, on both boards. Both files built correctly in the container; only the transfer destroyed them.

It is silent: make succeeds, the install succeeds, the board boots. It was caught only by diffing the installed module set against the reference kernel (2272 vs 2268 files).

Move kernel trees as a stream, never as files on a Mac filesystem:
ssh builder 'docker exec kbuild tar cf - -C /root/out .' | ssh board 'cat > /tmp/k.tar'
then extract on the board into a staging dir (never -C / — see below) and rsync into place. Verify afterwards: the installed .ko/.ko.xz count must equal the reference kernel's.

Note the two boards even differ here: .235's modules are uncompressed .ko, .236's are .ko.xz. Check before copying by name.

⚠⚠ Installing ANY kernel: two traps that are brick-adjacent

Both of these apply to every kernel install on either board, not just to #patch74.

1. update-initramfs silently re-points /boot/uInitrd. Armbian's hook re-aims /boot/uInitrd at whatever kernel you just installed — even though you did not ask it to and are not booting that kernel yet. The result is Image on one release and uInitrd on another: a mismatched pair, and an initramfs whose modules do not match the kernel that will load it. This happened on the first install of this work and was caught only by looking. It was latent in the existing procedure. Always re-assert all three symlinks (Image, uInitrd, dtb) after any install, and refuse to reboot while they disagree — which is what /boot/SELECT_KERNEL.sh and the installer used here both now do.

2. Never tar x -C / a payload containing ./lib/. On this usr-merged rootfs GNU tar replaces the /lib -> usr/lib symlink with a real directory, /lib/ld-linux-*.so vanishes, every exec dies, sshd still listens but every connection resets, and there is no remote recovery — it needs physical media access. That is how .235 was bricked on 2026-10-09. Use --keep-directory-symlink, or rsync a staged tree file-by-file (what this work did), or build a .deb; and assert test -L /lib before and after.

`CONFIG_DEBUG_LIST=y` is TEMPORARY — flip it off when the A/B is done

It is a diagnostic aid, not a permanent config: it adds integrity checks to every list operation kernel-wide. Leaving it on a benchmarking board taxes every future measurement, and the shift would later be attributed to the wrong cause. Both variants are already installed, so removal is a symlink flip and a reboot, not another build:

sudo /boot/SELECT_KERNEL.sh 6.1.115-vendor-rk35xx-fix && sudo reboot

Re-baseline after the flip. A number taken on -fixdbg is not comparable to one taken on -fix.

Measured cost of `CONFIG_DEBUG_LIST=y`: NOT DETECTABLE ON THIS WORKLOAD

Single-variable A/B on .235, both kernels unfixed, differing only in DEBUG_LIST; governors verified performance @ 2400 on both; ork_bench Qwen3-1.7B-Q8_0, P=256/G=64/ubatch=256:

kernel	`DEBUG_LIST`	prefill tok/s
`6.1.115-vendor-rk35xx-npu`	n	196.99, 202.90
`6.1.115-vendor-rk35xx-predbg`	y	199.94, 204.23
`6.1.115-vendor-rk35xx-fix` (production, post-flip re-baseline)	n	200.21, 200.57, 204.52

.236, same fix, its own harness (benchrun.sh, qwen3-1.7b-f16, P=128/G=32/ubatch=128, 3 reps, governors verified performance, dmc 2112, cpu4/cpu6 2400 after the DTB correction above):

kernel	prefill tok/s	decode tok/s
`-fence` (pre-install)	181.70, 187.70, 196.63 (mean 188.7)	6.23, 6.22, 6.20 (mean 6.217)
`-fencefix` (post-install)	190.09, 195.96, 197.74 (mean 194.6)	6.27, 6.24, 6.25 (mean 6.253)

Read decode, not prefill, for this comparison. Prefill ramps across reps in both arms (181→196 and 190→197), so the prefill means are dominated by warm-up position, not by the kernel. Decode is flat to ±0.5% within each arm and moves +0.6% between them — i.e. the fix is performance-neutral on .236 too, matching .235. Expected: a one-word change on an error path a healthy run never takes.

The checks measured +1.1% faster — i.e. no detectable cost; the ranges overlap almost entirely. The third row is the post-flip production re-baseline and is the number to compare future work against. All three overlap, which also confirms the #patch74 fix itself is performance-neutral — expected, since it is a one-word change on an error path that a healthy run never takes.

⚠ Do not cite this as "DEBUG_LIST is free". Two caveats are part of the result:

This is an NPU-bound prefill workload. DEBUG_LIST taxes list operations, which are not on this hot path. A CPU- or kernel-heavy workload can differ, and nothing here measures that.

The DEBUG_LIST arm ran second, and arm position on this board is worth up to 1.126x on its own (see the A/A control in Experiment-Log — arm position bias). That is larger than the effect being measured, so the +1.1% is not evidence of a speedup either.

Read it as "not detectable on this workload", not as "proven zero".

Kernel Modifications

Kernel Modifications (RK3588 board, 10.3.0.236)

Where the source lives

Change 1 — NPU SRAM enabler (arch/arm64/boot/dts/rockchip/rk3588s.dtsi)

Change 2 — RKNPU domain-refcount leak on job timeout (drivers/rknpu/rknpu_job.c)

Change 3 — RKNPU records no IOMMU domain for dma-buf imports (drivers/rknpu/rknpu_gem.c)

Measured, 0.8B, ORK_DOMAINS_FORCE, same binary either side

Change 4 — signal the fence when a job is torn down without completing (drivers/rknpu/rknpu_job.c)

Change 5 — per-core progress watchdog (drivers/rknpu/rknpu_drv.c)

⚠ POWER SAFETY — read before touching this code

Change 6 — RKNPU IRQ handler touches MMIO while the NPU is powered down (drivers/rknpu/rknpu_job.c)

Why this only became visible now

Log capture on this board — what actually works

Change 7 — NULL IOMMU domain dereferenced in the GEM DMA paths (drivers/rknpu/rknpu_iommu.c)

Change 8 — rknpu_job_abort publishes a freed job pointer (drivers/rknpu/rknpu_job.c)

Change 9 — doorbell-stall diagnostics and per-core reset (DIAGNOSTIC, default-off)

What it measured — four hypotheses refuted with counters

⚠ RETRACTION — "the IOMMU re-attach is what makes recovery work"

Watchdog cadence — a 100x throughput trap

Complete current state

Change 10 — rknpu_gem_destroy_ioctl holds an unreferenced GEM pointer across an ~18 s wait (drivers/rknpu/rknpu_gem.c)

Correction to Change 8's risk assessment

Change 11 — rknpu_gem_create_ioctl dereferences an unreferenced GEM pointer (drivers/rknpu/rknpu_gem.c)

Change 12 — the progress watchdog is ON by default, and wd_kick is deleted

12a. rknpu_wd_arm() never fired for blocking submits (the bug that made the watchdog useless)

12b. Detection is now default-ON (wd_period_us=10000, wd_samples=12, ~120 ms)

12c. wd_kick deleted

Change 13 — fast-abort a stalled BLOCKING submit (wd_abort_ms, OPT-IN, partial win)

Measured on the 1.5B-Q8 repro (ork_bench ... 256 32), two boots

Verdict: keep OPT-IN. It is a partial win, and it is detection-limited.

Corrections to earlier characterizations (from a second agent's independent run)

Change 14 — attribute a stall run to the JOB, not to a counter value

Measured, 1.5B-Q8 repro, counted WITHOUT circularity

Change 15 — the domain wedge: a reset/switch race, plus a refcount clamp

15a. REFUTED: refcount underflow

15b. FIXED: rknpu_soft_reset raced domain switching

Measured across the two fixes, same 1.5B-Q8 repro

15c. NOT a bug: the remaining switch timeouts

Change 16 — ★ opt-in timing profiler, and the strongest correlation found so far

★ Result: stalls correlate with BACK-TO-BACK commits on the same core

What this does NOT yet establish

Side effect worth recording: the repro now COMPLETES

Change 17 — RKNPU_ACT_REINIT: clear all IOMMU domain state without rebooting

It must ABORT, not refuse — this was got wrong first

Measured

Caller contract

The fence kernel (6.1.115-vendor-rk35xx-fence, 2026-08-25)

The three kernels on the board — a ready-made bisect

Reverting

Rules going forward

Change 18 — light IOMMU domain switch (drop the default_domain lie and the force-reset)

The defect

The change

Measured

What it does NOT fix

Change 19 — reclaim-and-retry on a domain-switch timeout (a stall stops being fatal)

The problem

The change

Measured (deterministic reproducer, ORK_MM_TIMEOUT=3000 ./orkd_dom_api)

What it does NOT fix

Related negative result — the "post-swap danger zone" is not supported

Change 20 — soft reset must reprogram the LIVE domain (fixes the silently-dropped op)

The bug — a regression introduced by Change 18

The fix

Measured

Cross-reference: mainline accel/rocket

Change 21 — audit: every remaining iommu_get_domain_for_dev() caller (4 more stale-domain bugs)

These are real bugs, but they are NOT the multi-domain stall

Change 22 — imports must map into the LIVE domain (★ this is the multi-domain dispatch stall)

The bug

The fix

Measured — 1.5B Q8_0 multi-domain ork_bench (failing since the start of the investigation)

How it was isolated

Note on the tooling

Change 23 — the progress watchdog is a DEVELOPER tool: default off

Where the kernel work lives

Build status of #390

Still only on the SBC

The SRAM enabler — OPTIONAL, and not intended for mainstream

What the board is actually running — #59 (2026-09-01)

Change 1 — NPU SRAM enabler (`arch/arm64/boot/dts/rockchip/rk3588s.dtsi`)

Change 2 — RKNPU domain-refcount leak on job timeout (`drivers/rknpu/rknpu_job.c`)

Change 3 — RKNPU records no IOMMU domain for dma-buf imports (`drivers/rknpu/rknpu_gem.c`)

Measured, 0.8B, `ORK_DOMAINS_FORCE`, same binary either side

Change 4 — signal the fence when a job is torn down without completing (`drivers/rknpu/rknpu_job.c`)

Change 5 — per-core progress watchdog (`drivers/rknpu/rknpu_drv.c`)

Change 6 — RKNPU IRQ handler touches MMIO while the NPU is powered down (`drivers/rknpu/rknpu_job.c`)

Change 7 — NULL IOMMU domain dereferenced in the GEM DMA paths (`drivers/rknpu/rknpu_iommu.c`)

Change 8 — `rknpu_job_abort` publishes a freed job pointer (`drivers/rknpu/rknpu_job.c`)

Change 10 — `rknpu_gem_destroy_ioctl` holds an unreferenced GEM pointer across an ~18 s wait (`drivers/rknpu/rknpu_gem.c`)

Change 11 — `rknpu_gem_create_ioctl` dereferences an unreferenced GEM pointer (`drivers/rknpu/rknpu_gem.c`)

Change 12 — the progress watchdog is ON by default, and `wd_kick` is deleted

12a. `rknpu_wd_arm()` never fired for blocking submits (the bug that made the watchdog useless)

12b. Detection is now default-ON (`wd_period_us=10000`, `wd_samples=12`, ~120 ms)

12c. `wd_kick` deleted

Change 13 — fast-abort a stalled BLOCKING submit (`wd_abort_ms`, OPT-IN, partial win)

Measured on the 1.5B-Q8 repro (`ork_bench ... 256 32`), two boots

15b. FIXED: `rknpu_soft_reset` raced domain switching

Change 17 — `RKNPU_ACT_REINIT`: clear all IOMMU domain state without rebooting

The fence kernel (`6.1.115-vendor-rk35xx-fence`, 2026-08-25)

Change 18 — light IOMMU domain switch (drop the `default_domain` lie and the force-reset)

Measured (deterministic reproducer, `ORK_MM_TIMEOUT=3000 ./orkd_dom_api`)

Cross-reference: mainline `accel/rocket`

Change 21 — audit: every remaining `iommu_get_domain_for_dev()` caller (4 more stale-domain bugs)

Measured — 1.5B Q8_0 multi-domain `ork_bench` (failing since the start of the investigation)

What the board is actually running — `#59` (2026-09-01)

Validation status of `#59`

#patch74 — the abort-path double `list_del` (the `.235` panic of 2026-10-11)

`d147723` — a second defect found in the same loop

⚠ `.236` was never immune

Kernels on `.235` after this work (2026-10-12)

★★ `.236`'s A76 2400 MHz ungating lives ONLY in its installed DTB — ANY rebuild loses it

⚠ `.236` and `.235` DO NOT share a boot contract

`CONFIG_DEBUG_LIST=y` is TEMPORARY — flip it off when the A/B is done

Measured cost of `CONFIG_DEBUG_LIST=y`: NOT DETECTABLE ON THIS WORKLOAD