Kernel Modifications

Kernel Modifications (RK3588 board, 10.3.0.236)

The board does NOT run a stock kernel. Two changes are ours. Both were undocumented until 2026-08-24, which made them a hidden variable in every board measurement taken before that date — including a multi-day IOVA-leak investigation that concluded "kernel-side, not ours" while running a patched driver.

uname -r                     6.1.115-vendor-rk35xx-sram
built by                     root@7235a93dce00   <- docker container "kbuild" on the .239 colima VM
source tree                  kbuild:/root/linux-rockchip   (changes are UNCOMMITTED working-tree edits)
revert                       /boot/RESTORE_STOCK_KERNEL.sh  (two symlink flips, idempotent)

Where the source lives

The build ran inside a stopped-but-intact docker container on the Colima VM at 10.3.0.239:

ssh 10.3.0.239 'export PATH=$PATH:/opt/homebrew/bin
  docker start kbuild
  docker exec kbuild sh -c "cd /root/linux-rockchip && git status --short && git diff"'

Both modifications are uncommitted working-tree edits — there is no commit to point at, which is why they were invisible. /root/kernel is a second, unmodified tree.

Change 1 — NPU SRAM enabler (arch/arm64/boot/dts/rockchip/rk3588s.dtsi)

Gives the NPU the on-chip SRAM that the video decoders normally own, and merges the two decoder regions into one 956 KB block (0xef000).

 	rknpu: npu@fdab0000 {
 		compatible = "rockchip,rk3588-rknpu";
+		rockchip,sram = <&rkvdec0_sram>;

 	rkvdec0: rkvdec-core@fdc38000 {
-		rockchip,sram = <&rkvdec0_sram>;
 	rkvdec1: rkvdec-core@fdc48000 {
-		rockchip,sram = <&rkvdec1_sram>;

 		rkvdec0_sram: rkvdec-sram@0 {
-			reg = <0x0 0x78000>;
-		};
-		rkvdec1_sram: rkvdec-sram@78000 {
-			reg = <0x78000 0x77000>;
+			reg = <0x0 0xef000>;
 		};

Paired with CONFIG_ROCKCHIP_RKNPU_SRAM=y — the only config line differing from stock (# CONFIG_ROCKCHIP_RKNPU_SRAM is not set).

Why: the NPU's on-chip SRAM reported 0 available; this hands it the decoders' 956 KB. Cost: rkvdec0 and rkvdec1 lose their SRAM — video decode is degraded/untested on this board. Verify live: /proc/device-tree/npu@fdab0000/rockchip,sram exists.

Change 2 — RKNPU domain-refcount leak on job timeout (drivers/rknpu/rknpu_job.c)

 static void rknpu_job_timeout_clean(struct rknpu_device *rknpu_dev, ...)
 			spin_unlock_irqrestore(&rknpu_dev->irq_lock, flags);
 
+			rknpu_iommu_domain_put(rknpu_dev); /* #patch: reap the domain ref this timed-out job
+			                                      leaked (job_done/job_abort put; timeout_clean
+			                                      forgot) -> fixes multi-domain switch wedge */
 			do {
 				schedule_work(&job->cleanup_work);

Why: rknpu_job_done and rknpu_job_abort release the IOMMU domain reference; rknpu_job_timeout_clean did not, so every timed-out job leaked one — and a held reference blocks the next domain switch, which is the multi-domain wedge.

⚠ THIS PATCH IS A SUSPECT IN THE CURRENT LEAK (Tier 17). It is an added put. If a path reaches timeout_clean after the reference was already released, this is a double-put — the refcount underflows, the domain is freed early, and a later unmap hits a stale domain, producing exactly the observed WARN_ON(unmapped != size) in __iommu_dma_unmap and the never-reclaimed IOVA. The symptom it was written to fix (switch iommu domain time out) is also the symptom still being observed, so it is either incomplete or overcorrecting.

Change 3 — RKNPU records no IOMMU domain for dma-buf imports (drivers/rknpu/rknpu_gem.c)

This is the multi-domain IOVA leak. Diff: 03-rknpu-import-domain-id.patch. Deployed 2026-08-24; backup of the pre-patch image is /boot/vmlinuz-6.1.115-vendor-rk35xx-sram.pre03-bak.

 	rknpu_obj->dma_addr = sg_dma_address(sgt->sgl);
+	/* #patch: the sg above was mapped against the CURRENTLY attached domain.
+	 * Record it, or destroy unmaps from domain 0 and strands this IOVA. */
+	rknpu_obj->iommu_domain_id = rknpu_dev->iommu_domain_id;

Three driver facts combine into the bug:

1. rknpu_gem_prime_import_sg_table() never sets iommu_domain_id, and rknpu_gem_init() uses kzalloc — so every imported object is filed under domain 0.
2. Its sg table was mapped by dma_buf_map_attachment() during PRIME_FD_TO_HANDLE, against whichever domain happened to be attached at that moment.
3. rknpu_gem_create_ioctl() does rknpu_gem_object_find(args->handle) and returns early on a hit. args->iommu_domain_id is therefore silently ignored for imports — userspace passes the domain and the kernel drops it. There is no uABI way to make it stick.

rknpu_gem_object_destroy() then faithfully switches to the recorded id (0) and unmaps from the wrong page table:

drm_release -> drm_gem_release -> idr_for_each -> rknpu_gem_object_destroy
  -> drm_prime_gem_destroy -> dma_buf_unmap_attachment -> __iommu_dma_unmap
WARNING: drivers/iommu/dma-iommu.c:691 __iommu_dma_unmap+0xb8/0x13c

The unmap silently does nothing and the IOVA in the real domain is stranded for the life of the boot. Compare the freshly-allocated path, rknpu_gem_object_create(): it switches to the domain first and then stamps iommu_domain_id. Correct. Only the import path has the gap.

Why it looked like a multi-domain-only bug: a single-domain process has the attached domain at 0, matching the recorded 0, so the unmap hits the right table. The handles leak identically — they just cost nothing.

Measured, 0.8B, ORK_DOMAINS_FORCE, same binary either side

	before (patch 01+02)	after (patch 03)
WARN_ON per multi-domain run	30, invariant across dtype / domain count / weight count	0
consecutive multi-domain runs before failure	2 (3rd: MEM_CREATE EFAULT at 176 MB of 3900)	5+, no degradation
int8, 250 imports under non-zero domains	30	0
PPL	36.3973 int4 / 17.5502 int8	identical

Userspace prediction check before patching: 26 imports performed under a non-zero domain vs 30 observed WARNs — "requested domain" is only a proxy for "domain attached at PRIME_FD time", which is what actually decides where the mapping lands.

Worth reporting upstream against both the vendor driver and drivers/accel/rocket.

Change 4 — signal the fence when a job is torn down without completing (drivers/rknpu/rknpu_job.c)

dma_fence_signal(job->fence) occurs at exactly ONE site: the RKNPU_JOB_DONE completion path. rknpu_job_abort() and rknpu_job_timeout_clean() tear a job down without signalling, and rknpu_job_free() only does dma_fence_put() — a reference drop, not a signal. So a stalled job never wakes its waiter, which defeats the point of waiting on a fence to detect a stall.

Fix: in rknpu_job_free(), if the fence is not yet signalled, dma_fence_set_error(-ETIMEDOUT) then dma_fence_signal() before the put. No-op on the success path (already signalled). Diff: 04-rknpu-fence-signal-on-abort.patch.

Note dma_fence is a generic Linux primitive (drivers/dma-buf), not an rknpu or NVDLA concept; the vendor driver implements the whole FENCE_IN/FENCE_OUT path and simply left it compiled out.

Change 5 — per-core progress watchdog (drivers/rknpu/rknpu_drv.c)

This is what root-caused the int4 doorbell "miss". rknpu_job_timeout_clean is not a watchdog: it runs only when ANOTHER submit arrives on that core, and it measures TOTAL JOB AGE, which is meaningless for a PC-chained job whose task list is extensible at runtime. Change 5 samples the PC completed-task counter (config->pc_task_status_offset & config->pc_task_number_mask) per core at 1 kHz while a job is outstanding, and reports "no task progress in N samples". Detector only — it logs, it does not reap.

Result: on every stall the counter reads 0x0 with the job >100 ms old on all three cores — the job is accepted (rc=0) and never dispatches. See NPU-Quirks for the full write-up and control.

Params under /sys/module/rknpu/parameters/: wd_period_us (0 disables), wd_samples; counters wd_ticks, wd_seen_job, wd_pm_skip, wd_stalls. Diff: 05-rknpu-progress-watchdog.patch.

⚠ POWER SAFETY — read before touching this code

Two earlier versions were wrong, and the first hard-hung the board (no ping, power cycle required):

1. v1 read NPU MMIO directly from the hrtimer callback. A register read on a powered-down block is an external abort. It hung on its first NPU workload.

2. v2 guarded with pm_runtime_get_if_in_use(rknpu_dev->dev). Still wrong on RK3588: this driver keeps its OWN refcount (power_refcount / power_lock) and rknpu_power_off() puts the three PER-CORE genpds (genpd_dev_npu0/1/2) directly, so a reference on the parent does not hold them up. It survived only because the deferred power-off keeps the NPU up during a busy run — luck, not correctness. RETRACTION (2026-08-26): the watchdog did NOT cause the board resets. They were blamed on it from a single A/B (n=1 each way), then the same workload with the watchdog ON survived. netconsole later caught the real fault — an unguarded MMIO read in the VENDOR rknpu_irq_handler (see Change 6). Only the v1 hard-hang below is genuinely attributable to this watchdog. It ships default-off anyway, because it is a diagnostic, not something a production run should carry.

3. v3 (shipped) samples in PROCESS context (hrtimer → queue_work) holding power_lock, which rknpu_power_off() must also take, so the genpds cannot drop mid-read. It deliberately does not call rknpu_power_get() — that powers the NPU ON just to read a counter and would pin it up forever; it checks power_refcount > 0 and skips otherwise. mutex_trylock so it can never delay a submit. Measured wd_pm_skip = 332 / ~35000 samples (0.9%), so coverage is essentially unaffected.

Change 6 — RKNPU IRQ handler touches MMIO while the NPU is powered down (drivers/rknpu/rknpu_job.c)

★ This is very likely the long-standing "hard wedge" that has cost this board power cycles and at least one physical SPI reflash.

rknpu_irq_handler() accesses registers unconditionally on BOTH paths — the no-job path does REG_WRITE(RKNPU_INT_CLEAR), the normal path does REG_READ(RKNPU_OFFSET_INT_STATUS) — with no power check. Power-off runs from a DEFERRED work item (rknpu_power_off_delay_work), so a late or spurious interrupt can land after the block has lost power. The register access then takes an external abort and the machine dies instantly: no console, no ping, power cycle required.

Captured via netconsole (nothing else on this board can retain it — see below):

pc : readl+0x4/0x20
lr : rknpu_irq_handler.isra.0+0x94/0x2f0
Call trace:
  readl+0x4/0x20
  rknpu_core0_irq_handler+0x20/0x30
  __handle_irq_event_percpu+0xbc/0x1d4
  ... el1_interrupt / cpuidle_enter / do_idle

CPU 0 was idle — i.e. the NPU had finished, powered down, and the IRQ arrived afterwards.

Corroboration that late IRQs are real on this platform: the neighbouring rk_iommu_irq() (drivers/iommu/rockchip-iommu.c:762) already guards this exact race with pm_runtime_get_if_in_use() + WARN_ON_ONCE, and that WARNING fires on this board under the same load. The IOMMU driver survives with a warning where rknpu died — because rknpu had no guard.

Fix: return IRQ_NONE without touching MMIO when power_refcount <= 0. power_refcount is an atomic, so unlike power_lock (a mutex) it is safe from hard-IRQ context. If the block is unpowered it cannot be asserting, so IRQ_NONE is correct and cannot cause a level-IRQ storm. Diff: 06-rknpu-irq-power-guard.patch.

Validated: 6 passes x 3 shapes x 2000 reps (36,000 int4 doorbell ops) with the Change-5 watchdog deliberately ENABLED — 0 panics, board healthy. The identical workload previously reset the board.

Why this only became visible now

It is not new. With the stock CONFIG_PANIC_ON_OOPS_VALUE=0, this fault had no clean path: the machine died silently and unrecoverably. Enabling panic_on_oops=1 + panic=10 (see the fence-kernel section) converts it into panic → backtrace → auto-reboot in 10 s, which is the ONLY reason the trace exists. The historical hard wedges recorded elsewhere in this wiki have the same fingerprint.

Log capture on this board — what actually works

• netconsole: works (this trace came from it). Persisted via /etc/systemd/system/netconsole.service (ordered After=network-online.target; modules-load.d is too early to bind eth0) + /usr/local/sbin/netconsole-up.sh. netpoll needs a destination MAC — the gateway's MAC is the one that reaches the capture host here.
• ⚠ console_loglevel defaults to 4, so only messages BELOW level 4 reach the console and hence netconsole. <4>/KERN_WARNING test messages are silently dropped — this cost an hour of wrongly concluding netconsole was broken. kernel.printk = 8 4 1 7 is persisted in /etc/sysctl.d/.
• pstore/ramoops: DOES NOT WORK on RK3588. The regions are reserved and re-registered every boot, but /sys/fs/pstore comes back empty even after a warm panic-reboot — the Rockchip restart re-initialises the DDR controller and wipes the region. Do not rely on it.
• /var/log is a 50 MB tmpfs (DietPi RAMlog, AUTO_SETUP_LOGGING_INDEX=-1) and journald retains no prior boots (journalctl -b -1 → "No journal boot entry found") despite Storage=persistent.
• A durable fallback runs on the NVMe: kmsglog.py → /home/michael/ork-outbox/kern.log, fsync per message, outside RAMlog.

Change 7 — NULL IOMMU domain dereferenced in the GEM DMA paths (drivers/rknpu/rknpu_iommu.c)

rknpu_iommu_dma_map_sg(), rknpu_iommu_dma_unmap_sg() and rknpu_iommu_dma_alloc_iova() each dereference iommu_get_domain_for_dev() (or the domain handed to them) with no NULL check, one line in:

struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
struct rknpu_iommu_dma_cookie *cookie = (void *)domain->iova_cookie;   /* boom */

And the device really can end up attached to no domain at all: rknpu_iommu_switch_domain() does iommu_detach_device(src) and then, if attaching dst fails, tries to re-attach src — if that also fails it returns with the device detached. This is reachable in normal operation, because a stalled job cannot be drained, so the switch hits its 6 s timeout (switch iommu domain time out) and the abort path runs.

Captured (netconsole):

Unable to handle kernel NULL pointer dereference at virtual address 0000000000000040
pc : rknpu_iommu_dma_map_sg+0x48    Comm: i4_widek_stall_
  rknpu_gem_object_create / rknpu_gem_create_ioctl / drm_ioctl

and, after guarding only map_sg, the identical shape at rknpu_iommu_dma_unmap_sg+0x3c on the next run.

Fix: a rknpu_iommu_cookie() helper that returns NULL with a ratelimited error instead of faulting (callers turn it into an ordinary failure), plus a switch failure path that attaches any still-valid domain rather than returning detached. Diff: 07-rknpu-null-iommu-domain-guards.patch — applies standalone to the base commit.

This is not theoretical: the guard fired 12 times in a single run. The detached state happens.

---

Change 8 — rknpu_job_abort publishes a freed job pointer (drivers/rknpu/rknpu_job.c)

if (job == subcore_data->job && !job->irq_entry[i]) {
        subcore_data->job = NULL;
        subcore_data->task_num -= rknpu_get_task_number(job, i);
}
...
rknpu_job_cleanup(job);          /* frees the job EITHER WAY */

The clear is gated on !job->irq_entry[i], but the free is not. Whenever irq_entry[i] is set, the job is freed while subcore_data->job still points at it. The IRQ handler, rknpu_job_next() and the progress watchdog all read that field, so the next reader touches freed slab.

Because the corruption and the crash are far apart, this never looked like an NPU bug. Observed panic sites, all downstream of the same root:

BUG: Bad page cache ... "still mapped when deleted"   (shmem_evict_inode)
__memcg_kmem_charge_page  <- copy_process <- fork, from dropbear
sg_free_table / __free_pages  <- rknpu_gem_object_destroy
cfs_rq_clock_pelt <- attach_entity_cfs_rq

Fix: clear subcore_data->job unconditionally; keep the task_num adjustment gated exactly as before so the accounting is unchanged. Diff: 08-rknpu-abort-published-freed-job.patch — applies standalone to the base commit.

Consequence worth knowing: with the dangling pointer gone, the Change-9 stall kick correctly finds NULL and declines — and measured recovery drops from "3/3" to 0/5. See the retraction below.

---

Change 9 — doorbell-stall diagnostics and per-core reset (DIAGNOSTIC, default-off)

A bundle built to root-cause the int4 NONBLOCK dispatch stall. Not intended to ship: RKNPU_ACT_RESET_CORE uABI (0x100, outside the vendor 0-25 range) + rknpu_soft_reset_core(); rknpu_job_kick_stalled() with wd_kick modes; KICKLAT timing; commit-register snapshot and full PC register-block dump; IOMMU attach/switch counters; MMU register dump at stall; a settle-only kick mode.

It also fixes a real vendor wart found on the way: rknpu_soft_reset() returned 0 (success) on a failed mutex_trylock without resetting. Harmless until something else took reset_lock — once the kick did, userspace ACT_RESET became a silent no-op and throughput fell ~10x (2.7 vs 34 commits/s).

What it measured — four hypotheses refuted with counters

hypothesis	instrument	result
deferred attach leaves a stale page table	rk_dbg_attach_deferred	refuted — 0 deferred vs 1017 programmed
the switch races an in-flight job	rknpu_dbg_switch_inflight	refuted — 0 in-flight vs 1012 idle
the MMU itself wedges	RK_MMU_STATUS at stall	refuted — all 4 banks 0x19 = paging, idle, replay-empty
recovery needs post-reset settle time	settle 2/10/40 ms	refuted — 0 recovered of 20

An idle MMU at stall time is the sharpest clue on record: the NPU core never issued a memory request, so the failure is upstream of the IOMMU entirely.

⚠ RETRACTION — "the IOMMU re-attach is what makes recovery work"

An earlier result here claimed the IOMMU detach/attach was the essential recovery ingredient (3/3 with it, 0/11 without). That is retracted. Once Change 8 stopped abort publishing a freed job, the kick recovers 0/5. The original measurement was almost certainly re-committing a job that had already been freed — consistent with its suspiciously identical ~4.47 ms completions (4466/4471/4473/4478 us). Do not build on it, and do not cite the 3/3-vs-0/11 split.

The kick also had a defect of its own: its iommu_attach_device() return was unchecked, so a failed re-attach left the device detached. One occurrence produced 153 switch timeouts — 933 s of a 955 s run. Now checked with a fallback, but not fully contained.

Leave wd_kick=0 and wd_period_us=0 (both defaults). Post-retraction the kick has no demonstrated benefit, and it can wedge the driver into 6-second-timeout purgatory.

Watchdog cadence — a 100x throughput trap

Detection latency is wd_samples * wd_period_us. wd_period_us=1000 (1 kHz) queues work every millisecond and contends on the power/reset locks: 2.8 commits/s vs 283-490/s warm. Use wd_period_us=10000 with wd_samples=12 — the same ~120 ms detection at a tenth the sampling cost.

Complete current state

The board is running kernel #12 = base commit + all of changes 1-9. The exact cumulative diff is 99-ALL-current-state.patch (794 insertions, 11 files) — apply it to 1cd878b3ac49 to reproduce the running kernel. Changes 7 and 8 are additionally provided standalone because they are real fixes worth carrying without the diagnostics.

---

Change 10 — rknpu_gem_destroy_ioctl holds an unreferenced GEM pointer across an ~18 s wait (drivers/rknpu/rknpu_gem.c)

rknpu_gem_object_find() (include/rknpu_gem.h) is the lookup-then-put anti-pattern:

obj = drm_gem_object_lookup(filp, handle);   /* takes a reference */
if (!obj) return NULL;
rknpu_gem_object_put(obj);                   /* ...and drops it immediately */
return to_rknpu_obj(obj);                    /* caller gets an UNREFERENCED pointer */

Its kerneldoc claimed "gem object reference count would be increased" — that comment was false, and it is how this went unnoticed. Change 10 corrects the comment as well as the caller.

An unreferenced pointer is survivable for a short ioctl. rknpu_gem_destroy_ioctl() is not short: it then blocks in rknpu_iommu_domain_get_and_switch() for up to 6 s per attempt, three attempts (~18 s), and dereferences rknpu_obj->iommu_domain_id inside the retry loop. If the owning process dies in that window — SIGTERM -> DRM file close -> every handle released -> objects freed — the pointer dangles.

The window is wide open in practice, not theoretical: a stalled NPU job is exactly what makes the domain switch time out, and a stuck run is exactly when the process gets killed.

Why this is the suspected source of the page-cache corruption. rknpu maps the GEM's shmem pages into userspace itself with vm_insert_page() (rknpu_gem.c:983,1122) and clears VM_PFNMAP (:1148), so they are ordinary mapped pages that raise page->_mapcount. But vma->vm_file remains the DRM node — there is no get_file(obj->filp) anywhere. So the shmem inode's i_mmap tree has no record of these VMAs and truncate/evict cannot unmap them; it can only report:

BUG: Bad page cache in process i4_widek_stall_   pfn:128a06
page dumped because: still mapped when deleted     (mapcount:1)
  filemap_unaccount_folio / shmem_undo_range / shmem_evict_inode

The GEM refcount is therefore the only thing preventing a premature free — which makes an unreferenced pointer held across a multi-second sleep the prime suspect.

Fix: take a real drm_gem_object_lookup() reference for the whole ioctl and put it on every exit path. The put is placed before rknpu_iommu_domain_put() so that, if ours is the last reference, the object's teardown still runs with the domain held — the same ordering the original had, where handle_destroy could free it at that exact point. Diff: 10-rknpu-gem-destroy-unreferenced-pointer.patch — applies standalone to the base commit.

Status: built and running as kernel #13; make test ALL TESTS PASSED. NOT yet proven to fix the corruption — that needs a run that reproduces the old crash (stall probe + SIGTERM mid-stall). The other two rknpu_gem_object_find() callers (rknpu_gem_create_ioctl, the mmap-offset path) have the same latent pattern but no blocking wait, so they were left alone deliberately.

Correction to Change 8's risk assessment

While landing this, Change 8 (clearing subcore_data->job unconditionally in rknpu_job_abort) was briefly suspected of stalling job dispatch. It does not, and the opposite is true. rknpu_job_next() advances only when the slot is empty:

if (subcore_data->job || list_empty(&subcore_data->todo_list)) { spin_unlock; return; }

A non-NULL entry blocks that core. So upstream's behaviour — leaving a pointer to a freed job published — would block the core permanently, and clearing it can only help progress.

---

The fence kernel (6.1.115-vendor-rk35xx-fence, 2026-08-25)

Built ON THE SBC (~/kbuild/linux-rockchip), armbian/linux-rockchip @ 1cd878b3ac49, changes 1+2+3+4+5, plus CONFIG_ROCKCHIP_RKNPU_FENCE=y (the only config delta besides LOCALVERSION).

CONFIG_LOCALVERSION="-vendor-rk35xx-fence" is deliberate: the running kernel was built with gcc 13.3.0, the board has gcc 14.2, and CONFIG_MODVERSIONS=y — a same-version rebuild could break module loading. A distinct version gives it its own /lib/modules tree and leaves the -sram kernel completely untouched.

Boot safety was checked first: CONFIG_NVME_CORE/BLK_DEV_NVME/EXT4_FS=y and STMMAC_ETH/DWMAC_ROCKCHIP=y are all BUILT-IN, so the board boots and keeps SSH even if no module loads.

Select / revert = one symlink flip (DietPi boots via boot.scr + the Image/dtb symlinks):

sudo ln -sfn vmlinuz-6.1.115-vendor-rk35xx-fence /boot/Image   # or ...-sram to revert
sudo ln -sfn dtb-6.1.115-vendor-rk35xx-fence     /boot/dtb
sudo reboot

Patch 03 will not apply with git apply or patch — its hunk header is a bare @@ with no line numbers. Apply it manually; the script used is ~/kbuild/p03.py on the board.

The three kernels on the board — a ready-made bisect

image	SRAM	job.c patch	date
vmlinuz-6.1.115-vendor-rk35xx	✗	✗	stock
vmlinuz-6.1.115-vendor-rk35xx-sram.prepatch-bak	✓	✗	Jul 8
vmlinuz-6.1.115-vendor-rk35xx-sram.pre03-bak	✓	✓	Aug 11
vmlinuz-6.1.115-vendor-rk35xx-sram (+change 3)	✓	✓	Aug 24
vmlinuz-6.1.115-vendor-rk35xx-fence (running, +changes 4,5,6, RKNPU_FENCE=y, lockup detectors)	✓	✓	Aug 26

cmp confirms the Aug 11 image differs from the Jul 8 backup, so .prepatch-bak is a genuine SRAM-without-the-patch kernel. Booting all three against the same 27B run isolates which change (if either) causes the leak — see Tier 17.

Reverting

sudo sh /boot/RESTORE_STOCK_KERNEL.sh repoints /boot/Image and /boot/dtb at the stock names. Nothing was overwritten; stock kernel, initrd, dtb dir and modules are untouched. Idempotent. If an -sram kernel fails to boot, the script documents a microSD rescue path (mount /dev/nvme0n1p1, chroot, run).

Rules going forward

1. Every kernel change gets an entry here, with its diff, before it is deployed. Both of these ran for weeks as invisible variables.
2. Commit them in the build tree. They are uncommitted working-tree edits in a stopped container — one docker rm from being lost permanently, with no diff recoverable anywhere else.
3. A board measurement is only comparable to another on the same kernel. Record uname -r alongside any result that matters.