Skip to content

README

Jump to bottom
Michael Fischer edited this page Aug 26, 2026 · 6 revisions

RK3588 board kernel patches (10.3.0.236)

Working-tree diffs recovered 2026-08-24 from kbuild:/root/linux-rockchip — a stopped docker container on the .239 Colima VM, where they existed as uncommitted edits and nowhere else. Exported here so a docker rm cannot destroy them.

  • 00-BASE-COMMIT.txt — the armbian/linux-rockchip commit they apply on top of
  • 01-rknpu-sram-enable.patch — give the NPU the video decoders' 956 KB SRAM (pairs with CONFIG_ROCKCHIP_RKNPU_SRAM=y, the only config line differing from stock)
  • 02-rknpu-timeout-domain-put.patch — add the IOMMU domain put that rknpu_job_timeout_clean omits. Suspect in the Tier 17 leak: it is an added put, so a path that already released the reference makes it a double-put -> refcount underflow -> domain freed early -> WARN_ON(unmapped != size).

Apply:

cd /root/linux-rockchip && git checkout 1cd878b3ac49
git apply 01-rknpu-sram-enable.patch 02-rknpu-timeout-domain-put.patch

Full context, rationale and revert procedure: Kernel-Modifications.

2026-08-25 additions (fence + progress watchdog)

Built and running on 10.3.0.236 as 6.1.115-vendor-rk35xx-fence (installed ALONGSIDE the -sram kernel: its own /lib/modules tree, selected by the /boot/Image + /boot/dtb symlinks; revert = point them back at *-vendor-rk35xx-sram). Config delta vs the -sram kernel: CONFIG_ROCKCHIP_RKNPU_FENCE=y (the fence path is fully implemented upstream but was compiled out) and CONFIG_LOCALVERSION="-vendor-rk35xx-fence" — a distinct version deliberately, because the running kernel was built with gcc 13.3.0, the board has 14.2, and CONFIG_MODVERSIONS=y.

  • 04-rknpu-fence-signal-on-abort.patch — a job torn down without completing (rknpu_job_timeout_clean / rknpu_job_abort) never reaches the RKNPU_JOB_DONE path, so its fence is never signalled and a waiter blocks for its own full timeout. Signal with -ETIMEDOUT in rknpu_job_free so the waiter wakes and can tell a stall from a completion. Also arms the watchdog on submit.
  • 05-rknpu-progress-watchdog.patch — 1 kHz per-core sampling of the PC completed-task counter, reporting "no task progress in N samples". This is what root-caused the doorbell miss: on every stall the counter reads 0x0 with the job >100 ms old, i.e. the job is accepted and NEVER DISPATCHES. Params: wd_period_us (0 disables), wd_samples; counters wd_ticks/wd_seen_job/wd_pm_skip/wd_stalls under /sys/module/rknpu/parameters/.

POWER-SAFETY WARNING — read before touching the watchdog. Two earlier versions were wrong: reading NPU MMIO from the hrtimer hard-hung the board (external abort on a powered-down block, needs a power cycle), and pm_runtime_get_if_in_use() on the parent device is NOT sufficient because this driver keeps its own power_refcount/power_lock and puts the three per-core genpds (genpd_dev_npu0/1/2) directly. The shipped version samples in PROCESS context holding power_lock (which rknpu_power_off() must also take) and never calls rknpu_power_get() (that would power the NPU on just to read a counter).

Apply on top of 01+02+03. Patch 03 will not apply with git apply/patch — its hunk header is a bare @@ with no line numbers; apply it manually (see ~/kbuild/p03.py on the board).

2026-08-26 — Change 6 (the big one)

  • 06-rknpu-irq-power-guard.patch — rknpu_irq_handler touches MMIO with no power check, so a late interrupt after the deferred power-off faults on readl and kills the board instantly (no console, no ping, power cycle). Very likely the long-standing "hard wedge" that has cost this board power cycles and at least one physical SPI reflash. Captured by netconsole; validated over 36,000 int4 doorbell ops with the Change-5 watchdog enabled, 0 panics.

Retraction: the board resets seen on 2026-08-25 were blamed on Change 5 (the progress watchdog) from a single A/B. That was wrong — the same workload with the watchdog ON later survived, and the real fault is Change 6's vendor bug. Only the Change-5 v1 hard-hang is attributable to the watchdog. It ships default-off.

2026-08-26 additions — changes 7-9 (two real fixes + the diagnostic bundle)

Running on 10.3.0.236 as kernel #12 of 6.1.115-vendor-rk35xx-fence.

  • 07-rknpu-null-iommu-domain-guards.patch — FIX. rknpu_iommu_dma_{map,unmap}_sg() and _alloc_iova() dereference the IOMMU domain with no NULL check, and rknpu_iommu_switch_domain() can leave the device attached to no domain (detach succeeds, both attaches fail). Reached whenever a stalled job makes the switch time out. Guard fired 12 times in one run — the detached state is real.
  • 08-rknpu-abort-published-freed-job.patch — FIX. rknpu_job_abort() clears subcore_data->job only when !job->irq_entry[i] but frees the job unconditionally, publishing a dangling pointer that the IRQ handler, rknpu_job_next() and the watchdog all read. Causes slab corruption that panics later somewhere unrelated (__memcg_kmem_charge_page via fork, sg_free_table, cfs_rq_clock_pelt).
  • 99-ALL-current-state.patch — the complete cumulative diff (changes 1-9, 794 insertions, 11 files) against base 1cd878b3ac49. This is the authoritative record of what the running kernel is; the numbered patches above are extracted from it for cherry-picking.

Both fixes were verified to git apply --check cleanly against a pristine worktree at the base commit, and to reverse-apply against the board tree (i.e. they match it exactly).

cd /root/linux-rockchip && git checkout 1cd878b3ac49
git apply 99-ALL-current-state.patch          # everything, or:
git apply 07-rknpu-null-iommu-domain-guards.patch 08-rknpu-abort-published-freed-job.patch   # just the fixes

Do not enable the Change-9 kick (wd_kick, wd_period_us — both default 0). Its headline result was retracted, and an unchecked re-attach in it once produced 153 domain-switch timeouts (933 s of a 955 s run).

2026-08-26 — change 10 (GEM destroy holds an unreferenced pointer)

  • 10-rknpu-gem-destroy-unreferenced-pointer.patch — FIX. rknpu_gem_object_find() returns an UNREFERENCED pointer (lookup-then-put; its kerneldoc claiming otherwise was false and is corrected by this patch), and rknpu_gem_destroy_ioctl() held it across an ~18 s domain-switch wait while dereferencing it. A process death in that window frees the object under the ioctl. Prime suspect for the "still mapped when deleted" page-cache corruption, because rknpu vm_insert_page()s GEM shmem pages with VM_PFNMAP cleared while vma->vm_file stays the DRM node — so the GEM refcount is the only thing preventing a premature free. Verified to apply cleanly to a pristine worktree at the base commit.
  • 99-ALL-current-state.patch refreshed: now changes 1-10, the exact content of kernel #13 (make test ALL TESTS PASSED on it).

2026-08-26 — change 11 (GEM create holds an unreferenced pointer)

  • 11-rknpu-gem-create-unreferenced-pointer.patch — HARDENING. The second live rknpu_gem_object_find() caller: rknpu_gem_create_ioctl() reads the object four times on an unreferenced pointer. Short window (nothing sleeps), so unlike change 10 this is not a suspected crash source — but it is real. Net refcount across the ioctl is unchanged; only the in-ioctl window closes. The third caller (rknpu_gem_dumb_map_offset) is #if-ed out below 4.19 and is NOT compiled here, so it was left alone; the live path is DRM core's drm_gem_dumb_map_offset, which refcounts correctly.
  • 99-ALL-current-state.patch refreshed: changes 1-11, the exact content of kernel #14 (make test ALL TESTS PASSED on it).

Clone this wiki locally