.236's A76 2400 MHz ungating lives only in its installed DTB, so any rebuild loses it
GPTQ is free and worth 12 percent; SmoothQuant costs 7 percent speed and 3.8 percent PPL here
Scatter beats contiguous 1.92x once resets are free; the cause is K=6144 weights at H=3
Index the PR #390 result, and flag it on Kernel-Modifications
The 'Build status of #390' section said the series had never been booted. It has
now, on a second board built as armbian plus exactly those ten commits, and it
is not sufficient: orkd_dom_api fails 3/3 until Change 19 (#patchB5) is added.
Flags the reason it was never caught: the runtime testing that section describes
was done on the armbian tree carrying the full board state, which silently
included B5.
Kernel-Modifications: #patch72 landed as PR #8; record that its race window is unobserved
Kernel-Modifications: #patch73, the async power-reference bug that wedged a core
Kernel-Modifications: record the A76 2400 MHz OPP unlock
Board modification only -- Rockchip's per-bin opp-supported-hw gates are deliberate and not
upstreamable. Records the 8 edits, why it is not an overclock (all three top OPPs share 1.00 V and the
rail is already there), the validation (make test + 120s all-core load holding 2400 at 49C), the revert
path, and that it MASKS but does not fix the cpu_capacity==max bug in ork-driver#6.
Kernel-Modifications: record what board kernel #59 actually contains
The board builds from its own tree, not from a branch, so a kernel carries whatever any agent left in
~/kbuild — #59 includes patch69/70, another agent's GEM-destroy reclaim work, which is in neither PR.
Records the delta, the vm_kbuild.sh trap, and that #58 -> #59 is not a clean single-variable step.
wiki: mark the SRAM enabler as optional and not for mainstream
It is a resource-allocation policy choice, not a defect fix -- it takes on-chip SRAM from the video
decoders -- and its benefit is unproven for our workload (ORK_SRAM_REGCMD measured no effect).
Also records a packaging gap: as a patched rk3588s.dtsi it is compiled into the kernel image, so it
is not actually optional for anyone running that kernel. A device tree overlay is the correct form
for an opt-in feature.
wiki: point the kernel docs at PR #390 (supersedes #388); correct the 'no git history' claim
Kernel-Modifications: record where the kernel work lives, and add a full working-tree snapshot
The multi-domain redesign has NO git history — it is 13 uncommitted files in a shallow detached-HEAD
clone on one SBC. Attachment 18 captures the complete 13-file state (kernel #55) as one authoritative
snapshot, closing the gap where rk3588s.dtsi (the NPU SRAM enabler) had no patch attachment at all.
Also records why the redesign is not yet in the oRKLLM fork: it targets armbian's tree, and
rockchip-iommu.c differs by 61 lines and rk3588s.dtsi by 1442, so it needs a real port plus a
vendor-tree build rather than a cherry-pick.
Kernel change 23: progress watchdog default-off (developer opt-in)
Kernel change 22: imports must map into the live domain — the multi-domain dispatch stall, fixed
Kernel change 21: audit remaining iommu_get_domain_for_dev callers (4 stale-domain bugs; not the stall)
Kernel change 20: soft reset must reprogram the live domain (fixes silently-dropped op)
Kernel change 19: reclaim-and-retry on domain-switch timeout; ORK_DOM_WASH negative result
Kernel change 18: light IOMMU domain switch (no default_domain overwrite, no force-reset)
kernel: change 17 — RKNPU_ACT_REINIT, clear all IOMMU domain state without rebooting
rknpu_job_abort releases the domain reference unconditionally, so with several cores
busy it drives iommu_domain_refcount to zero while others still execute, and
get_and_switch reads zero as safe-to-switch. That accumulates until every switch and
then every allocation fails. Filed upstream as rockchip-linux/kernel#387; until it is
fixed a reboot was the only way to clear it.
RKNPU_ACT_REINIT (0x101) aborts in-flight jobs, rebuilds every domain and zeroes the
refcount and diagnostic counters. Driven by tools/re/npu_reinit.c in ork-driver.
Records that the first cut got it wrong: it returned -EBUSY whenever a core had a
job, which made it refuse in exactly the situation it existed for (idle -> ok,
degraded -> -EBUSY). It now reaps first and only returns -EBUSY if a job survives.
Measured end to end: accumulation 29->57->96, REINIT ok on the degraded device,
counters zeroed, and the next run completes at 7.18 tok/s with no reboot — then
accumulates again from zero, confirming real state was reset.
States the caller contract (IOVAs change, so buffers and regcmds are invalid after)
and that this is a workaround, not the fix.
kernel: change 16 — opt-in timing profiler finds the first positive correlation: stalls follow back-to-back commits
prof_mask (default 0) records two quantities per job from timestamps the driver
already takes: gap since the previous commit on that core, and submit->commit queue
delay, accumulated separately for healthy and stalled jobs.
Result on the 1.5B-Q8 repro at 100% detection coverage (86 stalls / 86 independent
timed-out waits): healthy n=1069 mean gap 207,356 us; stalled n=86 mean gap 492 us.
Every stalled job was committed 0.24-0.9 ms after the previous commit on that core,
always in core-0/1/2 triplets. Queue delay ~0 in both populations, so it is not
queueing. This is the first positive correlation after every software stage was
individually exonerated.
Explicitly NOT established: the healthy mean is inflated by idle periods, so a short
gap is shown necessary but not sufficient — that needs the healthy distribution, and
prof_mask bit 1 (log healthy commits) is documented but not yet implemented. And it
is correlation; the causal test is to enforce minimum inter-commit spacing, which
must use udelay since rknpu_job_next can run from the completion IRQ.
Side effect: with changes 12-16 the repro now completes (prefill 7.27 tok/s, decode
11.49 tok/s, rc=0) instead of timing out, though still ~30x below baseline.
kernel: change 15 — domain wedge is a reset/switch race; refcount-underflow theory refuted
Two wedges were conflated. The refcount-underflow theory (our change 02 added a put,
a documented double-put suspect) was instrumented with a clamp plus an underflow
counter and REFUTED: dbg_domain_underflow stayed 0 across a full failing run. Clamp
kept as cheap insurance and a live tripwire.
The real one: rknpu_soft_reset holds reset_lock and does iommu_detach/attach, while
get_and_switch holds domain_lock — different locks, no mutual exclusion. A switch
reading iommu_get_domain_for_dev during the detached window bails with 'mismatch
domain'. Always latent; the change-13 fast-abort raised the reset rate ~30x and made
it dominant at 1600 lines per 300 s run. Fixed by taking domain_lock around the
reset's detach/attach (reset_lock -> domain_lock, safe ordering).
Measured: mismatch 1600 -> 0, gem_object_create errors many -> 0, timed-out waits
68 -> 9, coverage 64% -> 89%.
The remaining switch timeouts (1 -> 83) are downstream, not a defect: an in-flight
job holds a domain reference, so while one is stalled other threads' switches block.
The mismatch had been hiding them by failing fast.
Also records that counts of timed-out waits are bounded by budget/duration and are
therefore NOT comparable across wd_abort_ms settings.
kernel: change 14 — attribute stall runs per job; withdraw the per-core correction
wd_flat only reset when the PC counter changed, but consecutive jobs each start
their counter at 0, so across a job boundary wd_flat kept climbing. Both the report
and the fast-abort fire on equality, so only the FIRST stall in a run was ever seen.
Fixed by resetting when subcore_data->job identity changes.
Coverage, measured against rknpu_job_wait's own independent 'failed to wait job'
lines (read from the persistent disk log — the dmesg ring overflows on a bad run and
silently truncates the denominator, which invalidated one measurement): 50% -> 64%.
Fast-abort remains sharply bimodal, 2.0 s vs 61 s.
This also WITHDRAWS the earlier correction that stalls do not always name all three
cores. That was an artifact of this very bug — core 0's report had been absorbed by
an un-reset run. Post-fix the same workload gives core 0 x3, core 1 x3, core 2 x3,
exactly symmetric, vindicating the original observation.
Still open: ~36% of timed-out waits go undetected and the run does not complete. The
queued-jobs theory was tested and refuted (queued-aborts=0; the todo_list was empty
at every stall).
kernel: change 13 — fast-abort for stalled blocking submits (opt-in, partial win, detection-limited)
Lets the watchdog wake a stalled blocking waiter early instead of letting it burn
args->timeout x3 (61 s measured). A new internal RKNPU_JOB_STALLED flag routes into
the existing ret<=0 path, so the outcome is the same -ETIMEDOUT, just sooner. The
abort bound is deliberately separate from and much larger than the ~120 ms logging
threshold, because the PC counter advances per completed task and a long single task
is indistinguishable from a stall by that signal alone.
Measured on the 1.5B-Q8 repro across two boots: submit failures 14 -> 2,
reproducibly. But the run still does not complete and the worst-case wait is
unchanged, because only 2 of 14 failures are ever detected. Stays OPT-IN.
A regression was found and fixed en route: a fast-aborted job reaches
rknpu_job_abort with -ETIMEDOUT, which runs a DEVICE-WIDE soft reset — firing ~30x
more often it disrupted healthy jobs on other cores and tripled the worst wait to
184 s. Now resets only the cores the job owns, using the per-core reset kept from
change 9. Worst wait went back to 61 s.
Also records two corrections from a second agent's run: stalls do NOT always name
all three cores (an episode named cores 1 and 2 with staggered job ages), and
wd_stalls is a lower bound rather than a miss count.
kernel: change 12 — watchdog arms on all submits, detection default-ON, wd_kick deleted
12a: rknpu_wd_arm() sat inside the NONBLOCK branch of rknpu_submit, so blocking
submits were never sampled. ork_dyn_colsplit issues blocking submits by default
despite its ork_dyn_ prefix, which is why a second agent's 1.5B-Q8 repro stalled 8
times with wd_ticks=0 — the instrumentation could not see the path that stalls,
while the stalled submit sat in rknpu_job_wait for 60.6 s. Now armed on every
submit; verified wd_ticks 0 -> 66 on a blocking workload.
12b: detection is default-ON at 10 ms x 12 (~120 ms). Both reasons for opt-in
expired: the board-reset attribution was retracted (change 6), and the throughput
objection was a cadence problem. Measured cost ~1.5% (OFF 133/133/133 vs ON
131/133/129 commits/s, interleaved, warmup discarded) — small but real, not free.
It earned it immediately: make test on #16 passed while reporting wd_stalls=2.
12c: wd_kick deleted (param, rknpu_job_kick_stalled, wd_settle_us, KICKLAT, the MMU
dump it alone called). Its headline 3/3 result was retracted as a use-after-free
artifact; post-fix it recovers 0/5, and 0 across every mode over ~two dozen stalls,
while having caused 153 switch timeouts via an unchecked re-attach. The intended
20-stall trial was never achieved — the workload wedges the driver first — so this
rests on the totality, which the page states plainly.
rknpu_soft_reset_core and RKNPU_ACT_RESET_CORE are KEPT.
kernel: change 11 — rknpu_gem_create_ioctl dereferences an unreferenced GEM pointer
The second live caller of the lookup-then-put rknpu_gem_object_find(). It reads the
object four times with only the handle table keeping it alive, so a concurrent close
frees it underneath. Hardening rather than a suspected crash source: unlike change
10, nothing sleeps between the lookup and the last use, so the window is short — but
it is real, and the vendor had already left a commented-out object_get() in the
middle of it.
Both branches now leave exactly one reference owned by the ioctl, dropped before
return. Net refcount across the ioctl is unchanged.
The third caller, rknpu_gem_dumb_map_offset, is the worst on paper (it calls
drm_gem_create_mmap_offset, which can sleep, on an unreferenced object) but sits
inside #if KERNEL_VERSION(4,19,0) > LINUX_VERSION_CODE and is not compiled on
6.1.115 — confirmed against the built object file, not just the #if. Left untouched
deliberately; the live path is DRM core's drm_gem_dumb_map_offset.
Built and running as kernel #14, make test ALL TESTS PASSED.
kernel: change 10 — rknpu_gem_destroy_ioctl held an unreferenced GEM pointer across an ~18 s wait
rknpu_gem_object_find() is lookup-then-put and hands back an unreferenced pointer,
kept alive only by the handle table; its kerneldoc claimed the refcount would be
increased, which is false and is corrected here. rknpu_gem_destroy_ioctl then held
that pointer across rknpu_iommu_domain_get_and_switch's 3 x 6 s timeout, and
dereferenced it inside the retry loop. A process death in that window (SIGTERM ->
DRM file close -> handles released -> objects freed) leaves it dangling.
Not theoretical: a stalled NPU job is what makes the switch time out, and a stuck
run is when the process gets killed — the exact combination that produced the
corruption this session.
Prime suspect for 'BUG: Bad page cache ... still mapped when deleted', because
rknpu vm_insert_page()s the GEM's shmem pages and clears VM_PFNMAP while
vma->vm_file stays the DRM node, so the shmem inode's i_mmap tree cannot find those
VMAs to unmap at evict — the GEM refcount is the only protection.
Built and running as kernel #13, make test ALL TESTS PASSED. Not yet PROVEN to fix
the corruption; that needs a run reproducing the old crash.
Also corrects the record on change 8: it was briefly suspected of stalling dispatch,
but rknpu_job_next only advances when subcore_data->job is NULL, so a stale non-NULL
entry blocks the core and clearing it can only help.
kernel: export changes 7-9 — two real fixes (NULL iommu domain, abort's freed job pointer) + the diagnostic bundle
Changes 7 and 8 are genuine vendor bugs found while chasing the int4 doorbell
dispatch stall, and both are board-killers:
- rknpu_iommu_dma_{map,unmap}_sg / _alloc_iova dereference the IOMMU domain with
no NULL check, and rknpu_iommu_switch_domain can leave the device attached to no
domain at all. Reached whenever a stalled job makes the switch time out. The
guard fired 12 times in a single run.
- rknpu_job_abort clears subcore_data->job only when !job->irq_entry[i] but frees
the job unconditionally, publishing a dangling pointer that the IRQ handler,
rknpu_job_next and the watchdog all read. The resulting slab corruption panics
later somewhere unrelated, which is why it never looked like an NPU bug.
Both are exported standalone (verified to apply cleanly to a pristine worktree at
base 1cd878b3ac49, and to reverse-apply against the board tree), alongside
99-ALL-current-state.patch, the complete cumulative diff for kernel #12.
Change 9 is the diagnostic scaffolding and is documented as default-off. It carries
a RETRACTION: the earlier "IOMMU re-attach recovers 3/3 vs 0/11" result does not
survive Change 8 — the kick now recovers 0/5, and the original was almost certainly
re-committing an already-freed job. It also records four hypotheses refuted with
counters, and the 100x throughput trap in the watchdog sampling cadence.
kernel: Change 6 — rknpu IRQ handler faults on MMIO while powered down (the hard-wedge bug); retract the Change-5 attribution
wiki: root-cause the int4 doorbell miss (never dispatches); kernel changes 4+5; correct the mid-write framing
Tier 17 RESOLVED: rknpu files dma-buf imports under domain 0 (kernel patch 03)
Root-caused the multi-domain IOVA leak to the vendor driver's import path and
fixed it: 30 WARN_ON(unmapped != size) per multi-domain run -> 0, and the board
no longer dies on the third consecutive run. Documents the patch, the measured
before/after, and marks the two superseded hypotheses (switch-timeout mechanism,
per-domain scratch identification) as wrong.
Kernel-Modifications: record the two changes the board actually runs
The board runs 6.1.115-vendor-rk35xx-sram, not stock, and both changes were undocumented:
(1) rk3588s.dtsi hands the NPU the video decoders' 956 KB SRAM (+ CONFIG_ROCKCHIP_RKNPU_SRAM=y,
the only config line differing from stock), and (2) rknpu_job.c adds a domain-refcount put in
rknpu_job_timeout_clean to fix a multi-domain switch wedge.
Both are UNCOMMITTED working-tree edits inside a stopped docker container (kbuild) on the
.239 VM -- one docker rm from unrecoverable, which is why they were invisible.
Flags change 2 as a suspect in the Tier 17 leak: it is an added put, so a path that already
released the ref makes it a double-put -> refcount underflow -> domain freed early -> the
observed WARN_ON(unmapped != size). Records the three kernels on /boot as a ready-made
bisect (stock / SRAM-only / SRAM+patch).