Carry the reset retraction into the Experiment Log entry too
9314d45
Retract: the mode reset IS load-bearing; my clean run did not exercise the hazard
fc06bc6, the commit that ADDED ORK_XSPEC_NORESET, had already run this on the
real 27B mixed-tier workload: PPL 10.4421 -> 3,442,988.92. And 42cc655 measured
the complement, widening ORK_INT_DT across 4->3, at PPL 5,029,965.
My arm-T run came out byte-identical because the 1.7B config -- K=2048, single
domain, DT_I8_ROT -- is not the hazard configuration. fc06bc6 says exactly this
about its own positive control, and added that control to stop a coherent
small-config run being misread as a result. I misread it that way anyway.
The 6.17x stands only as a CEILING: what mixed-tier prefill would cost if the
transition were avoided. The lever is transition count, not the reset --
promoting contiguous whole layers crosses the boundary twice per forward instead
of ~110 times, with every reset intact. The XP_I4CHAIN edit is withdrawn.
6e583c2
The mixed-pack mode reset is not load-bearing: 6.17x for byte-identical output
ORK_XSPEC_NORESET=1 suppresses only the ACT_RESET, leaving the warm/size clears
and last_dt intact. On the mixed arm-T pack that is 9.02 -> 55.64 tok/s prefill
with PPL 26.3637 and mean NLL 3.271989 unchanged to every digit, every
intermediate window matching, and the generated text identical.
The run-time delta (23223 ms) matches the 220 x 105 ms predicted from the price
already measured in mode.c:148 to within 0.5%, which closes the attribution.
XP_I4CHAIN (profile 8) is { KWP_NONE, RC_ALWAYS } -- an unconditional reset with
no keep-warm predicate, which is why ORK_MIXED_NOTHRASH could never affect it.
That is the one-row candidate. Changing it still owes the mandatory XSPEC
validation procedure; this probe justifies that work rather than replacing it.
cc88210
The 4-bit ladder beats i4a8 on quality; the 18x is mode resets, not the tier
Four packs on one model with disjoint GPTQ calibration. Arm T (SmoothQuant +
GPTQ + attn_v/ffn_down promotion) reaches PPL 25.78-26.36 against i4a8's 34.30,
so native 4-bit built with the landed ladder is better, not merely acceptable.
The 18x prefill cost was first attributed to the DT_I8_ROT tier from an average.
That was wrong. Promoting to plain DT_I8 instead changed nothing (9.10 vs 9.08),
and ORK_DEBUG_RESET named the caller: the mixed pack takes 221 ACT_RESETs
(110 each way across I8_CHAIN <-> I4_CHAIN) where both homogeneous packs take 1.
At the 105 ms/reset already measured here, that is 23.1 s of a 25.0 s run.
Also corrects the stale "~54x" native-prefill claim (real 3.62x) and its stated
cause, records that native buys nothing at decode, that GPTQ already skips the
int8 tier, that ORK_GPTQ_CLIP_MIN=0.531 would regress the live default, and that
stub vs full GGUF silently changes which engine runs decode.
35be357
Record the mmap weight-ownership bug and the dead ggml-ork correctness gate
The device iface aliased get_host_buffer_type to get_buffer_type, so the
loader's "avoid a host buffer under mmap" rule disowned every weight: 311
tensors to CPU before, 115 after. Compute was never stranded, ownership was.
Also records that tests/test-ork-cache.cpp is red on pristine diverged-history
and cannot distinguish a backend failure from a wrong answer, and what to use
as a gate instead.
fa39159
Correct the mechanism: faster DRAM BREAKS it, so this is a race, not starvation
2819ca6
The int4 envelope is a DRAM-clock term, not a die property; add the geometry-ladder fix
5e83457
Root-cause the .235 int4 failures (N-tile width) and the panic (stranded jobs vs power-off)
6356a94
The .235 board-tree+SRAM kernel passes everything except the two int4 tests
7876f0e
Experiment Log: 2026-10-09 (V) the colsplit's fixed per-call cost was cache maintenance over dead bytes
52313c1
Exp-2026-10-09 fp16 K-Slice: §6's open item is resolved — point at the follow-up
268e4e1
Exp-2026-10-09: the colsplit's fixed per-call cost was whole-buffer cache maintenance
f2ba7ec
Experiment Log: 2026-10-09 (IV) the domain-reference leak, root-caused
b7370b9
Cross-link the leak root-cause; .235 moves to the board tree with SRAM; correct the A76 overlay claim
b52dd2c
Exp-2026-10-09: root-cause the IOMMU domain-reference leak
116dfc2
Correct the M.2 section: nothing is confirmed or eliminated
The previous revision presented three things as 'ruled out by measurement'.
They were not. Replaced with a table of what was observed and, for each, what
the observation does not establish:
- the regulator reading 'enabled' is the framework's software state, not 3.3 V
at the connector pins; no voltage was ever measured
- the DTB was compared against our OWN build, not against one captured from a
boot where the NVMe enumerated, so 'known-good' was not earned
- the SPI restore test ran after the link had already failed, so it shows only
that stock SPI does not revive a dead link
- the drive failing in both M-key slots is equally consistent with one latched
drive carried between them
The latched-drive explanation is an untested lead from a single anecdote, not a
cause. Names the two tests that would actually partition the space -- a
known-good device in the slot, and this drive in another host -- neither of
which has been run.
cf2bfb5
ROCK 5B+: the M.2 slot that stopped linking is probably a latched drive
Records the symptom and, more usefully, the three things already ruled out by
measurement so nobody re-chases them: not the regulator (the rail is verified
enabled for the whole training window), not the device tree (booted DTB is
byte-identical to a known-good one), not the bootloader (it linked twice after
the reflash and still failed after restoring the original image byte-for-byte).
The likely cause is the drive latching after an abrupt power cut; reported
independently from earlier Rock 5B experimentation that such a drive returns
once initialised in another machine. Moving it to the board's other M-key slot
fails identically, which misleads toward a 'two dead slots' reading.
Flags why it is worth resolving rather than routing around: those controllers
are the critical path for the intended two-node RDMA PCIe fabric. Also corrects
the table, which still said .235 roots on NVMe; it boots from micro SD now.
a983744
Index the PR #390 result, and flag it on Kernel-Modifications
The 'Build status of #390' section said the series had never been booted. It has
now, on a second board built as armbian plus exactly those ten commits, and it
is not sufficient: orkd_dom_api fails 3/3 until Change 19 (#patchB5) is added.
Flags the reason it was never caught: the runtime testing that section describes
was done on the armbian tree carrying the full board state, which silently
included B5.
1d65920
Exp-2026-10-09 RESOLVED: PR #390 is missing reclaim-and-retry
Bisected and verified. The ten upstreamable commits boot and mostly work but
leave the multi-domain wedge in place; the missing piece is .236's #patchB5
(Change 19). Ported minimally onto the clean series, orkd_dom_api goes FAIL 3/3
-> PASS 3/3, and the recovery is observed rather than inferred: one timeout this
boot instead of six, and one 'reclaim: no jobs in flight but refcount=2 --
forcing to 0'.
The mechanism is stated by PR #390's own code: the switch proceeds only when the
refcount reads exactly zero, and the PR clamps the counter at zero but cannot
recover a count that is already too high. One leaked reference therefore wedges
every later switch and, through rknpu_gem_object_create, every later allocation.
Two limits kept explicit. #patchB5 treats the symptom -- refcount=2 with nothing
in flight means a path still fails to put, and that root cause is NOT found. And
a single isolated pass proves nothing here: orkd_dom_api passes on the UNPATCHED
kernel too when run alone on a fresh boot, because the wedge is created by the
preceding tests. The causal claim rests on the full-suite comparison plus the
reclaim line.
Also records what the change did not cause: test_slice_rescue fails on both arms,
and the one in-suite f16colsplit failure passed 3/3 on isolated re-run.
Patch attached as 24-rknpu-reclaim-leaked-domain-ref.patch (98 lines, two files,
no debug counters) with a recommendation to add it to PR #390.
f153108
DRAFT Exp-2026-10-09: PR #390's ten commits alone do not fix the domain switch
First time the clean upstreamable series has been booted and exercised rather
than only compiled. Built on a second board (.235) from the last pure-armbian
commit plus exactly PR #390's ten commits, stock config, none of .236's
diagnostic hunks.
Most of the suite passes, but orkd_dom_api (the catalogued minimal reproducer)
and test_slice_rescue fail, and netconsole caught the kernel side: six
'switch iommu domain time out' + 'rknpu_gem_object_create error'. The board
survived -- a correctness gap, not a stability one.
Also records that the #patch73 hypothesis stated earlier today looks wrong: the
captured signature is the domain switch timing out, not a dropped completion
interrupt on a powered-down NPU.
Marked DRAFT: which of .236's ~49 extra hunks closes the gap is not yet known.
Changes 19 and 21 have no counterpart in the series and 19 is by its title about
this exact failure, but that is a reading of titles, not code. Bisect running.
b8b2572
Exp-2026-10-09: the fp16 K-slice win is a tie end-to-end, and why
ork_bench on .236, 3 reps per arm, new arm in the disadvantaged first position:
192.98 vs 192.23 prefill, 6.19 vs 6.20 decode. +0.4%/-0.2% -- inside the ~2%
screen and far below this board's arm-position bias band. A tie.
The reason is dispatch, not the fix. ORK_PROFILE reports run_i8 and nothing
else, because ggml-ork routes even an f16 GGUF through the int8 datapath, so
ork_f16_mm_* is never reached. The 2.3-5.0x measured on fp16 shapes in isolation
is real; it just lives in the training path and in any future fp16 routing, not
in inference as currently dispatched. So the 1.0.160 -> 1.0.165 submodule bump is
safe and neutral.
Records the caveat that makes this honest: ork-driver has run_i8/run_i4 profile
counters and no fp16 one, so 'zero fp16 calls' is an absent line rather than a
measurement -- which is why the timing A/B was run instead of resting on it.
Also indexes both of today's pages in the Experiment Log; the training-step page
had been written but never added to the index, and the log is the index of record.
Method note kept for the next person: sudo -E resets HOME to root's, so a bench
script using $HOME/model.gguf under 'sudo -E sh' resolved to /root, the binary
never ran, and the harness printed blank result lines rather than failing.
b3d5431
Exp-2026-10-09: final numbers (3.74x at one NPU core), the colsplit per-call cost, and the orki_f16_bfull regression+fix
1efaf28
Exp-2026-10-09: self-contained repro, make-test/golden-regen note, attribution table
fb28101
Exp-2026-10-09: the fp16 K-slice sat outside its own schedule window (head dX 5.0x), and a full 0.8B training step at 2.5x
964fb44
Correct the ROCK 5B+ page: the A76 2400 change was already documented
I claimed .236's lifted PVTM gate was undocumented. It is not -- Kernel-Modifications
has covered it since 2026-09-01, with better reasoning than I gave: 2304/2352/2400 all
specify the same 1.00 V against a 1.05 V rail ceiling and RK3588's A76 is specified at
2.4 GHz, so the part was factory-conservative, not incapable. That page also records
.236's two clusters binning differently from each other (2304 vs 2352) off identical DT
tables, which is a cleaner demonstration of the per-cluster PVTM measurement than
anything on my page.
What is actually new here is the mechanism: .236 carries the change as an rk3588s.dtsi
edit compiled into the kernel image, which that page itself flags as the wrong packaging
for an optional feature. .235 uses the overlay form it recommends.
Also sharpens the outstanding NPU work: rknpu is built-in on .235 so it needs a full
kernel build, the target is the clean rknpu-iommu-rework series rather than .236's #59
diagnostic tree, and the SRAM enabler should probably be skipped (measured no effect
2026-08-26, costs the video decoders their SRAM).
6fbcb9f
Board provisioning for the ROCK 5B+ (.235): DDR DVFS and the A76 2400 gate
Two things were missing out of the box, neither of them a kernel problem.
DDR DVFS: there was no dmc devfreq device at all, so the DDR governor could not
be pinned and step 3 of the benchmark standard was unperformable. Armbian's
rock-5b-plus U-Boot ships upstream TF-A as BL31, which has the generic Rockchip
SIP dispatcher but none of the DDR services behind it -- confirmed by symbol
presence in the SPI image, not inferred. Rebuilding v2026.04 from Armbian's own
.config with rkbin's BL31 gives ATF version 0x100 and LPDDR5 trained to 2400 MHz.
A76 clock: the cap is Rockchip's PVTM speed bin, not a misconfiguration. The
second cell of opp-supported-hw is a pvtm-volt-sel mask and this die reports
volt-sel=2, so only the 2304 OPP matches. Lifted with a DT overlay, which is
reversible and survives kernel updates where a patched DTB would not.
Notes the part that was not previously written down anywhere: .236 carries that
same gate overridden in its custom kernel's DTB, so every A76 @ 2.4 GHz figure
in this wiki was taken on a bin-overridden board.
9a6f377
Exp-2026-10-08: the clip-grid win SURVIVES held-out (57.90 -> 56.53, 2.4%) — the day's one real quality result
0effa0a
Exp-2026-10-08: mixed-radix rotation — 1.7% seen, 0.33% held-out; a seen-text gain is not evidence
fd7a298
Exp-2026-10-08: HELD-OUT eval — real gap is 1.27x; calibration-on-eval flattered us by ~3.6% relative
53f8404
Exp-2026-10-08: matched-budget promotion — hand prior beats qerr by 3.8%; plus the running-PPL misread
d9ba1f2