Repository navigation
Exp 2026 06 24 RKLLM Multicore Capture
Date: 2026-06-24
Method: LD_PRELOAD the regcmd_capture.so shim (submit-only mode, ORK_SUBMIT_ONLY=1) into the
orkllm service (node src/server.js, which loads librkllmrt-aarch64-v1.3.0.so), send one chat
completion to a .rkllm w8a8 model (Qwen3-0.6B-Base-rk3588-w8a8), and dump every RKNPU_SUBMIT
ioctl's core_mask / task_number / subcore_task. 1039 submits captured for one short prompt.
| count | |
|---|---|
core_mask=0x1 (core 0) |
365 |
core_mask=0x2 (core 1) |
365 |
core_mask=0x4 (core 2) |
309 |
core_mask=0x7 (all 3, single submit) |
0 |
task_number per submit |
count |
|---|---|
| 3 | 588 |
| 6 | 448 |
| 21 | 3 |
librkllmrt issues only SINGLE-core submits, round-robined across cores 0/1/2 (the masks cycle
0x2,0x1,0x4,0x1,0x2,0x4,…). It never uses core_mask=0x7. Each submit PC-chains 3–6 tasks
(21 for the 3 prefill submits). So the closed runtime's "multi-core" is temporal load-balancing of
single-core jobs across the 3 cores, not splitting one op across cores.
-
Option A is confirmed dead. The single-submit multi-core path (
core_mask=0x7+ per-coresubcore_taskranges) that hard-wedged our board is not how anyone drives the 3 cores — librkllmrt never emits it. It isn't a supported job mode; don't pursue it. -
Option B is the correct mechanism, and it's literally what the closed runtime does:
independent single-core submits (
core_mask=1<<i), and PC-chaining multiple tasks per submit — exactly ourrun_chain_i8+ per-core fan-out. Our chaining primitive matches the closed runtime's building block (3–6 chained tasks/submit). -
The closed runtime does NOT do intra-op multi-core (no single matmul split across cores). Our
run_multicore(N-split one matmul across 3 cores, 2.27×) is something librkllmrt's matmul path does not do — we're actually ahead there. librkllmrt instead overlaps the model's many sequential ops by round-robining whole single-core (chained) submits across cores.
Implication for cross-core chain fan-out (Tier 6)
Our fan-out (split one chain's programs across cores, barrier + join) measured ~1.1–1.5× — limited by DDR bandwidth (independent matmuls bring distinct weights) and concurrent-submit overhead. The capture shows the closed runtime's strategy is the same building block (single-core chained submits) but applied as a steady round-robin over a long op stream rather than a barriered fan-out of one chain. The likely path to better utilization for our batched workloads (EAGLE-3 verification) is to mirror that: assign successive independent matmuls/chains to cores round-robin and let them overlap asynchronously, rather than barrier-syncing one chain's slices.
Reproduce: gcc -shared -fPIC -Isrc -o regcmd_capture.so tools/regcmd_capture.c -ldl; stop orkllm;
LD_PRELOAD=…/regcmd_capture.so ORK_SUBMIT_ONLY=1 node /usr/lib/orkllm/src/server.js 2>cap.log; send a
chat completion to a .rkllm model; grep '=== SUBMIT' cap.log | grep -oE 'core=0x[0-9a-f]+' | sort | uniq -c.