RKLLM Runtime Internals

RKLLM Runtime Internals (librkllmrt.so)

Reverse-engineering recon of Rockchip's closed librkllmrt.so, done 2026-06-12 to assess whether the EAGLE-3 3-layer hidden-state problem (see Eagle-3 Debugging Session) is solvable by patching the runtime. Recon only — no patching done.

What it is

librkllmrt.so is a stripped, modified llama.cpp + ggml with a proprietary NPU ggml-backend. This is the single most important fact for any RE work: the runtime is not a black box — its graph structure, KV cache, sampling, and embeddings/output machinery are open-source llama.cpp; only the NPU backend (the matmul offload to /dev/rknpu) is Rockchip-proprietary.

Evidence (string scan of librkllmrt-aarch64-v1.2.3.so, 7.5 MB):

• Build banner: RKLLM SDK (version: 1.2.3 | target: Linux | build: 77daf2e2 2025-11-22 17:09:29)
• llama.cpp internals (mangled symbols in RTTI/asserts): llama_build_graph, llama_build_graph_defrag/_k_shift/_s_copy, llama_context, llama_batch, llama_tensor_quantize_internal, SchemaConverter (grammar), nlohmann::json.
• ggml: ggml_object, ggml_rope_back, ggml_backend_buffer_*, ggml_cpu_has_sve, GGML_MEM_ALIGN, cplan->work_size.
• NPU backend: /dev/rknpu, /%04d-npu_matmul-, config size_e: %d, NPU not support!, D rkllm: malloc input_embed_data mem error, banks num invalid, input_bank_num/weight_bank_num.
• llama.cpp CLI arg strings are linked in (from common): --model-draft, --gpu-layers-draft, --threads-draft, --embeddings, --embd-normalize, --multiline-input. (Their presence shows the llama.cpp speculative and embeddings code is compiled in; it is not wired to the public rkllm_* API.)

Binary properties

• ELF aarch64, stripped — nm shows 0 internal symbols, only 27 exported (all rkllm_*).
• Exported API: rkllm_init, rkllm_run, rkllm_run_async, rkllm_abort, rkllm_destroy, rkllm_clear_kv_cache, rkllm_get_kv_cache_size, rkllm_load_prompt_cache, rkllm_release_prompt_cache, rkllm_load_lora, rkllm_set_chat_template, rkllm_set_cross_attn_params, rkllm_set_function_tools, rkllm_createDefaultParam, rkllm_get_timings, rkllm_print_timings, rkllm_print_memorys, rkllm_accuracy_analysis, rkllm_is_running (+ a few more).
• GET_LAST_HIDDEN_LAYER maps onto llama.cpp's embeddings output path (lctx.embd_size, n_outputs, output_ids, batch.embd, backend_embd, corrupt output buffer (n_outputs=%d)). I.e. it returns the final-layer hidden states via the same mechanism llama_get_embeddings uses.

Feasibility verdict for exposing the 3 aux hidden layers

Feasible in principle, hard in practice — but the open-source base changes the calculus.

• The per-layer residual stream in llama_build_graph is a sequence of ggml tensors. Since GET_LAST_HIDDEN_LAYER already returns the last one as CPU-readable floats, intermediate layers are also ggml tensors that flow back from the NPU backend — i.e. capturable, not NPU-opaque.
• The clean way to expose 3 layers is the way upstream llama.cpp/EAGLE does it: mark layers L_low/L_mid/L_high outputs as graph outputs. But the binary is stripped, so you can't just call llama_build_graph; you'd either (a) hook the ggml NPU-backend compute entry and snapshot the residual at the target layer indices, or (b) match the stripped code against the exact llama.cpp base revision to locate the graph-build and output-selection logic.
• The NPU backend (the matmul offload) is the only proprietary piece; everything around it is known. A hook at the ggml-backend boundary is the most tractable instrumentation point.
• Legal: librkllmrt.so is Apache-2.0 (modification permitted); a patched variant would be a separate artifact, not the unmodified one oRKLLM ships.

Caveats that still apply even with the 3 layers:

• The GET_LOGITS verify path segfaults with keep_history (ggml attention-mask assert) — a separate runtime blocker, see the debugging session.
• Per-version fragility: any of this is tied to build 77daf2e2/1.2.3 and would need redoing per runtime release.

Located addresses (build 77daf2e2 / v1.2.3, file offset == vaddr)

From a deterministic ARM64 ADRP+ADD reference scan (r2's batch xref analysis couldn't resolve these; a ~40-line Python scanner did — decode every adrp + following add #imm and match the composed target):

Address	What	How found
0x2937d4	ggml_abort — the single assert/abort handler every GGML_ASSERT calls	738 sites do adrp/add x3=<expr>; adrp/add x2=<"GGML_ASSERT(%s) failed" @0x5c76a8>; adrp/add x0=<file>; mov w1=<line>; bl 0x2937d4
0x1cae6c	llama_output_reserve assert site ((n_outputs_prev+n_outputs_new)*n_embd <= lctx.embd_size) — hidden-state/logits output-buffer sizing	xref to assert string 0x5db218
0x1ba7c0, 0x1bab04	decode output-copy (corrupt output buffer (n_outputs=%d) @ 0x5d9a90) — moves graph results into the output buffer	xref to 0x5d9a90

Anchor strings: 0x5c76a8 GGML_ASSERT(%s) failed; 0x5db218 the embd_size assert; 0x5d9a90 corrupt-output.

Patch verdict (2026-06-12, attempted)

• Multi-layer hidden export — NOT a constant flip. The 3 aux layers are never written to the output buffer, so there's nothing in llama_output_reserve/the copy to redirect. They'd have to be added as graph outputs in llama_build_graph (code insertion → detour/code-cave) + the buffer resized + copied. Deep surgery into stripped optimized ARM64; multi-week, version-fragile, and high-risk for the shared runtime. Not done.
• Verify-crash assert — the only single-site lever is global and unsafe. All asserts funnel through ggml_abort @ 0x2937d4; NOP-ing it to ret (0xd65f03c0) would bypass every GGML_ASSERT during normal inference too (clean aborts → silent corruption), and wouldn't fix the underlying mask->ne[0]==a->ne[0] dimension mismatch. Bypassing a single specific site needs locating that assert's conditional branch (one of 738) and is still risky. Not applied to the live runtime.

Verify crash — FIXED by a 2-instruction binary patch (2026-06-12, empirically tested). The mask->ne[0]==a->ne[0] assert string is at 0x5e8508; its abort tail is at 0x2b56c8 (sibling mask->ne[1]>=a->ne[1] at 0x2b56b8), inside ggml_soft_max_ext's shape validation. The guarding conditional branches are:

• 0x2b55a8: b.ne 0x2b56c8 (54000901) — fires the ne[0] assert
• 0x2b55b8: b.lt 0x2b56b8 (5400080b) — fires the ne[1] assert

NOP-ing both (d503201f, little-endian 1f 20 03 d5) on a copy of the .so and testing on-board:

• The EAGLE verify (GET_LOGITS, token-input, keep_history=true) that previously segfaulted now returns valid logits [151936 × 8] — no crash, no OOB.
• Normal generation is unaffected ("The capital of France is **Paris**.") — expected, since the asserts pass on the normal path so the NOPs are no-ops there.

This corrects an earlier (wrong) prediction in this wiki that bypassing the mask assert would yield garbage/OOB: empirically the soft_max handles the size difference fine — the assert was stricter than the compute requires for this code path. So the verify crash is fixable with a surgical, safe 2-NOP patch (only those two asserts; all others intact). The patch is reproducible: NOP the two branches above. It does not by itself make Eagle-3 accelerate — see blocker #1 below.

Conclusion (status of the two binary-patchable blockers):

• Blocker #2 — verify keep_history crash: FIXED by the surgical 2-NOP patch above (tested: valid logits, normal gen unaffected). Safe to deploy. ✅
• Blocker #1 — draft quality (the 3 aux layers): still requires deep work. The 3 layers are never graph outputs, so exposing them needs code insertion into llama_build_graph (no constant to flip) to capture the raw residual at L_low/mid/high (truncating the layer loop only yields output_norm(h_L), not the raw residual the EAGLE head's fc was trained on). That's the genuine multi-week RE piece (interactive Ghidra + hardware debugger + matched llama.cpp base revision).
• Global ggml_abort NOP (0x2937d4) — rejected as unsafe for the shared runtime (the surgical 2-NOP above is the correct approach).

So: the verify crash is solved by a binary patch; Eagle-3 acceleration is still gated on the draft producing accepted tokens, which needs either the multi-week graph-output insertion or — the lower-risk route — training a single-last-layer draft head that fits RKLLM's native 1-layer GET_LAST_HIDDEN_LAYER output (no runtime modification). The verify patch is necessary infrastructure for either.

Version diff (2026-06-12): no shortcut, hidden-state code is stable

Compared the hidden-state/embeddings machinery across v1.1.0 → v1.2.0 → v1.2.3 (all runtimes are on the board). The embeddings/output path is the same standard llama.cpp code in every version (lctx.embd_size, n_outputs, corrupt output buffer, single-layer); no version exposes multi-layer/aux hidden states or eagle support. So:

• There is no older/newer runtime that returns the 3 layers — single-layer is fundamental.
• Version-diffing (a good technique) here yields a useful negative: it rules out a version shortcut and confirms the hidden-state code is the stable upstream llama.cpp path, so it can be matched against open-source llama.cpp to understand it. To do function-level diffing for locating code, you'd need Ghidra BinDiff/Diaphora (raw byte diff of stripped 5 MB binaries is too noisy — every address shifts).

On core dumps: useful for confirming the residual-buffer layout, but they don't sidestep the hard part — llama.cpp reuses one ggml residual buffer and overwrites it each layer, so the 3 intermediate layers are never simultaneously resident. Capturing them requires hooking the per-layer compute (in-loop snapshot), not reading a static address.

Layer-loop injection point for multi-layer capture (2026-06-12)

Located the graph-builder layer loops via xrefs to the "l_out" graph-node name (0x5deb38): 0x233bac, 0x234648, 0x235058, 0x235714, 0x236270, 0x23726c (one per model-arch builder; the Qwen3 one is among them). Disassembling the first builder's loop tail maps the registers needed for a multi-layer-capture patch:

0x233ba0  mov  x24, x0        ; x24 = cur  (this layer's output residual tensor*)
0x233ba8  ldr  x1, [x19,#0xf8]; x1 = cb std::function object (the build callback)
0x233bac  add  x0, x0, #0xb38 ; x0 = "l_out" name string
0x233bb4  stp  x24, x0, [sp,#0x120]
0x233bd8  blr  x4             ; cb(cur, "l_out", il)  — per-layer callback
0x233bd4  add  x20, x20, #1   ; x20 = il (layer index)
0x233be0  ldr  x0, [x19,#0x38]; x0 = n_layer (loop bound)
0x233be8  b.le 0x233d58       ; exit when n_layer <= il

This is the clean injection target. Each layer's raw residual cur (x24) and index il (x20) pass through a callback here — exactly where llama.cpp's EAGLE/aux-hidden capture attaches. The remaining notes about eval_callback: the runtime does not link ggml_backend_sched_set_eval_callback (only alloc_splits/split_graph), so the post-compute per-node callback route isn't available — the build-time cb above is the hook.

What a working multi-layer patch still requires (the multi-week part):

1. A code cave / detour at 0x233bd8 (or in the cb body) that, for il ∈ {low,mid,high}, calls ggml_set_output(cur) so those residuals survive compute (need ggml_set_output's address; the NPU backend must then sync them to host).
2. Patch the hidden-state extraction (llama_output_reserve 0x1cae6c + the copy 0x1ba7c0) to gather the 3 marked tensors into the result buffer with embd_size = 3×n_embd instead of just result_norm.
3. Addon side: stop the 3×-replication, consume the real 3-layer concat. Each step needs on-board iteration. This is a genuine multi-week effort with proper tooling (Ghidra + a hardware debugger on a dev board) — the injection point is found, but the cross-function code insertion + NPU-sync behaviour are the substantial, risky remainder.

NPU-controller boundary, and "can we marry it to a newer llama.cpp?" (2026-06-12)

Mapped how librkllmrt.so drives the NPU, to assess reusing Rockchip's NPU offload with an upstream llama.cpp (which has native EAGLE-3 / hidden-state support).

The NPU is NOT a registered ggml-backend. The ggml-backend registry in the binary only references the CPU backend (ggml_backend_cpu_buffer_type_alloc_buffer, ggml_backend_reg_init_backend_from_str, ggml_backend_registry_count). There is no ggml_backend_rknpu registration. So the NPU matmul is not a clean, swappable backend behind the ggml_backend_i vtable — it's hooked directly into ggml's compute.

The NPU controller is a low-level regcmd/task-submission layer (raw driver programming, not the high-level RKNN API):

• opens /dev/rknpu directly (failed to open … need to insmod rknpu driver!, failed to check rknpu hardware version)
• builds register command buffers (failed to read regcmd: must read size…)
• submits an NPU task queue: failed to submit, op id: %d, op name: %s, flags, task start, task number, run task counter, int status — i.e. it programs NPU tasks and polls completion/interrupt status
• profiles ops as /%04d-npu_matmul-
• also links an OpenCL path (clCreateCommandQueue/clReleaseCommandQueue — likely a Mali/GPU fallback for some ops) and uses DRM (/dev/dri, DRM_IOCTL_VERSION)

Feasibility verdict (marry NPU controller → newer llama.cpp): impractical as an extraction; viable only as a from-scratch backend.

• Extracting Rockchip's offload from the stripped monolith and linking it to a new llama.cpp won't work: it's not a self-contained ggml-backend (no clean ABI boundary), it's fused into their specific ggml fork, and it's coupled to the .rkllm weight format (NPU-quantized weights, "banks"). A newer llama.cpp has a different ggml ABI → incompatible.
• The sound version of the idea is writing a fresh ggml_backend_rknpu (the proper registrable kind) that targets the /dev/rknpu regcmd/task-submission interface, then using it with current llama.cpp (inheriting EAGLE-3, multi-layer hidden output, speculative decoding for free). But the regcmd/task protocol is undocumented, and the NPU matmul kernels + weight-bank/quantization layout are the proprietary hard part. That's a multi-month RE + kernel-authoring project — strictly bigger than the in-place patch, but the only sustainable architecture (you'd stop fighting a frozen llama.cpp fork).

Implication for the multi-layer goal: because the NPU computes matmuls via regcmd submission and writes results back into ggml tensors, intermediate-layer residuals are produced — but whether marking them as graph outputs forces a host sync depends on this custom offload, and must be tested on-board (it's not the standard ggml-backend sync path).

Output-setup + graph-finalize mapping (2026-06-12)

Disassembled builder #1's post-loop tail (0x233d58–0x233e28) — how the final hidden becomes the returned embedding:

0x233d78  bl 0x1b0b64        ; build final norm -> x20 = result tensor
0x233d88  add x1, "result_norm"(0x5deb40)
0x233dac  blr x4             ; cb(result, "result_norm", -1)
0x233df8  blr x4             ; cb(_, "result_embd", -1)   (0x5deb50)
0x233e08  bl 0x2bf5c4        ; graph-finalize / build_forward_expand(gf, result)  ← retains/marks output
0x233e28  ret

So 0x2bf5c4 is the "add this tensor to the compute graph as a retained output" primitive (called once on the final result). The per-layer cb(cur,"l_out",il) callback object is at [x19+0xf8] (loaded repeatedly, e.g. 0x233d80).

Refined multi-layer-capture patch design (for the next session):

1. In the layer loop, right after cur is finalized (x24 @ 0x233ba0), for il ∈ {low,mid,high} call 0x2bf5c4(gf, cur) so those residuals are computed and retained (instead of overwritten). Need: the gf graph pointer's location during the loop (TBD — trace what 0x2bf5c4 takes as args at 0x233e08: x0/x1 set at 0x233e00-0x233e04), and a code cave for the conditional.
2. Patch the hidden-state extraction (0x1cae6c reserve + 0x1ba7c0 copy, which key off the "result_norm" tensor 0x1c724c) to also emit the 3 retained tensors → embd_size = 3×n_embd.
3. Confirm the NPU offload syncs the retained intermediate tensors to host (the open risk — it's not the standard ggml-backend sync path; must test on-board).
4. Identify which of the 6 builders Qwen3 uses (l_out sites 0x233bac/0x234648/0x235058/0x235714/0x236270/0x23726c) — patch the right one.

This is the concrete worklist; each item is a separate RE+on-board-test increment.

Practical recommendation

Don't start with core dumps. Start with: (1) identify the llama.cpp base revision the build tracks (diff behaviour/strings against tagged llama.cpp releases near 2025-11), (2) locate the ggml NPU-backend compute function (it reads/writes the residual tensors), (3) prototype a hook (LD_PRELOAD on a ggml-backend symbol if one is dynamically resolved, else a trampoline) that copies the residual at chosen layer indices. This is a multi-week effort and version-fragile; weigh it against training a single-last-layer draft head (works within RKLLM's native 1-layer output — see the debugging page's options) which needs no runtime changes.