Skip to content

Home

Jump to bottom
Michael Fischer edited this page Oct 9, 2026 · 19 revisions

ork-driver Wiki

Reverse-engineering record and research notes for ork-driver — the clean-room userspace regcmd matmul library for the Rockchip NPU. The repo documents how the code works (README, docs/); this wiki documents what we discovered about the hardware and why the driver is shaped this way.

Pages

  • Hybrid Precision Decode Architecture — the hybrid CPU/NPU design: one int5 storage format (nibble + MSB bit-plane), NEON inflating it freely for the flexible CPU (W4A5/W5A8), the NPU absorbing the least-sensitive tail (W8A8 now / W4A4 later), sensitivity-routed and overlapped via a swap-hidden decode pipeline. The measurements that forced the shape (streaming-bound MoE offload; int5 ≈ Q4_K accuracy; free NEON inflate).
  • regcmd ISA Reference — the consolidated reference: submission model, regcmd encoding, register meanings, the M-tiling scheduler, operand tile layouts, limits, state gotchas, methodology, and the ruled-out interpretations. Start here.
  • NPU Quirks & Gotchas — hardware/driver quirks that bite silently: IOMMU-domain establishment, dma-buf import limits, int8-mode reset, and other "must-do-X-first" ordering traps.
  • Reverse-Engineering Roadmap — the milestone-by-milestone RE log: ISA decode → fp16/int8 GEMM → tiling/perf → transformer ops → a real model end-to-end.
  • Feasibility Log — the original engineering log (why raw regcmd instead of the closed librknnrt; the DRAM-throttle correction; the two compute routes).
  • Performance — same-state tok/s vs librkllmrt (decode now ~96% via multi-core
    • full-K single-submit), the NPU/RAM-BW telemetry showing decode is idle/parallelism-bound (not memory-bound), and the optimization levers.
  • Experiment Log — a running blog of NPU experiments across the open stack
  • Session findings 2026-09-05 — memory hierarchy, engine dispatch and weight reuse: what is closed, what paid (WEIGHT_REUSE 1.52×), and the method traps that nearly produced false results (ork-driver → ggml-ork → llama.cpp), most recent on top. Each entry has a benchmark table and links to a full writeup with the actual code (commit + symbol + env-knob), method, and verdict.
  • Optimization Roadmap — the prioritized next-steps for closing the open-vs-closed gap (prefill large-M multi-core; decode per-matmul submit; architectural), with status checkboxes and a progress log.
  • Building the Regcmd Graph — the replicable methodology for turning any model + NPU into the fewest-submit on-NPU regcmd graph: segments, the boundary taxonomy (hard / movable / free), the chaining mechanics, and the "collapse to #hard + 1 mega-segments" principle. Includes the Qwen3-1.7B worked example (softmax is the one load-bearing boundary).
  • Benchmark Standards & Methodology — the canonical procedure every perf number must follow (same model, llama-bench warm, governors verified, prefill + decode, ≥2 runs, librkllmrt on the same .rkllm) and the env knobs for A/B arms.
  • Benchmark Configs Registry — the reproducibility ledger: per-run git commits (both repos), env vars, exact tool + args, model/orkpack artifact, board config, result, and the gotchas that silently change the number (layout non-determinism, one-run-per-boot IOVA, CMA red-herring, mmap-eviction). Includes the 7B bandwidth A/B (ork 21% vs rkllm 44% DRAM load).
  • Quantization Paths & Baselines — the w8a8/.rkllm vs GGUF-Q8 vs int4 ceilings and the "never compare across paths" rule (the 178/12.8 figures are w8a8, NOT GGUF-Q8 ~83/4).
  • CPU Big-Core Contention — why a co-resident idle server (e.g. oRKLLM) craters ork-driver decode ~40× via CPU big-core scheduling (NOT NPU contention), and the affinity fix.
  • Heterogeneous Serving & Scheduling — design for splitting prefill/decode across NPU/GPU/CPU and multiplexing workloads (mechanism vs policy).
  • Board Provisioning — ROCK 5B+ — bringing a second RK3588 board (.235) to the benchmark standard: why Armbian's upstream-TF-A BL31 leaves it with no dmc devfreq at all (and the rkbin-BL31 rebuild that fixes it — LPDDR5 trains to 2400 MHz), and the PVTM speed-bin gate that caps the A76 at 2304 MHz (and the DT overlay that lifts it). Also records that .236 carries that same gate overridden — so every "A76 @ 2.4 GHz" figure in this wiki was taken on a bin-overridden board.
  • Recording Investigation Notes — how to add findings here.
  • Robustness & Defensive Guards — userspace protection layers (memory overlap, register sanitization) to prevent IOMMU page faults and hardware wedges.
  • Fuzzing & Dynamic Tracing — our structural exploration framework: dynamic tracing, memory mappings, and automated differential testing on the NPU silicon.
  • Prior Art on Model and Register Fuzzing — academic and industry methodologies (such as constraint-guided graph synthesis and structure-aware register sweeps) and how they apply to the RK3588 NPU.

Status

The driver runs a real model (stories15M) end-to-end (NPU matmul + CPU ops), NPU logits matching a CPU-fp16 reference to ~0.01. RK3588 validated; RK3576 shares the ISA with inherited params (needs on-device validation — see Adding an SoC).

Related (oRKLLM)

ork-driver spun out of oRKLLM (the LLM server that currently ships on Rockchip's closed librkllmrt). The oRKLLM wiki keeps the librkllmrt/Eagle-3 investigations; the open-driver RE moved here.

Clone this wiki locally