Skip to content

Home

Jump to bottom
Michael Fischer edited this page Jun 20, 2026 · 5 revisions

llama.cpp-rockchip Wiki

Welcome to the development wiki for llama.cpp-rockchip — the open-source fork of llama.cpp enabling high-performance LLM inference on Rockchip NPUs via the native, clean-room userspace ork-driver backend (ggml-ork).

Project Summary

llama.cpp-rockchip integrates the open-source ork-driver compute engine directly into the ggml ecosystem. By bypassing the proprietary, closed-source Rockchip runtimes (librknnrt and librkllmrt), this repository enables a fully open-source, high-efficiency software stack for running large language models natively on Rockchip RK3588 single-board computers (SBCs).

Core Features & Architecture

  • Native ggml-ork Backend: Direct register-level NPU programming via the lightweight ork-driver library, avoiding the CPU overhead of closed runtimes.
  • Heterogeneous Hybrid Inference: Executes heavy matrix prefill operations on the NPU's tensor cores, while seamlessly routing single-token decodes to optimized CPU NEON threads.
  • Accuracy and Consistency: Verified end-to-end against standard FP16 CPU references, preserving numerical precision and vocabulary logit correctness (to $|Δ| \le 0.01$).
  • Optimized CI Pipeline: Leverages a highly tuned, concurrent GitHub Actions workflow powered by a unified rolling ccache layer, delivering rapid cross-compilation feedback.

Technical Documentation & Insights

Explore detailed technical writeups and architectural discoveries:

  • NPU Weight-Residency Decode Bypass Bug — Technical breakdown of the Turn 2+ routing bug where resident weights bypassed the CPU NEON gate, leading to a catastrophic slowdown during single-token decodes ($M=1$).
  • Multi-Turn Decode Performance Slowdown & Resolution — Complete analysis of the $O(n_{\text{ctx}})$ recurrent cell tail-swapping bottleneck and redundant sampler re-reservation overhead that caused deterministic Turn 2+ decode performance drops, and details of their resolutions.
  • NPU Physical-Path Integration Gaps & Roadmap — Architectural comparison between llama.cpp and low-level ork-driver, detailing the MoE expert gating bias, native INT4 multi-M scheduling, and fused Hadamard coordinate rotation.
  • SBC Performance Optimization Roadmap — Memory bandwidth saturation analysis and architectural plan (CPU NEON vs. Mali-G610 OpenCL) to accelerate single-token decode ($M=1$) generation speeds.
  • CI ccache Optimizations — Explanation of the unified rolling cache strategy in GitHub Actions that solved persistent compilation slowdowns and avoided 10 GB GHA cache eviction limits.
  • Benchmark Standards & Methodology — Link to the core profiling protocols to ensure stable, repeatable performance results.

Clone this wiki locally