RFC: Introduce JZ's ggml-hexagon #26227
Replies: 1 comment
|
JZ's ggml‑hexagon has demonstrated performance advantages for certain models and under sub-4 GiB scenarios. I hope the llama.cpp community (maintainers and collaborators and developers) can support my long‑term efforts on JZ's ggml‑hexagon and help get this PR merged into upstream llama.cpp. To better illustrate the architectural differences between the two Hexagon backends, I'd like to introduce the well‑known duality of Intel backends in llama.cpp, which follows a similar hierarchical design pattern: ggml‑openvino and ggml‑syclare both Intel hardware acceleration backends built on the oneAPI ecosystem, but they adopt fundamentally different execution models (Intel experts/engineers please feel free to correct me if this statement is inaccurate):
Based on this paradigm, I'd like to draw a conceptual analogy for the two Hexagon DSP backends to help llama.cpp community (maintainers and collaborators and developers) quickly understand their architectural gap:
It is worth emphasizing that this is only a conceptual analogy. Unlike OpenVINO, dspqueue does not implement graph compilation or layer fusion. Both Hexagon backends reuse identical DSP‑side compute kernels ------ the performance difference does not come from the kernel operators themselves, but from the scheduling framework, cache policies and offload strategies. Furthermore, the maintenance model also follows this exact correspondence:
Qualcomm's ggml‑hexagon and JZ's ggml‑hexagon are complementary rather than competitive. Each has its own strengths, and both deserve upstream maintenance, much like ggml‑sycl versus ggml‑openvino. For Chinese and Asian developers, a Chinese‑language in-depth technical document is available to quickly explain JZ's ggml‑hexagon and its advantages over Qualcomm's reference implementation:ggml‑hexagon PP/TG Optimization Analysis. Your support for JZ's ggml‑hexagon is highly appreciated to help get this implementation merged into upstream llama.cpp. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Table of Contents
1. Overview
JZ's ggml-hexagon is an alternative llama.cpp backend for Qualcomm Hexagon NPU on Android smartphones, evolved from the original upstream PR #12326 submitted in March 2025. The history of JZ's ggml-hexagon can be reviewed at about ggml-hexagon.
JZ's ggml-hexagon is built on two fundamental architectural choices that originated from the upstream PR #12326 (pls refer to ion-mempool-vs-perbuffer-analysis). This design differs fundamentally from Qualcomm's official ggml-hexagon, which relies on dspqueue and multiple independent per-buffer/per-chunk shared buffers.
2. Architecture and Key Advantages
Single Shared Mempool + Native FastRPC
JZ's ggml-hexagon backend is built around a single shared ION mempool plus native FastRPC, enabling full lm-head offloading onto the NPU - a capability that Qualcomm's official ggml-hexagon cannot support under its current design (pls refer to why-perbuffer-cannot-economically-offload-lmhead). lm-head offloading is the primary factor behind the drastic PP and TG throughput advantage seen in JZ's ggml-hexagon.
The single shared mempool also reduces repeated buffer allocation/deallocation and expensive CPU-NPU cache coherence overhead (pls refer to ion-mempool-vs-perbuffer-analysis; or pls refer to Qualcomm's PR #26049, merge commit 0a50d99).
Role-Aware Cache Coherency
JZ's cache coherency strategy is role-aware: it distinguishes resident weights from per-batch activations and applies first-touch cache invalidation only once for resident weights. Qualcomm's current path is role-blind and performs a full CPU-NPU cache flush/invalidate on every batch, which repeatedly re-flushes large weights such as lm-head.
Benchmark Results
Benchmark results collected on Snapdragon 8 Gen 3 and Snapdragon 8 Elite (8 Gen 4) demonstrate that the PP and TG performance of the JZ ggml-hexagon backend exceeds Qualcomm's official ggml-hexagon when running some modern GQA models. Pls refer to: PP & TG Performance Comparison: Qualcomm's ggml-hexagon vs JZ's ggml-hexagon
Minimal Code Footprint & Coexistence
JZ's ggml-hexagon introduces minimal code footprint. Only 6/7 files are truly new or modified, it can coexist with Qualcomm's official ggml-hexagon and provides a higher-performance alternative for the llama.cpp community:
ggml/src/ggml-hexagon/ggml-hexagon-jz.cppggml/src/ggml-hexagon/kernels/entry.cggml/src/ggml-hexagon/kernels/dsp-ctx.hggml/src/ggml-hexagon/kernels/Makefileggml/src/ggml-hexagon/CMakeLists.txtscripts/build-run-ggmlhexagon-android.shscripts/ggml-hexagon.cfg(optional)Both Qualcomm's ggml-hexagon and JZ's ggml-hexagon build on the Hexagon kernel implementations originally provided by Qualcomm's engineers. Historically, JZ's ggml-hexagon and Qualcomm's ggml-hexagon shared a single DSP kernel directory (htp/) for ops kernels, HVX/HMX headers, and common helpers. This ended with Qualcomm's large PR #26049: after merging upstream master with that PR, JZ's default inference test produced garbled output while Qualcomm's remained normal. The root cause is that PR moved part of the cache maintenance logic into operator implementations, making JZ's cache subsystem incompatible and causing the garbled output, forcing JZ to fork
htp/intokernels/and selected stable upstream htp/ improvements are ported into kernels/ manually. The practical consequence is that JZ's upstream footprint grew from a handful of files to over 80 files.Two backends (Qualcomm's ggml-hexagon and JZ's ggml-hexagon) are isolated in source directories and controlled by independent build flags. Existing official GGML_HEXAGON build option remains untouched for backward compatibility. New flag GGML_HEXAGON_JZ enables the JZ's ggml-hexagon backend. This design serves the purpose of zero breaking changes: when the GGML_HEXAGON_JZ flag is not enabled, the official Hexagon backend can be compiled and run normally.
2.1 Qualcomm Platform Limitations
Community developers working on the Hexagon NPU backend face several platform-level constraints from Qualcomm's DSP software stack that are not present on other NPU platforms.
4 GiB DSP virtual address space limit. Hexagon V79 user-mode has a 32-bit virtual address space (4 GiB), which is a hardware-level hard cap, not a software configuration:
This means the DSP can only mmap/access <= 4 GiB at any time. Models larger than 4 GiB (e.g. Qwen3.5-9B at 5.1 GiB) cannot fit entirely in a single mempool. JZ's backend falls back to heap allocation with mirror memcpy for overflow weights, incurring significant TG overhead (~500 ms/token). Qualcomm's backend bypasses this via per-chunk scatter-gather allocation. AMD APU (Ryzen), Apple Silicon (M1-M4), and NVIDIA Grace Hopper do not have this limitation.
No true unified memory access (UMA). Qualcomm's developer documentation (line 29) claims: "Snapdragon's unified memory model where all buffers are fully accessible by the CPU and GPU." However, the DSP software stack does not implement true UMA:
AMD APU, Apple Silicon, and NVIDIA Grace Hopper have all implemented true UMA, where the NPU/GPU can directly access the full system memory address space without such constraints. Qualcomm's DSP shares LPDDR5X at the physical layer but not at the software layer, creating significant engineering burden for community developers.
Async FastRPC disabled. The FastRPC async API interface exists in the Hexagon SDK, but the underlying driver capability is disabled. JZ's code queries
ASYNC_FASTRPC_SUPPORTat runtime viaggmlhexagon_is_async_fastrpc_supported()(ggml-hexagon-jz.cpp:1460-1493) and the result isasync fastrpc supported 0. This means every FastRPC invoke is synchronous (blocking), forcing JZ's backend into a 12-phase serial submission path with ~21.4 ms per-call overhead. Qualcomm's own backend avoids this via dspqueue ring-buffer writes (which do not go through FastRPC for data exchange), achieving ~0.1 ms per-call overhead. Enabling async FastRPC for community developers would allow JZ's backend to close the 214x per-call overhead gap without adopting dspqueue.These limitations are outside JZ's control and require changes to Qualcomm's DSP OS and Hexagon SDK. If Qualcomm implemented true UMA (direct DSP access to system memory, no VA limit, no mirror requirement), the engineering effort for scatter-gather and async submission paths would be significantly reduced or eliminated.
3. Supported Hardware and Software Requirements
Supported SoCs
Table-1: Supported SoCs
Note: "Usable Threads" = Max HW Threads - 2. Two threads are reserved: one for the HMX queue thread, one for the FastRPC listener. An op needs thread_counts+1 co-resident threads; oversubscribing deadlocks the work-queue barrier because QuRT does not preempt equal-priority workers. The Max HW Threads value is queried at runtime via
qurt_sysenv_get_max_hw_threads(). Seethread_countsin ggml-hexagon.cfg for details.Snapdragon 8 Elite is strongly recommended because:
Software Dependencies (Auto-Downloaded by Script)
The build script (
scripts/build-run-ggmlhexagon-android.sh) automatically downloads and sets up these dependencies on first run:Table-2: Build Dependencies
prebuilts/prebuilts/prebuilts/version: 4Supported Model Aliases
The script supports these model aliases for
run_abtestandrun_llamacli:Table-3: Supported Model Aliases
qwen3-2bhttps://huggingface.co/unsloth/Qwen3.5-2B-GGUFqwen3-9bhttps://huggingface.co/unsloth/Qwen3.5-9B-GGUFgemma4-e2bhttps://huggingface.co/unsloth/gemma-4-E2B-it-GGUFgemma4-e4bhttps://huggingface.co/google/gemma-4-E4B-it-qat-q4_0-ggufqwen1https://huggingface.co/Qwen/Qwen1.5-1.8B-Chat-GGUFllama3https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUFnanbeige-3bhttps://huggingface.com/bartowski/Nanbeige_Nanbeige4.2-3B-GGUFnanbeige-3b-q80https://huggingface.co/bartowski/Nanbeige_Nanbeige4.2-3B-GGUFminicpm5-1bhttps://huggingface.co/Elmermoreno/MiniCPM5-1B-Q4_0-GGUFminicpm5-1b-q80https://huggingface.co/openbmb/MiniCPM5-1B-GGUF(default)https://huggingface.co/unsloth/gemma-4-E2B-it-GGUFNote:
qwen3-9b(5.1 GiB) exceeds the 4 GiB DSP virtual address space limit. The Hexagon V79 PRM documents this as a 32-bit byte-addressable memory address space (Qualcomm online docs). JZ's backend falls back to heap allocation with mirror memcpy for the overflow weights, incurring significant TG overhead. Qualcomm's backend uses scatter-gather to bypass this limit.4. Build Configuration
CMake Options
Table-4: CMake Options
GGML_HEXAGONGGML_HEXAGON_JZGGML_OPENCLHEXAGON_SDK_ROOTHEXAGON_TOOLS_ROOTHTP_ARCH_VERSIONImportant: Both
GGML_HEXAGONandGGML_HEXAGON_JZcan be enabled simultaneously. WhenGGML_HEXAGON_JZis not enabled, the official Hexagon backend builds normally without any changes. WhenGGML_HEXAGON_JZis enabled, JZ's ggml-hexagon backend is built usingggml-hexagon-jz.cpp+kernels/entry.c, and it coexists with the official backend in the same build tree. The build script'sbuildcommand enables both flags (-DGGML_HEXAGON=ON -DGGML_HEXAGON_JZ=ON), whilebuild_qcomenables only-DGGML_HEXAGON=ONfor Qualcomm's official backend.JZ Configuration File
The optional
scripts/ggml-hexagon.cfgfile controls runtime behavior without recompilation:thread_counts: DSP-side thread count (1-8, default 6)dsp_cache_mode: Cache optimization bitmask (default 5 = first-touch + bulk flush)ion_sync_mode: ION cache coherency mode (default 1 = ion_sync only)enable_graph_optimize: cgraph reorder pass (default 1 = enabled)enable_opfusion: QKV/FFN op fusion (default 1 = enabled)fa_select: Flash attention kernel selection (default 2 = HMX -> HVX -> CPU)5. Benchmark Reproduction Guide
5.1 Prerequisites
Linux host (Ubuntu 22.04+ recommended) with:
scripts/build-run-ggmlhexagon-android.sh)Android phone with:
adb(verify withadb devices)Clone the repository:
git clone https://github.com/zhouwg/ggml-hexagon cd ggml-hexagon git checkout self-build-jz5.2 Automated AB Test (Recommended)
This is the simplest way to reproduce benchmark results. The script builds both backends, pushes them to the device, runs N rounds each, and collects performance stats.
Build both backends (run once):
Run automated AB test:
The
run_abtestsyntax isrun_abtest [rounds] [model_alias]. Valid aliases:gemma4-e2b,gemma4-e4b,qwen3-2b,qwen3-9b,qwen1,llama3,minicpm5-1b,nanbeige-3b.Eight-model CI AB test (run all 8 models in one command):
Analyze the log:
prompt eval time(PP) andeval time(TG) for each roundggmlhexagon_dump_perf_statswith detailed phase statisticscommon_perf_print(no phase breakdown)5.3 Manual AB Test
For fine-grained control (e.g., custom prompts or parameters):
Build both backends (same as automated, step 1 above)
Run with Qualcomm's ggml-hexagon:
Run with JZ's ggml-hexagon:
Run with a specific model:
5.4 Expected Output and Log Analysis
A typical AB test log contains:
For each round:
JZ-only perf stats (emitted after each round):
Key metrics to compare:
6. Benchmark Results
Test Environment
Headline Results (2026-08-15, 3 rounds each; qwen3-9b 1 round)
Table-5: Eight-Model AB Test Results (JZ vs QCOM)
Screenshots
Snapdragon 8 Elite (aka 8Gen4), Vendor: OnePlus
PP&TG in Qualcomm's ggml-hexagon:
PP&TG in JZ's ggml-hexagon:
PP&TG in CPU backend:
7. WoA Porting & Linux Porting
Qualcomm engineers would need to implement several cache coherence functions inside ggml-hexagon-jz.cpp to support the WoA (Windows on ARM) platform: https://github.com/zhouwg/ggml-hexagon/blob/newpr_to_upstream/ggml/src/ggml-hexagon/ggml-hexagon-jz.cpp#L1077-L1181
I do not have a suitable Snapdragon‑based Linux device for testing and development, but I believe Linux porting might be easier than WoA porting.
8. Call for Testing and Feedback
You are welcome to share test results and issue reports from various Android phones powered by Qualcomm chips (8 Gen 2, 8 Gen 3, 8 Gen 4, 8 Gen 5). Pls attention benchmark results can diverge significantly when using different GGUF model variants or different Android devices. Below are the steps to reproduce the screenshots above on an Android phone equipped with a Qualcomm mobile SoC.
If you encounter any failures with JZ's ggml‑hexagon, please ping me with detailed runtime logs. I can help extract logs and debug the issue if required.
9. Conclusion
JZ's ggml-hexagon and Qualcomm's ggml-hexagon are complementary, not competitive. The two backends should be treated as complementary, not as competitors: each has its own strength areas, and both deserve upstream maintenance. The direction of the PP/TG gap depends on model type, not on which backend is "better" in general(Table-5).
This shows that JZ's architectural advantages (lm-head offload + mempool contiguous IOVA) are fully unleashed in the < 4 GiB no-split scenario, but suppressed by dual barriers in the > 4 GiB scenario. JZ's ggml-hexagon backend does not support 4+ GiB memory pool in a single FastRPC session due to Qualcomm's limitation -- this is the biggest technical challenge in JZ's ggml-hexagon. Qualcomm's developer documentation claims a "unified memory model where all buffers are fully accessible by the CPU and GPU", but the V79 DSP does not truly support UMA: the 32-bit VA hard cap, disabled async FastRPC, and mirror memcpy requirement all prove this. AMD APU, Apple Silicon, and NVIDIA Grace Hopper have all achieved true UMA; Qualcomm's DSP shares LPDDR5X at the physical layer but not at the software layer.
Practical recommendation: keep both backends and document a model-type-to-backend mapping for llama.cpp users - GQA models to JZ's ggml-hexagon, MHA / shallow-PP models to Qualcomm's ggml-hexagon. Choosing the backend per model type yields the best result on any Snapdragon SoC based device, which is the point of coexistence.
10. References
All reactions