Replies: 8 comments 3 replies
|
|
|
JZ ggml-hexagon: Structure and Optimization Analysis (2026-07-22)Author
1. BackgroundBoth the JZ and Qualcomm versions of the ggml-hexagon backend route through Qualcomm's JZ ggml-hexagon is built on two fundamental architectural choices that originated from the upstream PR #12326 (March 2025):
Based on these two choices, JZ ggml-hexagon sidesteps both the These are deliberate design choices, not limitations. The theoretical basis is that LLM inference is inherently serial (autoregressive TG + serially dependent subgraphs), which limits the benefit of async pipelining. 2. Current Benchmark (2026-07-22)2.1 Test Conditions
2.2 PP and TG Comparison (5 runs each, same device, same model, same day)Table 1: JZ vs Qualcomm PP/TG (5 runs, 2026-07-22)
JZ exceeds Qualcomm on both PP and TG:
Both backends run the same HMX kernels from 2.3 JZ-side RPC stats (run 1)Relevant FilesTable 2: Relevant files
3. FastRPC Call Pattern3.1 JZ Version
3.2 Qualcomm Version
3.3 Performance ImpactJZ's synchronous FastRPC architecture is a deliberate design choice. The pure FastRPC/ION transport overhead is ~89 us/call (warmup probe), which is negligible against the ~36 ms/token DSP execution time. With lm-head offloaded to DSP and 4. Op Fusion4.1 JZ Version (Phase 2.5)Table 3: JZ op fusion types (Phase 2.5)
4.2 VTCM budget check for MUL_MAT_ADD fusionThe MUL_MAT + ADD fusion checks the VTCM budget before firing, mirroring Qualcomm's guard (ggml-hexagon.cpp:3595): const size_t vtcm_budget = (size_t)ctx->socinfo.vtcm_size_in_mb * 1024 * 1024;
if ((size_t)kparams->vtcm_size > vtcm_budget) {
return false; // skip fusion, let MUL_MAT and ADD run separately
}4.3 Qualcomm Version (
|
| Fusion Type | Supported | Notes |
|---|---|---|
| RMS_NORM + MUL -> HTP_OP_RMS_NORM_MUL | Yes | Uses ggml_can_fuse |
| MUL_MAT + ADD -> HTP_OP_MUL_MAT_ADD | Yes | Uses ggml_can_fuse |
| MUL_MAT QKV merge -> HTP_OP_MUL_MAT_QKV | Yes | 3 mul_mat merged into 1, reordered to KVQ |
| MUL_MAT FFN merge -> HTP_OP_MUL_MAT_FFN | Yes | gate + up merged into 1 |
| Graph reorder | Yes | Stacks MUL_MATs with same src1 for VTCM reuse |
Fusion scope is at parity (all 5 fusion types). Graph reorder is implemented in JZ and runtime-configurable; for gemma4 PP=58 tokens on 8 Elite v79 it is a no-op (within noise), but kept on by default for future-proofing.
5. Graph Cache
The cache uses a content hash (FNV-1a over each node's {op, ne[4], nb[4], src[0..2] ptr, data ptr}) instead of the dead cgraph->uid key:
const uint64_t content_hash = compute_content_hash();
auto it = ctx->cgraph_cache.find(content_hash);
if (it != ctx->cgraph_cache.end() &&
it->second.n_nodes == cgraph->n_nodes &&
it->second.hex_ops.size() > 0) {
cache_hit = true;
ctx->cgraph_cache_hits++;
}On a hit, the cache skips Phase 1 (tensor dedup), Phase 2 (op descriptor build), and Phase 2.5 (op fusion) entirely. Current hit rate is 98.8% (253 hits / 256 calls). The 3 misses correspond to the first fill of 3 unique graph structures. After warmup, every subsequent token hits cache.
6. mm_params_cache (JZ only, enabled)
JZ caches precomputed htp_mm_kernel_params by a composite key (src0 tensor pointer XOR weight data pointer XOR ne11):
const uintptr_t cache_key = (uintptr_t) src0 ^ (uintptr_t) src0->data ^ ((uintptr_t) ne11 << 32);
auto it = ctx->mm_params_cache.find(cache_key);
if (it != ctx->mm_params_cache.end()) {
*kparams = it->second;
return;
}This skips the multi-hundred-microsecond thread/chunk search in htp_mm_hvx_vtcm_layout_build / htp_mm_hmx_vtcm_layout_build for repeated MUL_MAT calls with the same weight tensor. For TG (where ne11=1 for every token), the cache hits after the first token. The cached kparams are valid for the session lifetime because weights are static (never modified after model load).
Qualcomm's ggml_hexagon_precompute_matmul_params performs the same VTCM layout computation but does not cache the result across calls.
7. Weight Repack
JZ implements a repack buffer type with is_host=false:
static bool ggml_backend_hexagon_repack_buffer_is_host(ggml_backend_buffer_type_t buft) {
return false; // forces GGML core to call set_tensor
}When the model loader encounters quantized weights (Q4_0, Q4_1, Q8_0, IQ4_NL, MXFP4), the supports_op gate in MUL_MAT ensures they are allocated in the repack buffer type. Because is_host=false, GGML core routes data through set_tensor, which performs the in-place tile repack at model load time (one-time cost).
lm-head DSP offload
The lm-head (262144x1536, Q4_K) is offloaded to DSP. The ne[1] > 32768 rejection for large quantized weights was removed, and a new repack_q4k_as_q4_0_tiled_to_buf converter (ggml-hexagon-jz.cpp:4162) stores the lm-head as Q4_0 tiled layout in the single ION pool (~214 MB, session-resident, streamed from DDR once per token). This eliminates the ~30 ms/token CPU lm-head matvec that was the single largest TG cost. Qualcomm's per-buffer ION design makes a 214 MB per-buffer map/unmap per session prohibitive, so their lm-head stays on CPU.
8. Cache Coherency Management
8.1 JZ Version
- AP side: Configurable via
ion_sync_mode0= both (DC CVAC + ion_sync, default)1= ion_sync only (DMA_BUF_IOCTL_SYNC, driver-level) - optimal2= DC CVAC only (manual cache line management)
- DSP side: Configurable via
dsp_cache_modebitmask (default = 5)- bit 0: first-touch weight invalidation (exact sorted pointer array
g_weight_inval_ptrs,WEIGHT_INVAL_MAX_PTRS=4096in entry.c; repack weights written once at model load, single first-touch dcinva covers the whole session) - bit 1: skip dcinva for prior dst (off - deferred-flush pattern is unsafe to combine with bit2 for anything above a single cacheline)
- bit 2: bulk dst flush at batch end (collect/sort/merge dst ranges)
- bit 3: selective bulk flush (skip batch-end flush for intermediates still consumed by later ops in the same batch;
g_tensor_last_use_opin entry.c)
- bit 0: first-touch weight invalidation (exact sorted pointer array
8.2 dsp_cache_mode=5 default
Mode 5 (bit0 + bit2) is the default. The first-touch weight invalidation tracker uses an exact sorted pointer array instead of a hash bitmap (whose address collisions caused garble in earlier iterations). A two-pass defense (DSP-side unmark on dst write + AP-side ever-dst set) closes the cross-graph stale-read window. This is what made the session-resident repacked lm-head practical: per-token weight traffic is ~1.9 GB, and re-invalidating it every token would cost ~9.2 ms/token of DSP-side dcinva sweeping; with first-touch, that cost is paid once at session start.
8.3 ion_sync_mode=1 is optimal
Mode 1 uses the kernel's DMA_BUF_IOCTL_SYNC which is faster than userspace DC CVAC/CIVAC for large ranges. Mode 0 (double cache maintenance) drops PP significantly; mode 2 (manual only) is similar to mode 0.
8.4 Qualcomm Version
dspqueue driver automatic management via DSPQUEUE_BUFFER_FLAG_FLUSH_SENDER | DSPQUEUE_BUFFER_FLAG_INVALIDATE_RECIPIENT. The driver applies uniform per-batch flush/invalidate flags to every buffer - there is no per-role differentiation. JZ's user-space management distinguishes weight tensors (flags=2, written once at load) from activations, so bit 0 can eliminate their per-token re-invalidation. This policy flexibility is an advantage the closed driver cannot replicate without per-role buffer semantics.
9. Session Consistency Gate
JZ mirrors Qualcomm's ggml_hexagon_supported_buffer check to prevent the scheduler from mixing tensors across different Hexagon sessions or non-Hexagon buffers:
static bool ggmlhexagon_tensor_buffer_is_owned_by(ggml_backend_dev_t dev, const struct ggml_tensor * t) {
if (!t || !t->buffer) return true; // neutral
// Accept if buffer is hexagon (main or repack) on this device
// Reject if hexagon on different device or non-hexagon
}10. VTCM Session-Lifetime
VTCM is acquired once per session, not per batch. ggml_dsp_open does one HAP_compute_res_acquire_cached and ggml_dsp_close does one HAP_compute_res_release_cached. VTCM is held continuously for the session. This matches the Qualcomm HTP pattern (vtcm_acquire / vtcm_release only fire when transitioning between "active processing" and "forced release").
// ggml_dsp_open (after HAP_compute_res_acquire succeeds)
dsp_vtcm_acquire(); // once per session, sets vtcm_valid=1
// ggml_dsp_close (before HAP_compute_res_release)
dsp_vtcm_release(); // once per session, sets vtcm_valid=0
// execute_batch: no per-batch acquire/release callsTrade-off: lose the ability to respond to a forced-release callback from another session. For single-session use (the current deployment) this is a non-issue.
11. Dual Path Removal
JZ routes all ops through the shared htp/ execute_op path, exactly like Qualcomm. The entire algotype=32 path (JZ's own self-built kernel dispatch in kernels/mulmat.c, kernels/flash_attn.c, etc.) and the ggml-dsp port (kernels/ggml-dsp.c 9946 lines, kernels/ggml-dsp.h 2256 lines) are deleted - 24298 lines removed. JZ's four files are now merged into htp/ alongside Qualcomm's kernels:
entry.c- FastRPC entry point, cache management,dsptensor<->htp_tensorbridge,execute_opdispatchdsp-ctx.h-struct dsp_context,dsptensor,hex_tensor_desc,hex_op_desc,hex_batch_hdrggml_dsp.idl- FastRPC interfaceMakefile- buildslibggmldsp-skel.sofromentry.c+ allhtp/*.csources
The mulmat_algotype config knob is removed entirely. The "algotype=29" label survives only as a historical comment in ggml-hexagon.cfg and in this document filename.
dsptensor is retained as JZ's AP-side tensor descriptor format (single ION offset addressing), but it is now only a thin wrapper. The dsptensor_to_htp_tensor bridge in entry.c converts it to the shared htp_tensor (with bi=0) before calling execute_op.
12. Tensor Descriptor Data Structure
12.1 JZ Version
JZ uses three descriptor types at different stages:
AP-side: hex_tensor_desc (single ION offset addressing).
typedef struct hex_tensor_desc {
int32_t type;
int32_t ne[4];
int32_t nb[4];
int32_t op_params[16];
uint32_t flags; // 0=ION, 1=mirrored, 2=weight(skip flush)
uint32_t data_offset;
uint32_t data_len;
} hex_tensor_desc;DSP-side: dsptensor (defined in dsp-ctx.h)
struct dsptensor {
int32_t type;
int32_t ne[4];
int32_t nb[4];
int32_t op; // op code embedded in tensor descriptor
int32_t op_params[16]; // op-specific params embedded in tensor descriptor
int32_t flags;
void * data; // direct pointer (DSP address space is 32-bit)
int data_len;
};12.2 Qualcomm Version htp_tensor
struct htp_tensor {
uint32_t data;
uint32_t size;
uint32_t flags;
uint16_t type;
uint16_t bi; // buffer index
uint32_t ne[4];
uint32_t nb[4];
};12.3 Difference
Table 5: Tensor descriptor comparison
| Aspect | dsptensor (JZ) |
htp_tensor (Qualcomm) |
|---|---|---|
| Op metadata | Embedded (op + op_params[16]) |
Separated into htp_op_desc |
| Data addressing | Direct void * pointer |
bi (buffer index) + uint32_t offset |
| Multi-buffer support | No (single ION pool) | Yes (via bi indexing into htp_buf_desc[]) |
| Type width | int32_t |
uint16_t/uint32_t (more compact) |
The bi field is the key differentiator for multi-buffer support. JZ always passes bi=0 (single ION pool); Qualcomm uses it to index into htp_buf_desc[].
13. Upstream Merge Adaptations
13.1 Unary precompute port
Upstream commit fb30ba9a6 introduced a new op_unary pipeline that requires host-precomputed htp_unary_kernel_params (n_threads, col_tile, vtcm_size, etc.). JZ ported ggml_hexagon_precompute_unary_params from ggml-hexagon.cpp, adapted to use ctx->n_threads and ctx->socinfo.vtcm_size. HTP_OP_TRI is routed to op_unary() in entry.c to match upstream htp/main.c.
13.2 VTCM layout API
Upstream commit 81ff7abe5 brought in Qualcomm's new VTCM layout API:
htp_mm_hvx_get_vtcm_sizes->htp_mm_hvx_vtcm_layout_build+struct htp_mm_hvx_vtcm_layouthtp_mm_hvx_id_get_vtcm_sizes->htp_mm_hmx_vtcm_layout_build+struct htp_mm_hmx_vtcm_layoutbroadcast_rk2/rk3/rv2/rv3fields moved fromu.hvxunion member tohtp_fa_kernel_paramsstruct top level
JZ adapted via adapter functions in ggml-hexagon-jz.cpp and entry.c that translate old API calls to the new layout-build API.
13.3 Build system unification
The unified CMakeLists.txt is based on QCOM's version (minimal diff to upstream) with a single addition:
option(GGML_HEXAGON_JZ "Use JZ's AP implementation" OFF)GGML_HEXAGON_JZ=OFF(default): exactly QCOM's upstream behavior. Usesggml-hexagon.cpp, builds DSP skels viaExternalProject_Addfor v73/v75/v79/v81, linkshtp_ifacestub.GGML_HEXAGON_JZ=ON: usesggml-hexagon-jz.cpp, builds a single DSP skel viamake -C htp/(JZ's Makefile), linkscdsprpc, setsHEXAGON_DEFAULT_LIB_SEARCH_PATH, copiesggml-hexagon.cfg.
14. Compiler Optimization
14.1 AP-side (PARITY)
Both backends compile AP-side code with the same ARMv8.7-A + dotprod + fp16 + i8mm flags:
set(OPT_FLAG " -O3 -march=armv8.7-a+dotprod+fp16+i8mm -mcpu=cortex-x1 -mtune=cortex-x1 -ffp-model=fast -fno-finite-math-only")14.2 DSP-side (JZ-only)
JZ's htp/Makefile uses -O3 -ffast-math -fno-vectorize (no LTO) with -DNDEBUG for the DSP skel. Qualcomm's htp/cmake-toolchain.cmake uses -O2 -flto -fvectorize. A 3-row sweep confirmed LTO is a net regression for JZ's codebase (-9% to -14% PP); -O3 no-LTO is empirically the best choice.
14.3 flash-attn-ops.c -O2 workaround
flash-attn-ops.c is forced to -O2 because at -O3 Hexagon LLVM 19.0.07 emits a PromoteFloatResult fatal error on f16 = freeze. A 10-flag sweep confirmed no combination of -fno-X / -mllvm -disable-X flags can break the workaround. The bug is in the Hexagon backend (HexagonDAGToDAGISel -> PromoteFloatResult), not in any front-end pass. Needs an LLVM backend patch.
15. Profiler Infrastructure
15.1 AP-side profiler
Tracks cumulative time for phases p1, p2, p2.5, p3, p4, p4.5, p5, p6, p6.5, p7, p7.5, p8 per graph_compute_batch call. Breaks down Phase 7 into rpc_setup + dsp_exec + civac. Computes min/p50/p95/max histograms for the last 1024 calls. Reports cgraph cache hit/miss counts.
15.2 longtail profiler
A LOG_ALWAYS longtail probe inside the FastRPC dispatch path logs the op composition of any batch call whose dsp_exec exceeds 5 ms. Throttled to one log per 100 ms wall-clock. Probe code is preserved in source inside #if 0 ... #endif for future re-activation; runtime cost is zero.
15.3 mul_mat coverage tracer
Per-batch counters in hexagon_op_exec_stats_t: n_mul_mat_total, n_hmx_used, n_fused_qkv, n_fused_ffn, n_fused_mul_mat_add.
15.4 DSP-side profiler (gated off by default)
entry.c includes a per-op timing profiler that records min/max/avg execution time per op type, dumped via dump_op_prof. Wrapped in #if HEX_OP_PROF ... #endif with HEX_OP_PROF default to 0 (off). Restore by passing -DHEX_OP_PROF=1 to make.
16. v75/8Gen3 Thread Clamp
On Snapdragon 8 Gen 3 (v75), the code deadlocks at the default thread_counts=6 due to cDSP hardware-thread oversubscription. An op needs thread_counts+1 co-resident QuRT threads (FastRPC main + N-1 work-queue workers + 1 hmx_queue thread); v75 has 6 hardware threads (v79 has 8), so N=6 requires 7 > 6 and the unscheduled worker never decrements the task barrier.
ggml_dsp_setclocks clamps to max_hw_threads - 2 and reports the effective value through a rout int32 real_thread_counts IDL out-param; AP mirrors it into ctx->n_threads so the precomputed kparams->n_threads matches the DSP work-queue. On v75 this clamps to 4; on v79 it stays at 6.
17. Summary
Table 6: Performance summary (2026-07-22, 5-run mean)
| Backend | PP (tok/s) | TG (tok/s) |
|---|---|---|
| JZ | 686.46 | 26.91 |
| QCOM | 435.14 | 24.91 |
| JZ advantage | 1.58x | 1.08x |
JZ exceeds Qualcomm on both PP and TG. The three enabling changes:
- lm-head offloaded to DSP - Q4_K stored as Q4_0 tiled repack (~214 MB, session-resident in the single ION pool). Eliminated the ~30 ms/token CPU lm-head matvec. The single ION pool turns this into a one-time repack; Qualcomm's per-buffer ION design makes a 214 MB per-buffer map/unmap per session prohibitive.
- dsp_cache_mode=5 (bit0 + bit2) - First-touch weight invalidation via exact sorted pointer array + two-pass defense. Eliminates ~9.2 ms/token of redundant weight re-invalidation. bit3 (selective bulk flush) also added.
- DSP-side debug logging removed - Skel built with
-DNDEBUG; FARF debug paths compiled out.
Also: BF16 added to offloaded MUL_MAT types (stored as F16 in repack buffer, reusing F16 DSP kernels).
Table 7: Optimization status
| Item | Status |
|---|---|
| Weight repack timing | At set_tensor (one-time at model load) |
| lm-head on DSP | Q4_K -> Q4_0 tiled repack, ~214 MB session-resident |
| Op fusion scope | PARITY - all 5 fusion types; VTCM guard for MUL_MAT_ADD |
| Graph cache | Content-hash based (98.8% hit rate) |
| Graph reorder | Implemented, no measurable PP benefit for gemma4 PP=58 |
| Dual path | REMOVED - single shared htp/ kernel path |
| VTCM lifetime | Session-lifetime (matches Qualcomm pattern) |
| dsp_cache_mode | Mode 5 (bit0 + bit2); bit3 selective bulk flush |
| ion_sync_mode | Mode 1 (DMA_BUF_IOCTL_SYNC, optimal) |
| mm_params_cache | Enabled (caches VTCM layout by weight ptr + ne11) |
| DSP debug logging | Removed (-DNDEBUG skel build) |
| BF16 offload | Supported (stored as F16 in repack buffer) |
| v75/8Gen3 thread clamp | Clamps to max_hw_threads - 2 via IDL out-param |
| LTO | Rejected (net regression -9% to -14% PP) |
| flash-attn-ops.c -O3 | Blocked (LLVM 19.0.07 PromoteFloatResult bug) |
18. Related Documents
- ion-mempool-vs-perbuffer-analysis-20260713.md - JZ ggml-hexagon vs Qualcomm ggml-hexagon: Architecture Analysis
- warmup-ab-test-and-analysis-20260713.md - FastRPC/ION warmup A/B test; batch-level pipelining analysis of QCOM's dspqueue
|
the corresponding & latest doc: https://github.com/zhouwg/ggml-hexagon/blob/self-build-jz/docs/backend/jz-ggml-hexagon/algotype29-perf-analysis-en-20260711.md |
|
got another big progress: PP reach 378 at the first time. I hope the latest codes can be submitted to Github before 07-15-2025. |
|
PP in JZ ggml-hexagon: 
PP in Qualcomm ggml-hexagon: |
|
PP peak exceeds 400 tokens/s for the first time after adapting to upstream PR ggml-org#25762 (commit b2dd28a): runs on 2026-07-18 06:17 (build v0.99.3.7, device&command: Snapdragon 8 Elite v79, OnePlus 13, /data/local/tmp/llama-completion -ngl 99 -t 6 -n 256 --ctx-size 8192 --ubatch-size 64 --poll 1000 --no-warmup --no-mmap -fa on -st -no-cnv -m /sdcard/gemma-4-E2B-it-Q4_0.gguf -p "Hello, good morning, you are a powerful domain expert and know many things, now pls help to introduce the movie Once Upon a Time in America briefly, pls pay attention short then 1000 words\n"). 
|
|
PP peak exceeds 520 tokens/s for the first time(much more faster than PP in Qualcomm's ggml-hexagon) and TG is match TG 26 in Qualcomm's ggml-hexagon: build v0.99.3.9-dev, device: Snapdragon 8 Elite(aka 8Gen4), HTP arch v79, Vendor OnePlus 13, command: /data/local/tmp/llama-completion -ngl 99 -t 6 -n 256 --ctx-size 8192 --ubatch-size 64 --poll 1000 --no-warmup --no-mmap -fa on -st -no-cnv -m /sdcard/gemma-4-E2B-it-Q4_0.gguf -p "Hello, good morning, you are a powerful domain expert and know many things, now pls help to introduce the movie Once Upon a Time in America briefly, pls pay attention short then 1000 words\n").
PP&TG in Qualcomm ggml-hexagon: 
(1.2) llama-bench PP&TG in JZ's ggml-hexagon: 
PP&TG in Qualcomm's ggml-hexagon: 
(2.1)LLM inference test 
(2.2)llama-bench PP&TG in Qualcomm ggml-hexagon: 
PP&TG in JZ ggml-hexagon: 
detailed benchmark data can be found at https://github.com/zhouwg/ggml-hexagon/blob/self-build-jz/docs/backend/jz-ggml-hexagon/ion-mempool-vs-perbuffer-analysis-20260713.md. |
|
@ggml-org/ggml-hexagon, @max-krasnyansky, @lhez, I know your time is valuable and sorry to interrupt you. After resolving the stability issues on Snapdragon 8 Gen 3, JZ ggml-hexagon outperforms Qualcomm’s official ggml-hexagon in both PP and TG performance on Snapdragon 8 Gen 3 and Snapdragon 8 Elite (8 Gen 4). JZ's ggml-hexagon has some advantages over Qualcomm's official ggml-hexagon, even though both implementations share the same kernels provided by your excellent kernels. It would mark great collaboration between jz's ggml-hexagon and Qualcomm's ggml-hexagon if this updated PR ggml-org#12326(aka new version of PR-12326) could get accepted into the upstream llama.cpp community:no conflicts with official ggml-hexagon,clean merge into master,provides an alternative for users. All optimization work is implemented in ggml-hexagon-jz.cpp and entry.c, as I aim to avoid modifying core code within ggml and llama.cpp, along with your well-written kernel implementations. BTW, one of the key optimizations relies heavily on the unique architectural design of JZ’s ggml-hexagon: native/pure Fast RPC + single ION shared mempool, not your dspqueue + multiple per-buffer ION shared buffer. Please check the updated document:JZ ggml-hexagon vs Qualcomm ggml-hexagon: Architecture Analysis Please check the updated document: Why Qualcomm ggml-hexagon Cannot Offload lm-head to the DSP |
|
@ggerganov, @ngxson, I know your time is precious and sorry to interrupt you. I have prepared an updated PR ggml-org#12326 (pls refer to branch self-build-jz in this project). Its PP performance far exceeds Qualcomm’s ggml-hexagon, while TG performance is on par with Qualcomm’s implementation. May I submit this PR accordingly? @JohannesGaessler, I know your time is precious and sorry to interrupt you. Thanks for your kind reply on July 23 2025. Now I have prepared an updated PR ggml-org#12326. Could I submit a new PR(new version of PR-12326) accordingly? @ngxson, My review comment regarding your PR ggml-org#26012:
@ngxson, Sorry to interrupt you. Can you help me create a discussion thread in the upstream llama.cpp community? pls refer to PP & TG Performance Comparison: Qualcomm's ggml-hexagon vs JZ's ggml-hexagon, thanks so much! |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
llama.cpp for Qualcomm Hexagon NPU(aka ggml-hexagon)
Background
Android maintained its position as the leading mobile operating system worldwide in the fourth quarter of 2023 with a market share of 70.1 percent . Qualcomm is No.1 mobile SoC semiconductor company in our planet currently.
About Hexagon SDK
Each Qualcomm chip includes multiple Hexagon DSPs such as the compute DSP (cDSP), audio DSP (aDSP), and sensor DSP (SLPI -- Sensor Low Power Island). Each of these DSPs implement a specific Instruction Set Architecture (ISA) version. The compute DSP, which is intended for compute-intensive tasks such as image processing, computer vision, and camera streaming, also includes an instruction set extension for fixed-point vector operations called Hexagon Vector eXtensions (HVX).The following diagram provides an overview of the processing units within the cDSP and how they connect to the memory cache.
Compared to the host CPU, the DSP typically runs at a lower clock speed but provides more parallelism opportunities at the instruction level. This often makes the DSP a better alternative in terms of throughput and/or power consumption. As a result, it is preferable to offload as many large compute-intensive tasks as possible onto the DSP to reduce power consumption of the device and free up cycles on the CPU for additional features.
Hexagon SDK is a lightweight and low-level SDK provided by Qualcomm. developers and AI experts can operate cDSP hardware directly with Hexagon SDK.
Llama.cpp + Hexagon NPU
The llama.cpp Hexagon NPU backend(aka ggml-hexagon backend) is intended to support Qualcomm Hexagon NPU firstly, supported chipsets:
block-beta columns 1 block:llamacpp llamacpp["llama_cpp"] style llamacpp fill:#3c3,color:#000,stroke:#000 end block:ggml_backend ggml_backend["GGML backend subsystem"] style ggml_backend fill:#3c3,color:#000,stroke:#000 block:ggmlbackends ggml_cpu["ggml-cpu"] ggml_metal["ggml-metal"] ggml_sycl["ggml-sycl"] ggml_cuda["ggml-cuda"] ggml_hip["ggml-hip"] ggml_vulkan["ggml-vulkan"] ggml_cann["ggml-cann"] ggml_opencl["ggml-opencl"] ggml_hexagon["ggml-hexagon"] ggml_nnpa["ggml-nnpa"] ggml_ane["ggml-ane"] style ggml_cpu fill:#888,color:#000,stroke:#000 style ggml_metal fill:#888,color:#000,stroke:#000 style ggml_sycl fill:#888,color:#000,stroke:#000 style ggml_cuda fill:#888,color:#000,stroke:#000 style ggml_hip fill:#888,color:#000,stroke:#000 style ggml_vulkan fill:#888,color:#000,stroke:#000 style ggml_cann fill:#888,color:#000,stroke:#000 style ggml_opencl fill:#cc3,color:#000,stroke:#000 style ggml_hexagon fill:#cc3,color:#000,stroke:#000 style ggml_ane fill:#fff,color:#000,stroke:#f00,stroke-width:2,stroke-dasharray:5 style ggml_nnpa fill:#cc3,color:#000,stroke:#000 end end block:ggml_backendsubsystem ggml_backendsubsystem["GGML backend subsystem"] style ggml_backendsubsystem fill:#3c3,color:#000,stroke:#000 end block:group1：2 columns 2 block:ggml_tensor ggml_tensor["GGML tensor"] style ggml_tensor fill:#3c3,color:#000,stroke:#000 end block:ggml_cgraph ggml_cgraph["GGML cgraph"] style ggml_cgraph fill:#3c3,color:#000,stroke:#000 end end block:OS Windows Linux Android QNX end block:hardware_vendors Intel AMD Apple Nvidia Huawei Loongson Qualcomm IBM ggml_metal --> Apple ggml_cuda --> Nvidia ggml_hip --> AMD ggml_cann --> Huawei ggml_sycl --> Intel ggml_opencl --> Qualcomm ggml_hexagon --> Qualcomm ggml_ane --> Apple ggml_nnpa --> IBM end block:hardware_types CPU GPU NPU DSP end block:hardware_archs x86 arm risc loongson end%%{init: {"flowchart": {"htmlLabels": false, 'nodeSpacing': 30, 'rankSpacing': 30}} }%% flowchart LR classDef EXIST fill:#888,color:#000,stroke:#000 classDef DONE fill:#3c3,color:#000,stroke:#000 classDef WIP fill:#cc3,color:#000,stroke:#000 classDef NEW fill:#fff,color:#000,stroke:#f00,stroke-width:2,stroke-dasharray:5 subgraph Legend direction LR EXIST:::EXIST ~~~ WIP:::WIP ~~~ DONE:::DONE ~~~ NEW:::NEW endNews
07/26/2026,10:29
maintain separate kernels in JZ‘s ggml-hexagon again(c220b84): historically, JZ and Qualcomm shared a single DSP kernel directory (htp/) for ops kernels, HVX/HMX headers, and common helpers. This ended with Qualcomm's large PR hexagon: further improved pipeline of the core bits (L2, DMA, MM, FA) ggml-org/llama.cpp#26049 (hexagon: further improved pipeline of the core bits (L2, DMA, MM, FA) ggml-org/llama.cpp#26049, merge commit 0a50d99): after merging upstream master with this PR, JZ's default inference test produced garbled output while Qualcomm's remained normal. The root cause is that this PR moved part of the cache maintenance logic into operator implementations, making JZ's cache subsystem incompatible and causing the garbled output. To avoid chasing each upstream htp/ change, JZ forked htp/ into a new kernels/ directory pinned at baseline commit 2be3826 (where PP/TG fully exceeded Qualcomm and inference output was correct). JZ now maintains kernels/ independently; Qualcomm continues using htp/ which tracks upstream master. Selected stable upstream htp/ improvements are ported into kernels/ manually.
07/24/2026,12:30
add document:how-to-build-ggmlhexagon.md
refine document:JZ ggml-hexagon vs Qualcomm ggml-hexagon: Architecture Analysis
refine document:Why Qualcomm ggml-hexagon Cannot Offload lm-head to the DSP
07/23/2026,20:31
After resolving the stability issues on Snapdragon 8 Gen 3, JZ ggml-hexagon outperforms Qualcomm’s official ggml-hexagon in both PP and TG performance on Snapdragon 8 Gen 3 and Snapdragon 8 Elite (8 Gen 4), details can be found at https://github.com/zhouwg/ggml-hexagon/blob/self-build-jz/docs/backend/jz-ggml-hexagon/ion-mempool-vs-perbuffer-analysis-20260713.md
07/22/2026,23:47
Update docs for first-time readers
07/21/2026,20:30
On Snapdragon 8Elite(aka 8Gen4), PP and TG performance surpasses Qualcomm’s official ggml-hexagon. Latest code is available on GitHub.
07/14/2026,14:39
submit latest codes and docs to Github: fully latest codes can be found at the default branch "self-build-jz" https://github.com/zhouwg/ggml-hexagon, fully latest docs can be found at https://github.com/zhouwg/ggml-hexagon/tree/self-build-jz/docs/backend/jz-ggml-hexagon.
07/11/2026,08:25
remove dual path and trim the project, prepare for updating latest codes. now only a small number of files in the upstream master branch will be affected by JZ's ggml-hexagon modifications(PR-12326's new version: JZ's ggml-hexagon-jz.cpp & entry.c + Qualcomm's kernels):
ggml/src/ggml-hexagon/ggml-hexagon-jz.cpp
ggml/src/ggml-hexagon/CMakeLists.txt
ggml/src/ggml-hexagon/htp/dsp-ctx.h
ggml/src/ggml-hexagon/htp/entry.c
ggml/src/ggml-hexagon/htp/ggml_dsp.idl
ggml/src/ggml-hexagon/htp/Makefile
scripts/build-run-ggmlhexagon-android.sh
the reasons of ggml_dsp.idl is more simple/concise than Qualcomm's ggml_htp idl are:
from my perspective, to use an imperfect analogy: dspqueue is similar to the QNN SDK while native Fast RPC is comparable to the Hexagon SDK. of course, my understanding might not be correct.
07/10/2026,06:54
07/08/2026,23:28
here are the ideas:
under the path mulmat_algotype=29, the DSP side reuses all operators/kernels from Qualcomm’s ggml-hexagon, added a thin compatibility/translation layer for Qualcomm operators/kernels in entry.c on the DSP side and ggml-hexagon.cpp on the AP side, ggml-hexagon.cpp keeps the main data structures and core ideas from PR-12326. JZ's ggml-hexagon uses only native/pure FastRPC, without complicated dspqueue, this makes the implementation/solution more simpler and cleaner.
I hope the upstream community will accept my long-term work/effort and lift my block if JZ's ggml-hexagon gets better PP and TG performance than Qualcomm’s ggml-hexagon, this will be a great technical collaboration between me(independent contributor, aka IC) and Qualcomm’s tech expert(I provide a concise ggml-hexagon.cpp and entry.c/main.c, Qualcomm's tech expert provide highly-performance kernels).
07/02/2026,22:17
07/01/2026,21:04
07/01/2026,11:56
suddenly found there is official hexagon-sdk in https://github.com/snapdragon-toolchain/hexagon-sdk/tree/main, upgrade HEXAGON_SDK from 6.2.0.1 to 6.6.0.0 and upgrade HEXAGON_TOOLS from 8.8.06 to 19.0.07 accordingly.
got another good progress, now PP can reach 35-37 tokens per second on 8Elite phone although "consistency and stability of LLM inference test results" still exists.
06/20/2026,00:05
06/18/2026,22:39
06/01/2026
06/27/2025
performance of fp32 4096x4096 mulmat on cDSP:
before 05/27/2025: about 28 seconds
relaunched the dev activity of project ggml-hexagon since 05/27/2025
06/09/2025: about 7-8 seconds
06/25/2025: about 6-8 seconds
06/27/2025: about 3.4-4.2seconds
06/25/2025
06/09/2025
06/03/2025
05/10/2025
04/24/2025
04/17/2025
04/12/2025(April/12/2025)
04/09/2025(April/09/2025)
04/08/2025(April/08/2025)
04/07/2025(April/07/2025)
04/06/2025(April/06/2025)
04/05/2025
04/02/2025
03/31/2025
03/29/2025
03/25/2025-03/27/2025
03/19/2025---03/24/2025
03/12/2025---03/19/2025
implement a concise implementation of the special approach:"mapping the entire ggml cgraph to a single QNN graph"
01/29/2025---03/11/2025
05/28/2024---06/15/2024
04/26/2024
04/24/2024
03/29/2024---04/24/2024
03/25/2024
03/05/2024---03/16/2024
Hardware
Qualcomm Hexagon NPU
Verified devices
DataType Supports
Android
How to build ggml‐hexagon source code for Android and verify ggml-hexagon backend on Snapdragon based phone
Ubuntu 20.04,22.04,26.04 is validated and recommended as host machine(other Linux distributions might be also ok).
utilize build-run-android.sh to download Android NDK and a customized minimal Hexagon SDK automatically, the fully Qualcomm Hexagon SDK must be obtained with a Qualcomm Developer Account and cannot be downloaded automatically in the build script.
you will need an Android smartphone with adb-connected running on one of below Qualcomm SoCs:
SM8550 (Snapdragon 8 Gen 2)
SM8650 (Snapdragon 8 Gen 3，validated and verified)
SM8750-AB (Snapdragon 8 Elite)(aka Snapdragon 8 Gen 4, strongly recommend, validated and verified)
SM8850 (Snapdragon 8 Elite 5)(aka Snapdragon 8 Gen 5)
we can find that this backend works fine as expected from the log output of "adb logcat | grep ggml-hexagon". for programmers, we can use "adb logcat | grep ggml-hexagon" to help troubleshooting issues on AP side and "adb logcat | grep CDSP0" to help troubleshooting issues on DSP side.
How to do performance comparison of PP and TG between Qualcomm's ggml-hexagon and JZ's ggml-hexagon
for fair performance comparison, the same "running_params=" -ngl 99 -t 6 -n 256 --ctx-size 8192 --ubatch-size 64 --poll 1000 --no-warmup --no-mmap -fa on" " and the same prompt and same LLM model file and same 8Elite phone would be used in both inference test.
We can run automated AB tests on Snapdragon 8 Elite following the recommended steps below:
log_abtest_$(date +%Y%m%d-%H%M%S).txtWe can run non-automated AB tests on Snapdragon 8 Elite following the recommended steps below:
Known Issues & Limitations
Q&A
Feel free to submit issue reports here, even though the JZ ggml-hexagon implementation cannot be merged upstream.
Acknowledgement
[07/01/2026, July 1 2026] Thanks for Trae + GLM-5.2's great help, Qualcomm's ggml-hexagon also give me many help. GLM-5.2 and I co-create the ION-based op-batch solution after many many hours(I don't want to use Qualcomm's dspqueue in JZ's ggml-hexagon because ION-based op-batch without dspqueue is one of the highlights in JZ's ggml-hexagon), GLM-5.2 is one of the co-authors of the JZ's ggml-hexagon since June-03-2026.
[07/06/2026, July 6 2026] Thanks for Trae + DeepSeek-V4-Pro's great help. DeepSeek-V4-Pro did a good job in performance optimization.
[07/09/2026, July 9 2026] Thanks for Trae + MiniMax-M3's breakthrough help and profound insights, the PP performance has been boosted from around 180 to over 300 based on Qualcomm's new operators/kernels.
[07/09/2026, July 9 2026] GLM-5.2, DeepSeek-V4-Pro, MiniMax-M3 are both China's top AI Coding Models, sincerely thanks for the original authors of them.
[07/14/2026, July 14 2026] Kimi-K2.7 also made solid contribution on 07/13/2026 although Kimi-K2.7-Code joined this project on late evening 07-13-2026.
[07/14/2026, July 14 2026] There would be no jz-ggml-hexagon without the excellent operators/kernels implementation provided by Qualcomm.
[07/17/2026, July 17 2026] Kimi-K3 joined this project on late evening 07-17-2026.
[07/21/2026, July 21 2026] I learned a lot from the open-source community when I was a young programmer. Sincere thanks to every contributor in the wonderful open-source community ------ especially the original authors and contributors of Linux, Android, FFmpeg, ggml(another outstanding open-source project from the EU which is exactly similar to FFmpeg), llama.cpp and many other outstanding projects. I hope this project can bring modest value to the llama.cpp community, even though PR PR: Refine ggml-hexagon backend(Qualcomm Hexagon NPU backend) for latest ggml,whisper.cpp,llama.cpp ggml-org/llama.cpp#12326 has faced highly difficulties gaining acceptance and approval within the llama.cpp community.
AI usage disclosure
The core ideas originate from my fully original & fully handwritten PR-12326. Starting in June 2026, I have used AI coding agents to assist with brainstorming new ideas, writing code snippets and English tech docs (I provide questions and scope), and creating automated tests. I have already checked, tested, and fully understand all the code(excluding quantization type conversion) in ggml-hexagon-jz.cpp and entry.c. What matters most is that all decisions, directions and adjustments are made by JZ.
All reactions