Repository navigation
Replies: 24 comments 30 replies
|
|
|
JZ ggml-hexagon: Structure and Optimization Analysis (2026-07-22)Author
1. BackgroundBoth the JZ and Qualcomm versions of the ggml-hexagon backend route through Qualcomm's JZ ggml-hexagon is built on two fundamental architectural choices that originated from the upstream PR #12326 (March 2025):
Based on these two choices, JZ ggml-hexagon sidesteps both the These are deliberate design choices, not limitations. The theoretical basis is that LLM inference is inherently serial (autoregressive TG + serially dependent subgraphs), which limits the benefit of async pipelining. 2. Current Benchmark (2026-07-22)2.1 Test Conditions
2.2 PP and TG Comparison (5 runs each, same device, same model, same day)Table 1: JZ vs Qualcomm PP/TG (5 runs, 2026-07-22)
JZ exceeds Qualcomm on both PP and TG:
Both backends run the same HMX kernels from 2.3 JZ-side RPC stats (run 1)Relevant FilesTable 2: Relevant files
3. FastRPC Call Pattern3.1 JZ Version
3.2 Qualcomm Version
3.3 Performance ImpactJZ's synchronous FastRPC architecture is a deliberate design choice. The pure FastRPC/ION transport overhead is ~89 us/call (warmup probe), which is negligible against the ~36 ms/token DSP execution time. With lm-head offloaded to DSP and 4. Op Fusion4.1 JZ Version (Phase 2.5)Table 3: JZ op fusion types (Phase 2.5)
4.2 VTCM budget check for MUL_MAT_ADD fusionThe MUL_MAT + ADD fusion checks the VTCM budget before firing, mirroring Qualcomm's guard (ggml-hexagon.cpp:3595): const size_t vtcm_budget = (size_t)ctx->socinfo.vtcm_size_in_mb * 1024 * 1024;
if ((size_t)kparams->vtcm_size > vtcm_budget) {
return false; // skip fusion, let MUL_MAT and ADD run separately
}4.3 Qualcomm Version (
|
| Fusion Type | Supported | Notes |
|---|---|---|
| RMS_NORM + MUL -> HTP_OP_RMS_NORM_MUL | Yes | Uses ggml_can_fuse |
| MUL_MAT + ADD -> HTP_OP_MUL_MAT_ADD | Yes | Uses ggml_can_fuse |
| MUL_MAT QKV merge -> HTP_OP_MUL_MAT_QKV | Yes | 3 mul_mat merged into 1, reordered to KVQ |
| MUL_MAT FFN merge -> HTP_OP_MUL_MAT_FFN | Yes | gate + up merged into 1 |
| Graph reorder | Yes | Stacks MUL_MATs with same src1 for VTCM reuse |
Fusion scope is at parity (all 5 fusion types). Graph reorder is implemented in JZ and runtime-configurable; for gemma4 PP=58 tokens on 8 Elite v79 it is a no-op (within noise), but kept on by default for future-proofing.
5. Graph Cache
The cache uses a content hash (FNV-1a over each node's {op, ne[4], nb[4], src[0..2] ptr, data ptr}) instead of the dead cgraph->uid key:
const uint64_t content_hash = compute_content_hash();
auto it = ctx->cgraph_cache.find(content_hash);
if (it != ctx->cgraph_cache.end() &&
it->second.n_nodes == cgraph->n_nodes &&
it->second.hex_ops.size() > 0) {
cache_hit = true;
ctx->cgraph_cache_hits++;
}On a hit, the cache skips Phase 1 (tensor dedup), Phase 2 (op descriptor build), and Phase 2.5 (op fusion) entirely. Current hit rate is 98.8% (253 hits / 256 calls). The 3 misses correspond to the first fill of 3 unique graph structures. After warmup, every subsequent token hits cache.
6. mm_params_cache (JZ only, enabled)
JZ caches precomputed htp_mm_kernel_params by a composite key (src0 tensor pointer XOR weight data pointer XOR ne11):
const uintptr_t cache_key = (uintptr_t) src0 ^ (uintptr_t) src0->data ^ ((uintptr_t) ne11 << 32);
auto it = ctx->mm_params_cache.find(cache_key);
if (it != ctx->mm_params_cache.end()) {
*kparams = it->second;
return;
}This skips the multi-hundred-microsecond thread/chunk search in htp_mm_hvx_vtcm_layout_build / htp_mm_hmx_vtcm_layout_build for repeated MUL_MAT calls with the same weight tensor. For TG (where ne11=1 for every token), the cache hits after the first token. The cached kparams are valid for the session lifetime because weights are static (never modified after model load).
Qualcomm's ggml_hexagon_precompute_matmul_params performs the same VTCM layout computation but does not cache the result across calls.
7. Weight Repack
JZ implements a repack buffer type with is_host=false:
static bool ggml_backend_hexagon_repack_buffer_is_host(ggml_backend_buffer_type_t buft) {
return false; // forces GGML core to call set_tensor
}When the model loader encounters quantized weights (Q4_0, Q4_1, Q8_0, IQ4_NL, MXFP4), the supports_op gate in MUL_MAT ensures they are allocated in the repack buffer type. Because is_host=false, GGML core routes data through set_tensor, which performs the in-place tile repack at model load time (one-time cost).
lm-head DSP offload
The lm-head (262144x1536, Q4_K) is offloaded to DSP. The ne[1] > 32768 rejection for large quantized weights was removed, and a new repack_q4k_as_q4_0_tiled_to_buf converter (ggml-hexagon-jz.cpp:4162) stores the lm-head as Q4_0 tiled layout in the single ION pool (~214 MB, session-resident, streamed from DDR once per token). This eliminates the ~30 ms/token CPU lm-head matvec that was the single largest TG cost. Qualcomm's per-buffer ION design makes a 214 MB per-buffer map/unmap per session prohibitive, so their lm-head stays on CPU.
8. Cache Coherency Management
8.1 JZ Version
- AP side: Configurable via
ion_sync_mode0= both (DC CVAC + ion_sync, default)1= ion_sync only (DMA_BUF_IOCTL_SYNC, driver-level) - optimal2= DC CVAC only (manual cache line management)
- DSP side: Configurable via
dsp_cache_modebitmask (default = 5)- bit 0: first-touch weight invalidation (exact sorted pointer array
g_weight_inval_ptrs,WEIGHT_INVAL_MAX_PTRS=4096in entry.c; repack weights written once at model load, single first-touch dcinva covers the whole session) - bit 1: skip dcinva for prior dst (off - deferred-flush pattern is unsafe to combine with bit2 for anything above a single cacheline)
- bit 2: bulk dst flush at batch end (collect/sort/merge dst ranges)
- bit 3: selective bulk flush (skip batch-end flush for intermediates still consumed by later ops in the same batch;
g_tensor_last_use_opin entry.c)
- bit 0: first-touch weight invalidation (exact sorted pointer array
8.2 dsp_cache_mode=5 default
Mode 5 (bit0 + bit2) is the default. The first-touch weight invalidation tracker uses an exact sorted pointer array instead of a hash bitmap (whose address collisions caused garble in earlier iterations). A two-pass defense (DSP-side unmark on dst write + AP-side ever-dst set) closes the cross-graph stale-read window. This is what made the session-resident repacked lm-head practical: per-token weight traffic is ~1.9 GB, and re-invalidating it every token would cost ~9.2 ms/token of DSP-side dcinva sweeping; with first-touch, that cost is paid once at session start.
8.3 ion_sync_mode=1 is optimal
Mode 1 uses the kernel's DMA_BUF_IOCTL_SYNC which is faster than userspace DC CVAC/CIVAC for large ranges. Mode 0 (double cache maintenance) drops PP significantly; mode 2 (manual only) is similar to mode 0.
8.4 Qualcomm Version
dspqueue driver automatic management via DSPQUEUE_BUFFER_FLAG_FLUSH_SENDER | DSPQUEUE_BUFFER_FLAG_INVALIDATE_RECIPIENT. The driver applies uniform per-batch flush/invalidate flags to every buffer - there is no per-role differentiation. JZ's user-space management distinguishes weight tensors (flags=2, written once at load) from activations, so bit 0 can eliminate their per-token re-invalidation. This policy flexibility is an advantage the closed driver cannot replicate without per-role buffer semantics.
9. Session Consistency Gate
JZ mirrors Qualcomm's ggml_hexagon_supported_buffer check to prevent the scheduler from mixing tensors across different Hexagon sessions or non-Hexagon buffers:
static bool ggmlhexagon_tensor_buffer_is_owned_by(ggml_backend_dev_t dev, const struct ggml_tensor * t) {
if (!t || !t->buffer) return true; // neutral
// Accept if buffer is hexagon (main or repack) on this device
// Reject if hexagon on different device or non-hexagon
}10. VTCM Session-Lifetime
VTCM is acquired once per session, not per batch. ggml_dsp_open does one HAP_compute_res_acquire_cached and ggml_dsp_close does one HAP_compute_res_release_cached. VTCM is held continuously for the session. This matches the Qualcomm HTP pattern (vtcm_acquire / vtcm_release only fire when transitioning between "active processing" and "forced release").
// ggml_dsp_open (after HAP_compute_res_acquire succeeds)
dsp_vtcm_acquire(); // once per session, sets vtcm_valid=1
// ggml_dsp_close (before HAP_compute_res_release)
dsp_vtcm_release(); // once per session, sets vtcm_valid=0
// execute_batch: no per-batch acquire/release callsTrade-off: lose the ability to respond to a forced-release callback from another session. For single-session use (the current deployment) this is a non-issue.
11. Dual Path Removal
JZ routes all ops through the shared htp/ execute_op path, exactly like Qualcomm. The entire algotype=32 path (JZ's own self-built kernel dispatch in kernels/mulmat.c, kernels/flash_attn.c, etc.) and the ggml-dsp port (kernels/ggml-dsp.c 9946 lines, kernels/ggml-dsp.h 2256 lines) are deleted - 24298 lines removed. JZ's four files are now merged into htp/ alongside Qualcomm's kernels:
entry.c- FastRPC entry point, cache management,dsptensor<->htp_tensorbridge,execute_opdispatchdsp-ctx.h-struct dsp_context,dsptensor,hex_tensor_desc,hex_op_desc,hex_batch_hdrggml_dsp.idl- FastRPC interfaceMakefile- buildslibggmldsp-skel.sofromentry.c+ allhtp/*.csources
The mulmat_algotype config knob is removed entirely. The "algotype=29" label survives only as a historical comment in ggml-hexagon.cfg and in this document filename.
dsptensor is retained as JZ's AP-side tensor descriptor format (single ION offset addressing), but it is now only a thin wrapper. The dsptensor_to_htp_tensor bridge in entry.c converts it to the shared htp_tensor (with bi=0) before calling execute_op.
12. Tensor Descriptor Data Structure
12.1 JZ Version
JZ uses three descriptor types at different stages:
AP-side: hex_tensor_desc (single ION offset addressing).
typedef struct hex_tensor_desc {
int32_t type;
int32_t ne[4];
int32_t nb[4];
int32_t op_params[16];
uint32_t flags; // 0=ION, 1=mirrored, 2=weight(skip flush)
uint32_t data_offset;
uint32_t data_len;
} hex_tensor_desc;DSP-side: dsptensor (defined in dsp-ctx.h)
struct dsptensor {
int32_t type;
int32_t ne[4];
int32_t nb[4];
int32_t op; // op code embedded in tensor descriptor
int32_t op_params[16]; // op-specific params embedded in tensor descriptor
int32_t flags;
void * data; // direct pointer (DSP address space is 32-bit)
int data_len;
};12.2 Qualcomm Version htp_tensor
struct htp_tensor {
uint32_t data;
uint32_t size;
uint32_t flags;
uint16_t type;
uint16_t bi; // buffer index
uint32_t ne[4];
uint32_t nb[4];
};12.3 Difference
Table 5: Tensor descriptor comparison
| Aspect | dsptensor (JZ) |
htp_tensor (Qualcomm) |
|---|---|---|
| Op metadata | Embedded (op + op_params[16]) |
Separated into htp_op_desc |
| Data addressing | Direct void * pointer |
bi (buffer index) + uint32_t offset |
| Multi-buffer support | No (single ION pool) | Yes (via bi indexing into htp_buf_desc[]) |
| Type width | int32_t |
uint16_t/uint32_t (more compact) |
The bi field is the key differentiator for multi-buffer support. JZ always passes bi=0 (single ION pool); Qualcomm uses it to index into htp_buf_desc[].
13. Upstream Merge Adaptations
13.1 Unary precompute port
Upstream commit fb30ba9a6 introduced a new op_unary pipeline that requires host-precomputed htp_unary_kernel_params (n_threads, col_tile, vtcm_size, etc.). JZ ported ggml_hexagon_precompute_unary_params from ggml-hexagon.cpp, adapted to use ctx->n_threads and ctx->socinfo.vtcm_size. HTP_OP_TRI is routed to op_unary() in entry.c to match upstream htp/main.c.
13.2 VTCM layout API
Upstream commit 81ff7abe5 brought in Qualcomm's new VTCM layout API:
htp_mm_hvx_get_vtcm_sizes->htp_mm_hvx_vtcm_layout_build+struct htp_mm_hvx_vtcm_layouthtp_mm_hvx_id_get_vtcm_sizes->htp_mm_hmx_vtcm_layout_build+struct htp_mm_hmx_vtcm_layoutbroadcast_rk2/rk3/rv2/rv3fields moved fromu.hvxunion member tohtp_fa_kernel_paramsstruct top level
JZ adapted via adapter functions in ggml-hexagon-jz.cpp and entry.c that translate old API calls to the new layout-build API.
13.3 Build system unification
The unified CMakeLists.txt is based on QCOM's version (minimal diff to upstream) with a single addition:
option(GGML_HEXAGON_JZ "Use JZ's AP implementation" OFF)GGML_HEXAGON_JZ=OFF(default): exactly QCOM's upstream behavior. Usesggml-hexagon.cpp, builds DSP skels viaExternalProject_Addfor v73/v75/v79/v81, linkshtp_ifacestub.GGML_HEXAGON_JZ=ON: usesggml-hexagon-jz.cpp, builds a single DSP skel viamake -C htp/(JZ's Makefile), linkscdsprpc, setsHEXAGON_DEFAULT_LIB_SEARCH_PATH, copiesggml-hexagon.cfg.
14. Compiler Optimization
14.1 AP-side (PARITY)
Both backends compile AP-side code with the same ARMv8.7-A + dotprod + fp16 + i8mm flags:
set(OPT_FLAG " -O3 -march=armv8.7-a+dotprod+fp16+i8mm -mcpu=cortex-x1 -mtune=cortex-x1 -ffp-model=fast -fno-finite-math-only")14.2 DSP-side (JZ-only)
JZ's htp/Makefile uses -O3 -ffast-math -fno-vectorize (no LTO) with -DNDEBUG for the DSP skel. Qualcomm's htp/cmake-toolchain.cmake uses -O2 -flto -fvectorize. A 3-row sweep confirmed LTO is a net regression for JZ's codebase (-9% to -14% PP); -O3 no-LTO is empirically the best choice.
14.3 flash-attn-ops.c -O2 workaround
flash-attn-ops.c is forced to -O2 because at -O3 Hexagon LLVM 19.0.07 emits a PromoteFloatResult fatal error on f16 = freeze. A 10-flag sweep confirmed no combination of -fno-X / -mllvm -disable-X flags can break the workaround. The bug is in the Hexagon backend (HexagonDAGToDAGISel -> PromoteFloatResult), not in any front-end pass. Needs an LLVM backend patch.
15. Profiler Infrastructure
15.1 AP-side profiler
Tracks cumulative time for phases p1, p2, p2.5, p3, p4, p4.5, p5, p6, p6.5, p7, p7.5, p8 per graph_compute_batch call. Breaks down Phase 7 into rpc_setup + dsp_exec + civac. Computes min/p50/p95/max histograms for the last 1024 calls. Reports cgraph cache hit/miss counts.
15.2 longtail profiler
A LOG_ALWAYS longtail probe inside the FastRPC dispatch path logs the op composition of any batch call whose dsp_exec exceeds 5 ms. Throttled to one log per 100 ms wall-clock. Probe code is preserved in source inside #if 0 ... #endif for future re-activation; runtime cost is zero.
15.3 mul_mat coverage tracer
Per-batch counters in hexagon_op_exec_stats_t: n_mul_mat_total, n_hmx_used, n_fused_qkv, n_fused_ffn, n_fused_mul_mat_add.
15.4 DSP-side profiler (gated off by default)
entry.c includes a per-op timing profiler that records min/max/avg execution time per op type, dumped via dump_op_prof. Wrapped in #if HEX_OP_PROF ... #endif with HEX_OP_PROF default to 0 (off). Restore by passing -DHEX_OP_PROF=1 to make.
16. v75/8Gen3 Thread Clamp
On Snapdragon 8 Gen 3 (v75), the code deadlocks at the default thread_counts=6 due to cDSP hardware-thread oversubscription. An op needs thread_counts+1 co-resident QuRT threads (FastRPC main + N-1 work-queue workers + 1 hmx_queue thread); v75 has 6 hardware threads (v79 has 8), so N=6 requires 7 > 6 and the unscheduled worker never decrements the task barrier.
ggml_dsp_setclocks clamps to max_hw_threads - 2 and reports the effective value through a rout int32 real_thread_counts IDL out-param; AP mirrors it into ctx->n_threads so the precomputed kparams->n_threads matches the DSP work-queue. On v75 this clamps to 4; on v79 it stays at 6.
17. Summary
Table 6: Performance summary (2026-07-22, 5-run mean)
| Backend | PP (tok/s) | TG (tok/s) |
|---|---|---|
| JZ | 686.46 | 26.91 |
| QCOM | 435.14 | 24.91 |
| JZ advantage | 1.58x | 1.08x |
JZ exceeds Qualcomm on both PP and TG. The three enabling changes:
- lm-head offloaded to DSP - Q4_K stored as Q4_0 tiled repack (~214 MB, session-resident in the single ION pool). Eliminated the ~30 ms/token CPU lm-head matvec. The single ION pool turns this into a one-time repack; Qualcomm's per-buffer ION design makes a 214 MB per-buffer map/unmap per session prohibitive.
- dsp_cache_mode=5 (bit0 + bit2) - First-touch weight invalidation via exact sorted pointer array + two-pass defense. Eliminates ~9.2 ms/token of redundant weight re-invalidation. bit3 (selective bulk flush) also added.
- DSP-side debug logging removed - Skel built with
-DNDEBUG; FARF debug paths compiled out.
Also: BF16 added to offloaded MUL_MAT types (stored as F16 in repack buffer, reusing F16 DSP kernels).
Table 7: Optimization status
| Item | Status |
|---|---|
| Weight repack timing | At set_tensor (one-time at model load) |
| lm-head on DSP | Q4_K -> Q4_0 tiled repack, ~214 MB session-resident |
| Op fusion scope | PARITY - all 5 fusion types; VTCM guard for MUL_MAT_ADD |
| Graph cache | Content-hash based (98.8% hit rate) |
| Graph reorder | Implemented, no measurable PP benefit for gemma4 PP=58 |
| Dual path | REMOVED - single shared htp/ kernel path |
| VTCM lifetime | Session-lifetime (matches Qualcomm pattern) |
| dsp_cache_mode | Mode 5 (bit0 + bit2); bit3 selective bulk flush |
| ion_sync_mode | Mode 1 (DMA_BUF_IOCTL_SYNC, optimal) |
| mm_params_cache | Enabled (caches VTCM layout by weight ptr + ne11) |
| DSP debug logging | Removed (-DNDEBUG skel build) |
| BF16 offload | Supported (stored as F16 in repack buffer) |
| v75/8Gen3 thread clamp | Clamps to max_hw_threads - 2 via IDL out-param |
| LTO | Rejected (net regression -9% to -14% PP) |
| flash-attn-ops.c -O3 | Blocked (LLVM 19.0.07 PromoteFloatResult bug) |
18. Related Documents
- ion-mempool-vs-perbuffer-analysis-20260713.md - JZ ggml-hexagon vs Qualcomm ggml-hexagon: Architecture Analysis
- warmup-ab-test-and-analysis-20260713.md - FastRPC/ION warmup A/B test; batch-level pipelining analysis of QCOM's dspqueue
|
the corresponding & latest doc: https://github.com/zhouwg/ggml-hexagon/blob/self-build-jz/docs/backend/jz-ggml-hexagon/algotype29-perf-analysis-en-20260711.md |
|
got another big progress: PP reach 378 at the first time. I hope the latest codes can be submitted to Github before 07-15-2025. |
|
PP in JZ ggml-hexagon: 
PP in Qualcomm ggml-hexagon: |
|
PP peak exceeds 400 tokens/s for the first time after adapting to upstream PR ggml-org#25762 (commit b2dd28a): runs on 2026-07-18 06:17 (build v0.99.3.7, device&command: Snapdragon 8 Elite v79, OnePlus 13, /data/local/tmp/llama-completion -ngl 99 -t 6 -n 256 --ctx-size 8192 --ubatch-size 64 --poll 1000 --no-warmup --no-mmap -fa on -st -no-cnv -m /sdcard/gemma-4-E2B-it-Q4_0.gguf -p "Hello, good morning, you are a powerful domain expert and know many things, now pls help to introduce the movie Once Upon a Time in America briefly, pls pay attention short then 1000 words\n"). 
|
|
PP peak exceeds 520 tokens/s for the first time(much more faster than PP in Qualcomm's ggml-hexagon) and TG is match TG 26 in Qualcomm's ggml-hexagon: build v0.99.3.9-dev, device: Snapdragon 8 Elite(aka 8Gen4), HTP arch v79, Vendor OnePlus 13, command: /data/local/tmp/llama-completion -ngl 99 -t 6 -n 256 --ctx-size 8192 --ubatch-size 64 --poll 1000 --no-warmup --no-mmap -fa on -st -no-cnv -m /sdcard/gemma-4-E2B-it-Q4_0.gguf -p "Hello, good morning, you are a powerful domain expert and know many things, now pls help to introduce the movie Once Upon a Time in America briefly, pls pay attention short then 1000 words\n").
PP&TG in Qualcomm ggml-hexagon: 
(1.2) llama-bench PP&TG in JZ's ggml-hexagon: 
PP&TG in Qualcomm's ggml-hexagon: 
(2.1)LLM inference test 
(2.2)llama-bench PP&TG in Qualcomm ggml-hexagon: 
PP&TG in JZ ggml-hexagon: 
detailed benchmark data can be found at https://github.com/zhouwg/ggml-hexagon/blob/self-build-jz/docs/backend/jz-ggml-hexagon/ion-mempool-vs-perbuffer-analysis-20260713.md. |
|
@ggml-org/ggml-hexagon, @max-krasnyansky, @lhez, I know your time is valuable and sorry to interrupt you. After resolving the stability issues on Snapdragon 8 Gen 3, JZ ggml-hexagon outperforms Qualcomm’s official ggml-hexagon in both PP and TG performance on Snapdragon 8 Gen 3 and Snapdragon 8 Elite (8 Gen 4). JZ's ggml-hexagon has some advantages over Qualcomm's official ggml-hexagon, even though both implementations share the same kernels provided by your excellent kernels. It would mark great collaboration between jz's ggml-hexagon and Qualcomm's ggml-hexagon if this updated PR ggml-org#12326(aka new version of PR-12326) could get accepted into the upstream llama.cpp community:no conflicts with official ggml-hexagon,clean merge into master,provides an alternative for users. All optimization work is implemented in ggml-hexagon-jz.cpp and entry.c, as I aim to avoid modifying core code within ggml and llama.cpp, along with your well-written kernel implementations. BTW, one of the key optimizations relies heavily on the unique architectural design of JZ’s ggml-hexagon: native/pure Fast RPC + single ION shared mempool, not your dspqueue + multiple per-buffer ION shared buffer. Please check the updated document:JZ ggml-hexagon vs Qualcomm ggml-hexagon: Architecture Analysis Please check the updated document: Why Qualcomm ggml-hexagon Cannot Offload lm-head to the DSP |
|
@ggerganov, @ngxson, I know your time is precious and sorry to interrupt you. I have prepared an updated PR ggml-org#12326 (pls refer to branch self-build-jz in this project). Its PP performance far exceeds Qualcomm’s ggml-hexagon, while TG performance is on par with Qualcomm’s implementation. May I submit this PR accordingly? @JohannesGaessler, I know your time is precious and sorry to interrupt you. Thanks for your kind reply on July 23 2025. Now I have prepared an updated PR ggml-org#12326. Could I submit a new PR(new version of PR-12326) accordingly? @ngxson, My review comment regarding your PR ggml-org#26012:
@ngxson, Sorry to interrupt you. Can you help me create a discussion thread in the upstream llama.cpp community? pls refer to PP & TG Performance Comparison: Qualcomm's ggml-hexagon vs JZ's ggml-hexagon, thanks so much! |
|
Hi Jeff, I’ve been doing a separate SM8850/v81 investigation around your FastRPC/JZ work, and I think I now have enough device-side evidence to share a consolidated set of findings. Your PR ggml-org#27642 was particularly useful because it exposes the large-model crossover very clearly: FastRPC is not universally faster than dspqueue, and the Qwen3.5-9B (~5 GiB) case is actually -14.3% versus dspqueue. The PR also explicitly states that the >4 GiB VA limitation is known for v75/v79 and that v81 was not tested yet. That matches the direction of what I am seeing on SM8850/v81, but there are several additional measurements that may be useful for the upstream JZ/FastRPC design. Device: Snapdragon SM8850 / HTP v811. MTP is a confirmed algorithmic winOn Qwen 9B Q4_0:
An independent CPU-only reference produced the same correct output as both HTP baseline and HTP+MTP, and 24/88 draft tokens were actually rejected, so the speculative verification was not simply accepting everything. For the current GGML/JZ path, MTP is therefore the strongest validated throughput optimization I have measured. 2. CPU placement is highly workload-dependent
However, in the full llama-server pipeline:
So the same placement optimization becomes -30% in the production server path. This suggests that backend transitions, CPU-side sampling/server work and ownership/transfer costs matter more than the isolated lm_head kernel cost. I therefore consider CPU lm_head offload benchmark-specific rather than production-worthy. 3. IOMMU/remapping is NOT the steady-state decode bottleneckI initially observed a very large Results:
Also, the Actual FastRPC transport measurement showed only 29 round-trips in the analyzed run (p50 ~386 us, p95 ~1.33 ms). So I no longer consider per-token weight remapping or raw FastRPC round-trip count to be the main steady-state explanation. 4. Large-model memory capacity is still a first-class issueThe 9B model is ~5.07 GiB, so the DSP virtual-address/mempool regime is critical. Your PR ggml-org#27642 explicitly documents the 4 GiB VA limitation for v75/v79 and the resulting heap/mirror-copy fallback for the ~5 GiB Qwen3.5-9B case. It also explicitly marks v81 as not tested. That makes SM8850/v81 a useful missing data point:
are the metrics I am now interested in measuring directly. 5. GPU/HTP hybrid is a real secondary optimizationOn the same 9B Q4_0 workload:
So the best hybrid point was GPU50/HTP50, +8.2% vs HTP-only. The relationship is clearly non-monotonic, so this looks more like a placement/resource-balance problem than simply “more GPU = more throughput”. 6. Runtime gap: QAIRT/QNN vs GGMLI now have both legs live on the same SM8850 device: GGML/Hexagon:
QAIRT/QNN:
So the live decode ratio is ~1.73x. The older 15.6 tok/s QAIRT result gives ~1.92x against the same GGML baseline. I do NOT attribute the whole gap to bit-width. A tensor-level analysis of the actual GGUF showed approximately:
and the W4A16 representation is ~13% smaller than the Q4_K_M representation in the measured 8B case. So format explains only part of the gap; the remaining difference must come from the execution path: graph planning, dispatch, packing, kernel implementation, cache behavior, etc. 7. QAIRT bundle target mismatchThe real QAIRT bundle I inspected contains:
The actual device is SM8850 / HTP v81. So the current 14.1 tok/s QAIRT result is running a v79-targeted bundle in compatibility mode on v81. A native v81 rebuild is therefore a clear experiment, although I am not assuming it will necessarily improve performance. 8. Important implementation finding from your current FastRPC/JZ codeThe current FastRPC backend is much more sophisticated than a “one RPC per GGML op” model. From
That suggests the next optimization target is not simply “remove FastRPC”. The more interesting JZ direction is: model-size/VA analysis 9. Proposed JZ-next directionI think the most useful next step is a measurement-driven planner on top of the existing FastRPC/JZ backend: |
|
Hi Jeff, I’ve been digging through your JZ implementation and the PR-to-upstream branch, and I think my SM8850/v81 measurements can add a useful complementary data point to your work. First, I want to say that your architectural direction is now very clear to me. In particular:
This is exactly the part of JZ that I am reverse-engineering now. Your Discussion #18 and the I also noticed an important correction to my own earlier interpretation. I initially described the GGML path as “one FastRPC round-trip per op”. Your current FastRPC implementation shows that this is too simplistic: the host builds a batch descriptor and submits it through a single That means the interesting question is not simply “how many RPC calls per token?”, but rather: This is the model I am now using. My own SM8850 / HTP v81 measurements are:
Qwen 9B Q4_0: 3 paired repetitions, CV 1.3% baseline / 0.6% MTP. Acceptance was 64/88 = 72.7%, with 24/88 actual rejections. I also checked correctness against an independent CPU-only reference. CPU, HTP baseline and HTP+MTP produced the same correct output on the validation prompt. So MTP is now a validated algorithmic optimization on top of the JZ/GGML path.
An isolated llama-bench experiment gave: But the full llama-server pipeline inverted this completely: So CPU lm_head is benchmark-only in my environment; it is not a production optimization.
I initially saw a very large The huge Actual FastRPC transport in the analyzed run was only 29 round-trips, p50 ~386 us, p95 ~1.33 ms. So I no longer consider continuous weight remapping or raw FastRPC transport count to be the steady-state decode bottleneck. The remaining large-model memory questions are instead: This seems particularly relevant to your PR ggml-org#27642.
Your PR ggml-org#27642 was very useful because it shows that FastRPC is not universally faster than dspqueue. In particular, the ~5 GiB Qwen3.5-9B result goes the other way. That suggests the large-model regime is qualitatively different, and I would like to compare the following on SM8850/v81: My device is SM8850 / HTP v81, so the v81 result is particularly interesting because PR ggml-org#27642 explicitly identifies v81 as a remaining test point.
I now have both sides live on the same device: So the live decode ratio is ~1.73x. An older QAIRT run was 15.6 tok/s (~1.92x). I do not think the gap is explained by bit-width alone. In my layer-level weight analysis: and W4A16 is only ~13% smaller in the measured 8B case. So the remaining gap has to come from some combination of:
The real W4A16 QAIRT bundle I inspected targets: while the device is SM8850 / v81. So the current 14.1 tok/s QAIRT number is already interesting, because the bundle is not natively targeted at v81. A native v81 bundle is one of my next experiments. My main proposal for the JZ direction is therefore not simply: but rather: I think your existing JZ implementation is already much closer to this architecture than I initially realized. The most useful thing for me now would be to expose or standardize the existing runtime measurements: Then I can correlate them with my SM8850 hardware counters and determine which part of the remaining GGML → QAIRT gap is actually recoverable. Thanks for putting the JZ architecture and measurements in the open. Your work makes this kind of device-side reverse engineering much easier to do rigorously. |
|
Hi Tagman45, Thank you for your interest in this project. As I am in a difficult situation at the moment (PR‑27642 was suddenly closed and I have been blocked from upstream since 2025-07-14), the FastRPC‑based ggml‑hexagon is maintained at https://github.com/zhouwg/ggml‑hexagon. I plan to move this project to a dedicated GitHub organization for the FastRPC-based ggml‑hexagon. This way, the FastRPC‑based ggml‑hexagon will no longer be my personal project. Any developers or experts interested in ggml‑hexagon can conveniently take part in active development and discussions there. I do not have an 8 Gen 5 phone for testing, so your test and cross‑validation results on 8 Gen 5 are important for the ggml‑hexagon project. I look forward to your cross‑validation report following the steps outlined in PR‑27642. |
|
@gat45, I have read your comment in PR-27642, thank you for your interest in fastrpc-based ggml-hexagon. The goal of fastrpc-based ggml-hexagon is not replace dspqueue with FastRPC, the fastrpc backend and the dspqueue backend are complementary, not competitive. The two backends should be treated as complementary, not as competitors: each has its own strength areas, and both deserve upstream maintenance. The direction of the PP/TG gap depends on model type, not on which backend is "better" in general. "If the v81 large-model path avoids the overflow/mirror penalty seen in older DSP architectures, that would be a very interesting opportunity for a specialized SM8850 runtime.",Yes, you are absolutely correct. That will be strong evidence for the single‑mempool design in the FastRPC‑based ggml‑hexagon. I look forward to your cross‑validation report on 8Gen5 following the steps outlined in ggml-org#27642. |
|
contact me You are right. The previous report mixed the wall-clock MTP result with the ordinary decode result and omitted the ~16 tok/s result obtained with the 16-token MTP setting. Corrected key performance pictureFor Qwen3.5-9B-D2-A-MTP on SM8850 / HTP, we observed two different classes of MTP measurements:
Configuration | Performance
-- | --
Standard decode, no MTP | ~8.6–9.1 tok/s
MTP enabled, wall-clock measured | ~8.95–9.09 tok/s
MTP with 16-token speculative window | ~16 tok/s
Best later stabilized HTP run | ~11.0–11.3 tok/s
The ~16 tok/s figure must absolutely be retained in the report. It is not interchangeable with the later ~11 tok/s measurement because the benchmark configurations and measurement methodology differ. The important interpretationThe ~16 tok/s MTP result demonstrates that MTP can substantially raise effective decode throughput when the speculative configuration is favorable. The mistake in my previous report was treating:
and
as though they were necessarily the same measurement. They are not. The useful way to present the result is therefore:
|
|
Thanks for your time and thanks for sharing this information. |
|
Target platform: Qualcomm SM8850-class Android device 1. Executive SummaryA series of controlled experiments was conducted on a Qualcomm SM8850 platform using the Hexagon HTP/NPU through GGML/llama.cpp. The experiments produced three important observations:
The data strongly suggests that decode performance depends on the complete execution path: model topology → tensor placement → HTP execution → activation repacking → DMA → memory system → synchronization/orchestration → CPU/runtime overhead rather than simply on raw HTP compute throughput. 2. Hardware / Software Configuration2.1 Target hardwareThe experiments were performed on a Qualcomm SM8850-class device exposing multiple Hexagon HTP execution domains. Observed configuration:
Each HTP session was observed to have approximately:
while the aggregate addressable HTP capacity was approximately:
This is important because a large model can exceed the practical capacity of one HTP session while remaining usable across multiple HTP domains. 3. HTP InitializationA critical reproducibility issue was discovered during initialization. Initial error:
The problem was resolved by explicitly exposing the Qualcomm runtime libraries:
The HTP runtime could then initialize correctly. This environment configuration should therefore be included in reproducibility scripts. 4. Experimental MethodologyEvery performance experiment should record at minimum:
The principal metrics are:
For example:
5. Baseline: Qwen3.5-9B-D2-A-MTPModel:
Representative benchmark:
This established the initial HTP decode regime. The most important observation was the enormous difference between:
Therefore, the workload is not limited by the same mechanism during prefill and autoregressive decode. 6. Single-HTP Memory ExperimentAn 8B Q4_K_M model was executed entirely on HTP0. Configuration:
Measured result:
Process RSS:
This was particularly important because it demonstrated that a single HTP session can sustain a model with approximately 5 GiB of process memory under the tested configuration. 7. Layer-Offload SweepA controlled
ngl | Decode
-- | --
16 | 6.66 tok/s
32 | 5.38 tok/s
48 | 4.86 tok/s
64 | 4.69 tok/s
80 | 4.08 tok/s
The progression is important:
The experiments therefore demonstrate that the performance ceiling is configuration-dependent rather than a single fixed HTP hardware limit. 22. Main ConclusionsConclusion 1 — Quantization alone is insufficientA:
model reduction did not materially improve decode:
Therefore, reducing weight size alone is not enough to optimize autoregressive HTP inference. Conclusion 2 — HTP compute is not necessarily the bottleneckThe Q4 attention experiment reduced:
without improving wall-clock decode. Therefore:
Conclusion 3 — CPU can beat the HTP pathMeasured:
This strongly suggests a runtime/memory/orchestration bottleneck. Conclusion 4 — Activation repacking is a major architectural considerationThe HTP backend converts activations toward:
before matrix execution. Consequently, the true data path is not simply:
but includes:
Conclusion 5 — ~10 tok/s Gemma invalidates a universal 6–7 tok/s ceilingThe latest Gemma result demonstrates that the platform can reach approximately:
under a favorable workload/configuration. The optimization problem therefore becomes:
That is potentially more valuable than another isolated quantization optimization. 23. Reproducible Research HypothesisThe current evidence supports this working hypothesis:
The practical optimization target should therefore be:
rather than simply:
24. Final Experimental StateThe current work has established a reproducible experimental framework and several independently useful findings:
The next decisive step is a Gemma-vs-Qwen controlled differential experiment, keeping the runtime and hardware configuration identical and changing only the model. That can reveal which tensor shapes, layer dimensions, memory patterns, or operator mix are responsible for the ~10 tok/s behavior. B. RED TEAM NOTES
C. ALTERNATIVESFor publication-quality reproducibility, the report should ultimately include an appendix containing the exact shell commands, llama.cpp commit hash, GGUF SHA-256 hashes, complete benchmark logs, thermal telemetry and device build fingerprint. D. GOVERNANCE VERDICTACCEPT — technically coherent and reproducible framework; Gemma ~10 tok/s requires the exact run metadata before being classified as a fully reproducible benchmark. |
|
Thanks for your time and thanks for sharing the valuable information.
|
|
Regarding
These optimization directions are valuable. For the FastRPC‑based ggml‑hexagon, our goal is to avoid modifying any code within the HTP kernels — that scope belongs to Qualcomm’s team. We are all volunteers with limited resources, so we will focus our efforts on Thank you for your interest in this project. Would you be interested in collaborating on this project? If yes, I can send you a collaboration invitation. |
|
@KraftMK, sorry to bother you. The FastRPC‑based ggml‑hexagon has merged Qualcomm's large PR‑26501. It has only been tested and verified on Snapdragon 8 Gen 3 and Snapdragon 8 Gen 4 devices(Android phone). Could you help validate it on your multi‑NPU hardware and share detailed runtime logs(build steps can be found at https://github.com/ggml-hexagon/ggml-hexagon/discussions/77)? This will allow us to improve AP-side allreduce or implement DSP‑side allreduce accordingly. Of course, you are welcome to submit PRs to this public project at your convenience; your contributions are greatly appreciated. |
|
Hi @zhouwg — thank you for the detailed A/B reports and for the
collaboration invitation. I’d be glad to collaborate and to run the
Snapdragon 8 Gen 5 cross-validation you requested. Here is a precise status
update so you know exactly where we are. **Device:** OnePlus 15 — SM8850 /
Hexagon HTP v81, Android 16, rooted. Your `self-build-jz` branch targets
exactly this platform (`v81 — Snapdragon 8 Elite Gen 5`). We are not aware
of published v81 dspqueue-vs-FastRPC measurements yet, so if that remains
the case, we intend to provide the first fully reproducible comparison.
**1. The FastRPC mempool skel for v81 is built.** We compiled
`libggml-htp-v81.so` from the committed HEAD of `self-build-jz` (`entry.c`
+ IDL-generated `ggml_htp_skel.c`, `GGML_HEXAGON_USE_MEMPOOL=ON`,
`DSP_VERSION=v81`, `PREBUILT_LIB_DIR=toolv19_v81`, Hexagon SDK 6.6.0.0 /
tools 19.0.07). It exports `ggml_htp_skel_handle_invoke` with
`_modver=1.0&_idlver=0.0.2`, and the interface is byte-identical to what
the device's FastRPC AP backend requests. Two independent from-scratch
builds produced the same MD5, confirming deterministic builds. Build notes:
- `qaic` and `hexagon-clang` are Linux ELF binaries, so WSL2/Linux is
required. - We had to build from `git archive HEAD`, because the experiment
branch's working tree is currently partially merged: staged `matmul-ops.c`
references kernels for which no corresponding header definitions exist in
the tree. **2. First A/B attempt — invalid, but it confirmed the documented
warning.** We swapped only the backend `.so` and skel, then encountered
exactly the issue described by `build-run-ggmlhexagon-android.sh`:
`libggml-hexagon.so` exposes different libc++ symbols between the
mempool/FastRPC and dspqueue variants. The FastRPC HTP backend did not
successfully register, resulting in a silent initialization/dlopen failure
and an OpenCL-only fallback. The session subsequently hung in
`fastrpc_dspsignal_wait`. Your 08-22 report explains the underlying issue:
each variant needs its own coherent library set: -
`libggml-hexagon-fastrpc.so` + `libggml-htp-v81-fastrpc.so` - versus the
corresponding dspqueue pair. We had deployed a stale mempool AP library
against a newer runtime. The root cause is now understood and does not
appear to be an architectural FastRPC problem. **3. Next step — official
protocol, followed by the reproducible A/B report.** We will rebuild the
complete FastRPC and dspqueue library sets coherently using the official
build/update paths: `build_dspqueue` / `build` `update_dspqueue_libs` /
`update_fastrpc_libs` We will then run the exact fair-comparison parameters
from this thread: `-ngl 99 -t 6 -n 256 --ctx-size 8192 --ubatch-size 64
--poll 1000 --no-warmup --no-mmap -fa on` using identical prompt and model
files for both variants. The resulting report will include a v81
dspqueue-vs-FastRPC table, raw logs, build information and file hashes.
**4. Early Snapdragon 8 Gen 5 measurements we can already share.** Valid
dspqueue-variant measurements on Qwen3-4B Q4_0 using llama-bench p128/n64:
- **Prefill:** HTP ≈20× CPU `pp128: 407.6 vs 20.3 tok/s` - **Decode:** at
4B, the workload is strongly DRAM-bound; Adreno OpenCL improves decode from
approximately `15.3 → 20.1 tok/s`, while HTP alone adds little at this
model size. - **CPU crossover:** CPU wins decode below approximately 2–3B.
We measured: - 1.7B: CPU `33.7` vs GPU+HTP `20.9 tok/s` - 4B: GPU+HTP
`20.4` vs CPU `15.3 tok/s` One important reproducibility caveat should be
included in benchmark scripts: on this device, the idle/battery state can
cap the CPU at approximately 48% of its maximum frequency. That alone
changed: `pp128: 407 → 245 tok/s` `tg: 20.4 → 7.4 tok/s` between two
sessions using identical binaries. CPU frequency/cap state therefore needs
to be recorded before every run. **5. Contribution to the
build/reproduction workflow.** I can also prepare a PR for
`scripts/build-run-ggmlhexagon-android.sh` on `self-build-jz` documenting
and/or checking the environment requirements we identified during
reproduction: - WSL2/Linux requirement (`qaic` / `hexagon-clang` are ELF
binaries); - build-from-committed-HEAD requirement to avoid the current
WIP-tree trap; - the `/vendor/lib64` `LD_LIBRARY_PATH` pitfall that can
break the FastRPC HAL fallback for non-root UIDs (llama.cpp#27677); -
CPU-frequency-cap state as a benchmark precondition. If you prefer, I can
put these points into a separate reproduction document instead. The
complete on-device evidence package is ready: exact commands, raw logs, MD5
hashes and the reproduction dossier. I will attach these with the A/B
benchmark report. Looking forward to the v81 cross-validation and
collaboration.
Le ven. 4 sept. 2026 à 13:17, gat45 ***@***.***> a écrit :
… *Target platform:* Qualcomm SM8850-class Android device
*Runtime:* GGML / llama.cpp + Qualcomm Hexagon HTP backend
*Models tested:* Qwen3.5-9B-D2-A-MTP, Gemma
*Primary objective:* determine the real inference bottlenecks and
identify reproducible runtime optimizations.
------------------------------
1. Executive Summary
A series of controlled experiments was conducted on a Qualcomm SM8850
platform using the Hexagon HTP/NPU through GGML/llama.cpp.
The experiments produced three important observations:
1. *HTP decode performance is not simply proportional to model size or
quantization.*
2. Reducing the Qwen3.5-9B-D2-A-MTP model from *7.29 GiB to 5.07 GiB*
by requantizing attention weights from Q8 to Q4 produced almost no
improvement in decode throughput:
- Q8 attention: *6.72 tok/s*
- Q4 attention: *6.68 tok/s*
3. A later *Gemma configuration reached approximately 10 tok/s*,
indicating that the previously observed ~6–7 tok/s ceiling is *not a
universal limitation of the SM8850 HTP hardware*.
The data strongly suggests that decode performance depends on the complete
execution path:
*model topology → tensor placement → HTP execution → activation repacking
→ DMA → memory system → synchronization/orchestration → CPU/runtime
overhead*
rather than simply on raw HTP compute throughput.
------------------------------
2. Hardware / Software Configuration 2.1 Target hardware
The experiments were performed on a Qualcomm SM8850-class device exposing
multiple Hexagon HTP execution domains.
Observed configuration:
HTP0
HTP1
HTP2
Each HTP session was observed to have approximately:
~3.4–3.6 GiB usable/addressable memory
while the aggregate addressable HTP capacity was approximately:
~10–11.4 GiB
This is important because a large model can exceed the practical capacity
of one HTP session while remaining usable across multiple HTP domains.
------------------------------
3. HTP Initialization
A critical reproducibility issue was discovered during initialization.
Initial error:
ggmlhexagon_init_dsp error 0x80000600
failed to open domain 3 HTP
The problem was resolved by explicitly exposing the Qualcomm runtime
libraries:
export LD_LIBRARY_PATH=/data/local/tmp/npuexport ADSP_LIBRARY_PATH=/data/local/tmp/npu
The HTP runtime could then initialize correctly.
This environment configuration should therefore be included in
reproducibility scripts.
------------------------------
4. Experimental Methodology
Every performance experiment should record at minimum:
device
SoC
Android build
kernel
llama.cpp commit
GGML/Hexagon backend revision
model filename
model size
quantization
context length
ngl
batch size
ubatch size
CPU threads
HTP domain
GPU usage
prompt length
generated token count
cold/warm state
thermal state
memory state
prefill throughput
decode throughput
The principal metrics are:
ppX = prompt/prefill throughput
tgX = token-generation/decode throughput
For example:
pp64 = prefill benchmark using 64 tokens
tg32 = decode benchmark using 32 generated tokens
------------------------------
5. Baseline: Qwen3.5-9B-D2-A-MTP
Model:
Qwen3.5-9B-D2-A-MTP
Parameters: ~9.20 B
Original GGUF: 7.29 GiB
Quantization: Q4_0
Backend: Hexagon HTP
Representative benchmark:
pp64 = 225.83 tok/s
tg32 = 5.51 tok/s
This established the initial HTP decode regime.
The most important observation was the enormous difference between:
prefill ≈ 225.8 tok/s
decode ≈ 5.5 tok/s
Therefore, the workload is not limited by the same mechanism during
prefill and autoregressive decode.
------------------------------
6. Single-HTP Memory Experiment
An 8B Q4_K_M model was executed entirely on HTP0.
Configuration:
HTP domain: HTP0
ngl: 99
MBUF: 5000
quantization: Q4_K_M
Measured result:
tg8 = 6.14 tok/s
pp32 = 10.28 tok/s
Process RSS:
~4.97 GiB
This was particularly important because it demonstrated that a single HTP
session can sustain a model with approximately 5 GiB of process memory
under the tested configuration.
------------------------------
7. Layer-Offload Sweep
A controlled ngl sweep was performed on the 8B Q4_K_M model.
ngl | Decode -- | -- 16 | 6.66 tok/s 32 | 5.38 tok/s 48 | 4.86 tok/s 64 |
4.69 tok/s 80 | 4.08 tok/s
The progression is important:
5.51
↓
6.14
↓
6.68
↓
9.02 CPU
↓
~10 Gemma
The experiments therefore demonstrate that the performance ceiling is
configuration-dependent rather than a single fixed HTP hardware limit.
------------------------------
22. Main Conclusions Conclusion 1 — Quantization alone is insufficient
A:
7.29 GiB → 5.07 GiB
model reduction did not materially improve decode:
6.72 → 6.68 tok/s
Therefore, reducing weight size alone is not enough to optimize
autoregressive HTP inference.
------------------------------
Conclusion 2 — HTP compute is not necessarily the bottleneck
The Q4 attention experiment reduced:
DSP time ≈ 14%
MUL_MAT time ≈ 17%
without improving wall-clock decode.
Therefore:
compute optimization ≠ necessarily throughput optimization
------------------------------
Conclusion 3 — CPU can beat the HTP path
Measured:
CPU ≈ 9.02 tok/s
HTP ≈ 6.68 tok/s
This strongly suggests a runtime/memory/orchestration bottleneck.
------------------------------
Conclusion 4 — Activation repacking is a major architectural consideration
The HTP backend converts activations toward:
Q8_0_TILED
before matrix execution.
Consequently, the true data path is not simply:
quantized weights → HTP
but includes:
activation quantization
→ tiled repacking
→ DMA
→ HTP computation
------------------------------
Conclusion 5 — ~10 tok/s Gemma invalidates a universal 6–7 tok/s ceiling
The latest Gemma result demonstrates that the platform can reach
approximately:
10 tok/s
under a favorable workload/configuration.
The optimization problem therefore becomes:
*Identify which model/runtime characteristics allow Gemma to avoid the
bottleneck affecting the previous Qwen/HTP configuration.*
That is potentially more valuable than another isolated quantization
optimization.
------------------------------
23. Reproducible Research Hypothesis
The current evidence supports this working hypothesis:
For autoregressive LLM decode on SM8850 HTP,
end-to-end throughput is governed less by raw HTP MAC throughput
than by the combined cost of:
activation transformation
- tensor layout/repacking
- DMA
- memory latency/bandwidth
- synchronization
- graph/runtime orchestration
- model-specific tensor geometry.
The practical optimization target should therefore be:
MINIMIZE DATA MOVEMENT
MINIMIZE REPACKING
MINIMIZE SYNCHRONIZATION
MAXIMIZE TENSOR/SHAPE FITNESS
KEEP THE CRITICAL PATH SHORT
rather than simply:
MAXIMIZE HTP LAYERS
------------------------------
24. Final Experimental State
The current work has established a reproducible experimental framework and
several independently useful findings:
✓ HTP initialization procedure identified
✓ Multiple HTP domains characterized
✓ ~3.4–3.6 GiB/session observed
✓ ~10–11.4 GiB aggregate HTP capacity observed
✓ Single-HTP ~5 GiB workload demonstrated
✓ ngl scaling characterized
✓ ubatch behavior investigated
✓ Q8 → Q4 attention A/B test completed
✓ ~30% model-size reduction measured
✓ ~14% DSP-time reduction measured
✓ ~17% MUL_MAT reduction measured
✓ virtually zero decode gain from attention requantization
✓ CPU control exceeded HTP in one configuration
✓ activation Q8_0_TILED path identified
✓ latest Gemma result reached ~10 tok/s
The next decisive step is a *Gemma-vs-Qwen controlled differential
experiment*, keeping the runtime and hardware configuration identical and
changing only the model. That can reveal which tensor shapes, layer
dimensions, memory patterns, or operator mix are responsible for the ~10
tok/s behavior.
------------------------------
B. RED TEAM NOTES
- The *~10 tok/s Gemma result is not yet fully reproducible from the
information currently available* because the exact final Gemma
command/configuration is not present in this turn.
- Therefore, it is reported as an *observed result*, not yet as a
fully specified benchmark record.
- The Qwen A/B data is substantially stronger because both
configurations and measured values are known.
- The claim that memory/orchestration dominates decode is a *strong
hypothesis supported by the experiments*, not yet a direct
hardware-counter proof.
- The HTP-domain capacity numbers are observed experimental values and
should not be interpreted as Qualcomm's official architectural
specification.
C. ALTERNATIVES
For publication-quality reproducibility, the report should ultimately
include an appendix containing the *exact shell commands, llama.cpp
commit hash, GGUF SHA-256 hashes, complete benchmark logs, thermal
telemetry and device build fingerprint*.
D. GOVERNANCE VERDICT
*ACCEPT — technically coherent and reproducible framework; Gemma ~10 tok/s
requires the exact run metadata before being classified as a fully
reproducible benchmark.*
—
Reply to this email directly, view it on GitHub
<#18>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/CC2W4LFHTUMWPH4D5NFIJBL5NKQFHAVCNFSNUABHKJSXA33TNF2G64TZHM4TGMJZGA3TGNZWHNCGS43DOVZXG2LPNY5TQMBZGEZDCNVBOYBA>
.
Triage notifications, keep track of coding agent tasks and review pull
requests on the go with GitHub Mobile for iOS
<https://github.com/notifications/mobile/ios/CC2W4LGN3V67QAOKXL3646T5NKQFHA5CNFSNUABIM5UWIORPF5TWS5BNNB2WEL2ENFZWG5LTONUW63SDN5WW2ZLOOQXTCOBSHA3DQMRVUZZGKYLTN5XKOY3PNVWWK3TUUVSXMZLOOSVGM33PORSXEX3JN5ZQ>
and Android
<https://github.com/notifications/mobile/android/CC2W4LHT66XNJMYUAQMXWRD5NKQFHA5CNFSNUABIM5UWIORPF5TWS5BNNB2WEL2ENFZWG5LTONUW63SDN5WW2ZLOOQXTCOBSHA3DQMRVUZZGKYLTN5XKOY3PNVWWK3TUUVSXMZLOOSXGM33PORSXEX3BNZSHE33JMQ>.
Download it today!
You are receiving this because you commented.Message ID:
***@***.***
com>
|
|
Thanks so much and thanks for your time.
|
RAPPORT SWEEP GRANULARITÉ DSPQUEUE (GGML_HEXAGON_OPBATCH) — 2026-09-05Date : 2026-09-05 · Device : OnePlus 15 CPH2747, SM8850/HTP v81 · Modèle : Mécanique du batching (code,
|
| OPBATCH (ops/msg) | pp128 (t/s) | tg64 (t/s) | T (°C) | Verdict |
|---|---|---|---|---|
| 4 | 428,54 ± 36,26 | 18,02 ± 0,18 | 35 | OK |
| 8 | 406,28 ± 16,43 | 18,01 ± 1,43 | 55 | OK |
| 16 | 408,70 ± 17,14 | 18,88 ± 0,39 | 54 | OK |
| 32 | 437,18 ± 7,98 | 19,31 ± 0,22 | 56 | OK |
| 64 | 426,61 ± 25,40 | 19,00 ± 0,54 | 59 | OK |
| 128 | 425,53 ± 20,64 | HANG fastrpc_dspsignal_wait |
64,3 | pp OK, tg bloqué |
Verdict principal — PAS d'optimum de batching dans 4→64 : tg64 est plat (18,0 → 19,3 t/s, écart < bruit ±0,5) et pp128 est plat aussi (406-437, bruit ±8-36). La taille du message DSPQueue (4 → 64 ops) ne déplace pas le débit au decode ni au prefill dans cette plage. L'hypothèse « il pourrait exister un optimum de granularité que JZ ne choisit pas » (piste DSPQueue/batching) est réfutée par la mesure sur ce device/modèle pour 4→64 ops/message. Le coût par message (dspqueue_write) est négligeable devant le travail par op — cohérent avec le goulot mesuré ailleurs (orchestration par-op, ~65 ms/token de T_AP/RPC, pas le nombre de messages).
Le hang à OPBATCH=128 — frontière thermique, pas frontière de batching
- pp128 passe à 128 (425,53 t/s, même niveau que 4→64) ; c'est tg64 qui se bloque en
fastrpc_dspsignal_wait(T redescendue à 45 °C = DSP idle, réponse de batch jamais arrivée). Même signature que le hang mempool CASE-B de ce matin. - Point survenu au plus haut cumul thermique de la série (T=64,3 °C de départ, après ~10 min de charge soutenue 55-64 °C ; dmesg montre les contraintes kgsl/CPU qui se déclenchent à ce moment).
- Non reproductible à plus basse température : OPBATCH=128 a complété pp128+tg32 à 48 °C (série boundary) et pp128+tg64 à 44,6 °C (série verify), rc=0.
- Interprétation : hang thermique/DSP reliability déclenché par la charge soutenue >64 °C, pas une limite dure liée à 128 ops/message. À confirmer par un re-test à caps max + départ froid (voir §Limites). À signaler au mainteneur : la voie dspqueue peut aussi se coincer sous chaleur soutenue — jusqu'ici on croyait ce mode réservé à fastrpc/mempool.
⚠️ Événement protocolaire — données 256+ et sweeps suivants CONTAMINÉS
- Le
kill -9du bench 128 n'a tué que llama-bench : le shell parent de la sweep originale a survécu et a continué (log original : 256→2048, dont un pic thermique à 81,4 °C au point 512 qui n'a produit aucun résultat). - Les sweeps boundary (64→512) et verify (128/1024/2048) ont donc tourné en concurrence avec la sweep originale toujours vivante + sous caps effondrés (voir §Caps).
- Conséquence : les chiffres OPBATCH 256→2048 (sweep original) et TOUS les points boundary/verify (pp ~345-355, tg ~15-17) sont inexploitables — concurrence de 2 llama-bench sur le même HTP + pkill croisés (
pkill -9 llama-benchdu script boundary a tué les benches de la sweep originale) + caps à ~0,30. Ils sont listés ici à titre indicatif uniquement, jamais comme mesure. - Leçon : toujours vérifier
ps+ tuer le process group complet (kill -- -PGIDoupkill -f sweep_opbatch) avant de relancer une campagne ; ne jamais laisser un sweep en arrière-plan.
Tableaux contaminés (indicatifs, NE PAS citer comme mesures)
| Série | OPBATCH | pp | tg | État réel |
|---|---|---|---|---|
| orig (post-kill) | 256 | 409,56 | tg64 14,21 ± 2,85 | caps en cours de chute, seul |
| orig (post-kill) | 512 | — | aucun résultat | T=81,4 °C, tué par pkill boundary |
| orig (post-kill) | 1024 | 441,32 | tg64 18,34 ± 0,11 | concurrent |
| orig (post-kill) | 2048 | 344,13 | tg64 16,88 ± 2,06 | concurrent |
| boundary | 64→512 | 341-394 | tg32 14,0-17,2 | concurrent + caps ~0,30 |
| verify | 128/1024/2048 | 345-350 | tg64 16,6-17,3 | concurrent + caps ~0,30 |
⚠️ Caps CPU effondrés (nouveau pitfall bench, à consigner)
Après l'excursion thermique (64 °C → pic 81 °C), les caps CPU sont tombés à cpu4 1 555 200 / cpu7 1 267 200 kHz (vs max 3 628 800 / 4 608 000) → caps_ratio ≈ 0,30, encore plus profond que ce matin (0,48). L'écriture scaling_max_freq échoue silencieusement (rc=0 mais valeur re-clampée à 1,27-1,55 GHz) : le HAL oplus verrouille. thermal_zone0/mode refusé (rc=1). → Même mécanisme que le pitfall du matin (DOZE_CAPPED) mais déclenché par la chaleur cette fois. Restauration probable : reboot ou chargeur. Toute campagne bench doit vérifier caps max AVANT chaque run et intercaler des cooldowns sous ~50 °C.
Conclusions
- Sweep 4→64 (mesure propre, caps max) : PAS d'optimum de granularité DSPQueue — tg64 plat 18,0-19,3 t/s, pp128 plat 406-437. La taille du message n'est pas le levier au decode/prefill sur Qwen3-4B.
- Hang OPBATCH=128 = événement thermique (charge soutenue >64 °C → DSP bloqué en
fastrpc_dspsignal_wait), non reproductible à froid → nouvelle donnée de fiabilité : la voie dspqueue se coince aussi sous chaleur, pas seulement fastrpc. - Données 256+ contaminées par erreur protocolaire (sweep parent survivant + lancement concurrent + caps 0,30) — exclues des conclusions.
- Nouveau pitfall bench : excursion thermique → caps verrouillés ~0,30 par le HAL oplus, écriture bloquée → cooldown obligatoire sous ~50 °C + vérification caps avant chaque run.
Prochaines étapes possibles
- Re-test propre OPBATCH 128-2048 à caps max + départ froid (après reboot/chargeur) pour confirmer l'absence d'effet granularité sur la plage haute et trancher le hang 128 (thermique vs batching).
- Re-test du hang : OPBATCH=128, départ 45 °C, n64, caps max — si OK, le hang 128 du sweep est bien thermique.
- Signaler au mainteneur (discussion PP & TG Performance Comparison: dspqueue-based ggml-hexagon vs FastRPC-based ggml-hexagon #71) : dspqueue sensible à la chaleur soutenue (>64 °C) — complément au hang fastrpc/mempool déjà documenté.
|
Thanks for sharing this report and thank you for your time.
You can use GLM‑5.2 or other LLMs to analyze
|
|
I have downloaded the two models you used from https://huggingface.co/gat45/snapdragon-test-npu/tree/main, here are my local test results( on 8Gen4 phone ):
2.1 PP&TG in FastRPC-based ggml-hexagon: 2.2 PP&TG in dspqueue-based ggml-hexagon: Here the reason for the PP & TG difference is similar to |
This comment was marked as duplicate.
This comment was marked as duplicate.
|
This comment is a duplicate of #71 (comment) |
|
I have a few commits I’d like to propose internally, based on the SM8850/HTP v81 validation and the FastRPC profiling work. I’d prefer to share them internally first so we can review them together before making anything public. |
|
Another question: |
|
yes |
|
Thank you. Now I see. Regarding this: "I have a few commits I’d like to propose internally, based on the SM8850/HTP v81 validation and the FastRPC profiling work." I prefer a highly simplified design with minimal dependencies. This is also one of the key strengths of upstream ggml/llama.cpp. BTW, if you have any good ideas on refactoring the large function |
|
Thanks Jeff. Before proposing a refactor, I went back through the project history, the FastRPC/dspqueue work, the upstream RFC, and the recent profiling discussions, including results from other contributors as well as my own measurements. I think the safest conclusion is not that A few observations led me to this conclusion. 1. FastRPC vs dspqueue is not simply a question of one backend being faster. The upstream FastRPC RFC describes FastRPC and dspqueue as complementary execution paths, with different trade-offs depending on the workload. The current Hexagon work also shares the HTP operator implementation between the paths, so a significant part of the difference is in the AP-side execution model, batching, memory/offload behavior, cache handling and scheduling rather than simply different HTP kernels. The public results also show that the relative behavior changes with the model and workload, including the >4 GiB cases where virtual-address / mirror-memory constraints become relevant. Source: 2. Memory/residency is already an independent source of behavior. The project history documents the evolution from per-buffer handling toward the ION/mempool approach, as well as the mirror/overflow behavior when the DSP virtual-address space becomes a constraint. That makes me think memory residency should not be mixed with the actual RPC execution step. Otherwise it becomes difficult to distinguish: This is particularly relevant for large models and for the >4 GiB cases discussed in the project. Source: 3. The In the recent measurements, enabling the NX path substantially reduced the number of The important point for the refactor is not the absolute gain itself, but that: and the number of That suggests there are useful costs between graph construction, node handling, batching and kernel execution that should be independently observable. Source: 4. ARGSORT shows that execution-path selection is workload-dependent. The recent results also show different behavior between PP and TG for ARGSORT: the preferable CPU/HTP path can change with the execution regime. I therefore would not put this kind of policy directly inside the RPC submission code. It seems cleaner to make the existing execution-path decision explicit before the batch is constructed. Again, this does not mean that CPU or HTP is universally better. It means that the decision itself is a separate concern worth isolating. Source: 5. The historical profiler already exposed several different phases. The older profiling work split the path into multiple phases, including AP-side preparation, RPC setup, DSP execution and completion/cache-related work. I agree with removing detailed profiling from the normal path when it has measurable overhead. I would not turn the old But the profiler results are still useful as evidence that the current function contains several logically different operations. So I would use the old profiling phases only as evidence for identifying source-level boundaries, not as the API of the refactor. Source: 6. The recent SM8850 / HTP-v81 work gives another reason to keep these domains separate. My measurements on the v81 path also show a large difference between host-side dispatch/overhead, DMA/gaps and actual accelerator computation. I don't want to generalize those numbers to all devices or workloads. The useful conclusion for this refactor is simply that the end-to-end token time is not equivalent to HTP kernel time. That is another reason why I think the backend should make the following stages easier to isolate: Proposed refactor I would therefore keep the design deliberately small. I would not introduce:
Instead, I would extract small Conceptually: prepare_graph_state(...);
prepare_graph_ops(...);
select_execution_paths(...);
prepare_residency(...);
build_batch(...);
execute_batch(...);
complete_batch(...);The exact number and boundaries would be determined from the current ~1500 LoC implementation rather than imposed beforehand. Most importantly, I would make the first patch strictly structural:
Then the SM8850/v81 changes, NX changes, or other optimizations can remain separate commits with independent A/B measurements. This should also preserve the main strength you mentioned: a very small upstream-oriented implementation with minimal dependencies. For profiling, I would keep it completely optional. The source-level boundaries above would provide natural temporary instrumentation points for AP preparation, RPC setup, DSP execution and completion, without bringing the old profiler back into the normal execution path. So rather than claiming that graph/batch orchestration is already proven to be the bottleneck, I think the evidence supports a more conservative statement:
If this direction makes sense, I can prepare the RFC by mapping the current ~1500 LoC function to the concrete extraction points first, including the relevant benchmark evidence for each boundary, before changing the implementation. |
|
Thanks for your time and sharing your insights.
|
This comment was marked as low quality.
This comment was marked as low quality.
|
Thanks for your time and sharing your insights.
3. I notice an issue with most of your posts: they are too long and lack proper formatting.
|
|
Hi everyone, I’m currently trying to cross-check the HTP v81/SM8850 measurements from this discussion with independent profiling data. Would anyone be willing to run [gat45/profiler-v3](https://github.com/gat45/profiler-v3) on their device and share the results? I’m particularly interested in understanding where the actual latency is going:
A comparison between HTP v81/SM8850 and older Snapdragon/HTP generations would be especially useful. If you can run it, even a single reproducible measurement would help. Ideally, please include: I’m mainly looking for raw measurements rather than conclusions. In particular, I’d like to separate actual HTP kernel execution time from CPU/RPC/synchronization/memory overhead. If someone has already run Thanks! |
|
Sorry that I only have 8 Gen3 & 8 Gen4 devices. |
|
No problem at all — actually, that works well for me. The profiler does not need to be integrated into the main project. My goal is mainly to use it as an external research/debugging tool to understand where the time is spent on HTP, identify bottlenecks, and validate optimization ideas without adding overhead or complexity to the normal execution path. So keeping ggml-hexagon simple and lightweight while maintaining the profiling work separately is perfectly fine. And yes, I’ll also take a look at ggmlhexagon_backend_graph_compute_batch. Refactoring such a large function could be useful independently of the profiling work. |
|
Thanks for your time and sharing your insights. Please note that PP and TG performance for Qualcomm's dspqueue-based ggml-hexagon improved significantly after PR-28994： |
ggml-hexagon MEMPOOL regression: execute_batch fails with 0x80000401 on HTP v81 — bisected to 882fe09 ("mirror optimization")(Version FR en bas — context : device OnePlus 15 / SM8850 / HTP v81, GGML_HEXAGON_USE_MEMPOOL=ON, AP-side only change) SummaryOn Snapdragon 8 Elite Gen 5 (HTP v81), builds with Environment
Reproduction# A) tip 37f752b77 (upstream b11855) + MEMPOOL=ON + patched pool targets {2048,1024,512}
# (or GGML_HEXAGON_RPC_POOL_MB=2048, env override added in our tree)
cd /data/local/tmp/upb11855_mp && LD_LIBRARY_PATH=$PWD ADSP_LIBRARY_PATH=$PWD CDSP_LIBRARY_PATH=$PWD \
GGML_HEXAGON_RPC_POOL_MB=2048 ./llama-bench -m Qwen3.5-9B-D2-A-MTP.gguf -ngl 60 -dev HTP0 -t 4 -p 1 -n 1 -r 1
# → init OK (rpcmempool 2040 MiB allocated), then:
# test_prompt: failed to decode prompt batch, res = -3
# [ggmlhexagon_backend_graph_compute_batch, 7431]: ggml_htp_execute_batch failed: 0x80000401
# FAILS at pp1 (and pp8/pp64), both Qwen 9B and Marco MoE — pool-size independent:
# swept 512/768/1024/1280/1536/1792/2048/2560/3072 MiB → identical failure
# FAILS identically after `magiskpolicy --live "allow shell vendor_qdsp_device chr_file {...}"`
# (0 avc remaining) → not SELinux/device-node
# logcat during failure: session opened "thru HAL" (avc denied /dev/fastrpc-nsp1000 in shell domain),
# probe caps degraded: has_vtcm=0, has_hvx=0, has_hmx=1 (vtcm=8MB in dump but probe reports 0)
# B) 882fe09a3^ (0984881aa) + same pool patch → WORKS
cd /data/local/tmp/up882p && ... GGML_HEXAGON_RPC_POOL_MB=2048 ./llama-bench ... -p 1 -n 1
# → | pp1 2.22 | tg1 5.39 | (no error), rpc overhead avg 64 µsWhat we eliminated
Secondary issue: mirror/overflow regime (pre-882 build)With the working pre-882 build, any HTP weight footprint ≥ pool size activates the
Extra data point: MTP draft placement (dspqueue, upstream b11855)Server flags Files
FR — RésuméBisect ciblé terminé : la régression MEMPOOL |
|
Sorry to reply so late. I'll give a clear reply on Sunday (2026-10-04). |
|
@gat45, thank you so much for your time and this detailed regression report. Commit 882fe09 didn’t surface any failures in 15-model CI tests on my side, but it triggered this bug under your test setup: Snapdragon 8 Elite Gen5 + Qwen3.5-9B-D2-A-MTP.gguf (7.29 GiB). This commit indeed has a flaw, and it’s my oversight. If you have time, could you test the latest code on your Snapdragon 8 Elite Gen5 device with Qwen3.5-9B-D2-A-MTP.gguf (7.29 GiB)? Thanks. Would you be interested in collaborating on this project? If yes, I can send you a collaboration invitation. BTW, you can also help to improve the project kan, such as helping refine the README.fr.md. |
|
Hi jz, we run a local assistant on a Snapdragon 8 Elite (Hexagon v79) and have been tuning the official dspqueue ggml-hexagon for a few weeks (Gemma-4-E4B ~800 PP / ~15 TG, Qwen3-4B ~800–1000 PP / ~20 TG, llama-server on-device). We'd like to share our findings and patches with you — is there a preferred way to contact you privately (e-mail or similar)? No pressure, thanks for your work on ggml-hexagon. |
|
Hi Skuddi, Impressive results on Snapdragon 8 Elite! I'm working on the official ggml-hexagon backend on a Snapdragon 8 Elite Gen 5 (Hexagon v81), particularly on performance optimization, NPU operator support, and CPU fallback issues. I noticed your GitHub repositories don't currently include your Hexagon patches. Would you be willing to share your patches, a diff against upstream, or even a development branch? I'm especially interested in:
I've also been benchmarking Qwen3.5-9B, Marco-Nano MoE, and Nanbeige4.2-3B on Hexagon. I'd be happy to test your changes on v81, compare benchmarks, and share the results with you and the community. Thanks for your work! |
|
Hi @gat45, @GaTmaNnes, and @zhouwg — thanks for the interest! We've published our changes as a branch on top of upstream llama.cpp ( It's three commits: the ggml-hexagon decode stack (Q4_0/IQ4_NL tile layout, thermal governor, FP16 KV shadow + HMX flash-attention in decode, profiling, DSP-queue EINTR retry), a small A note on how this came about: I worked on it together with Claude Code (two instances, one on the phone and one on the PC). Every change was measured on the device before we kept it. @gat45: I'll run Qwen3.5-9B, Marco-Nano and Nanbeige4.2-3B on v79 and post PP/TG plus any unsupported ops, including failures. Happy to compare with your v81 numbers. |
Sorry to reply so late. @gat45/GaTmaNnes. thanks for your help. |
|
Hi Skuddi, Great results on your Snapdragon 8 Elite (Hexagon v79)! I’m testing ggml-hexagon on the Snapdragon 8 Elite Gen 5 (SM8850, HTP v81). Would you be willing to benchmark these three models with your optimized dspqueue backend?
I’ve already tested these architectures on HTP v81. Marco-Nano would be particularly interesting for comparing MoE decoding performance, while Nanbeige could help identify operator compatibility issues and CPU fallbacks. Could you share PP/TG performance, NPU offload configuration and any unsupported operators? Even failed tests would be valuable for comparing v79 and v81. Thanks! |
|
@gat45 — results on v79 (SM8750, RedMagic 10 Pro), our 0045 build (upstream bd4eeaa + 6 hexagon commits + our branch), llama-server,
|
|
Hi Skuddi, Thanks again for the detailed v79 results and for publishing your Jarvis branch. I wanted to share the thermal-controlled benchmark protocol we've been using on my OnePlus 15 (SM8850, HTP v81), so we can compare results under more closely matched conditions. Thermal control
Reproducibility
Our Qwen3-4B Q4_0 reference OnePlus 15 / SM8850 / HTP v81, build Starting temperatures were 33–39 °C, with peaks around 57–60 °C. Our standard benchmark uses Your recent runs reached NPU temperatures of 92–98 °C, including a governor transition to If you're interested, we could first establish a common Qwen3-4B Q4_0 baseline with the same GGUF, context, batch, ubatch and thermal conditions. Then we can compare the NIB layout and FP16 KV shadow optimizations separately. I'm also interested in reproducing your Marco-Nano The goal is to identify portable improvements and share reproducible fixes across v79 and v81, rather than rank devices by throughput. Thanks! |
|
@gat45 , @gatman45 , @GaTmaNnes , thanks for your help while I'm busy working on project Kan(Kan-Linux). Are https://github.com/gat45, https://github.com/gatman45 and https://github.com/GaTmaNnes owned by the same person? If yes, could you use one GitHub account for activities in the ggml-hexagon project to avoid confusion and misunderstanding? Thanks so much. |
|
yes sorry |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
llama.cpp for Qualcomm Hexagon NPU(aka ggml-hexagon)
Background
Android maintained its position as the leading mobile operating system worldwide in the fourth quarter of 2023 with a market share of 70.1 percent . Qualcomm is No.1 mobile SoC semiconductor company in our planet currently.
About Hexagon SDK
Each Qualcomm chip includes multiple Hexagon DSPs such as the compute DSP (cDSP), audio DSP (aDSP), and sensor DSP (SLPI -- Sensor Low Power Island). Each of these DSPs implement a specific Instruction Set Architecture (ISA) version. The compute DSP, which is intended for compute-intensive tasks such as image processing, computer vision, and camera streaming, also includes an instruction set extension for fixed-point vector operations called Hexagon Vector eXtensions (HVX).The following diagram provides an overview of the processing units within the cDSP and how they connect to the memory cache.
Compared to the host CPU, the DSP typically runs at a lower clock speed but provides more parallelism opportunities at the instruction level. This often makes the DSP a better alternative in terms of throughput and/or power consumption. As a result, it is preferable to offload as many large compute-intensive tasks as possible onto the DSP to reduce power consumption of the device and free up cycles on the CPU for additional features.
Hexagon SDK is a lightweight and low-level SDK provided by Qualcomm. developers and AI experts can operate cDSP hardware directly with Hexagon SDK.
Llama.cpp + Hexagon NPU
The llama.cpp Hexagon NPU backend(aka ggml-hexagon backend) is intended to support Qualcomm Hexagon NPU firstly, supported chipsets:
block-beta columns 1 block:llamacpp llamacpp["llama_cpp"] style llamacpp fill:#3c3,color:#000,stroke:#000 end block:ggml_backend ggml_backend["GGML backend subsystem"] style ggml_backend fill:#3c3,color:#000,stroke:#000 block:ggmlbackends ggml_cpu["ggml-cpu"] ggml_metal["ggml-metal"] ggml_sycl["ggml-sycl"] ggml_cuda["ggml-cuda"] ggml_hip["ggml-hip"] ggml_vulkan["ggml-vulkan"] ggml_cann["ggml-cann"] ggml_opencl["ggml-opencl"] ggml_hexagon["ggml-hexagon"] ggml_nnpa["ggml-nnpa"] ggml_ane["ggml-ane"] style ggml_cpu fill:#888,color:#000,stroke:#000 style ggml_metal fill:#888,color:#000,stroke:#000 style ggml_sycl fill:#888,color:#000,stroke:#000 style ggml_cuda fill:#888,color:#000,stroke:#000 style ggml_hip fill:#888,color:#000,stroke:#000 style ggml_vulkan fill:#888,color:#000,stroke:#000 style ggml_cann fill:#888,color:#000,stroke:#000 style ggml_opencl fill:#cc3,color:#000,stroke:#000 style ggml_hexagon fill:#cc3,color:#000,stroke:#000 style ggml_ane fill:#fff,color:#000,stroke:#f00,stroke-width:2,stroke-dasharray:5 style ggml_nnpa fill:#cc3,color:#000,stroke:#000 end end block:ggml_backendsubsystem ggml_backendsubsystem["GGML backend subsystem"] style ggml_backendsubsystem fill:#3c3,color:#000,stroke:#000 end block:group1：2 columns 2 block:ggml_tensor ggml_tensor["GGML tensor"] style ggml_tensor fill:#3c3,color:#000,stroke:#000 end block:ggml_cgraph ggml_cgraph["GGML cgraph"] style ggml_cgraph fill:#3c3,color:#000,stroke:#000 end end block:OS Windows Linux Android QNX end block:hardware_vendors Intel AMD Apple Nvidia Huawei Loongson Qualcomm IBM ggml_metal --> Apple ggml_cuda --> Nvidia ggml_hip --> AMD ggml_cann --> Huawei ggml_sycl --> Intel ggml_opencl --> Qualcomm ggml_hexagon --> Qualcomm ggml_ane --> Apple ggml_nnpa --> IBM end block:hardware_types CPU GPU NPU DSP end block:hardware_archs x86 arm risc loongson end%%{init: {"flowchart": {"htmlLabels": false, 'nodeSpacing': 30, 'rankSpacing': 30}} }%% flowchart LR classDef EXIST fill:#888,color:#000,stroke:#000 classDef DONE fill:#3c3,color:#000,stroke:#000 classDef WIP fill:#cc3,color:#000,stroke:#000 classDef NEW fill:#fff,color:#000,stroke:#f00,stroke-width:2,stroke-dasharray:5 subgraph Legend direction LR EXIST:::EXIST ~~~ WIP:::WIP ~~~ DONE:::DONE ~~~ NEW:::NEW endNews
07/26/2026,10:29
maintain separate kernels in JZ‘s ggml-hexagon again(c220b84): historically, JZ and Qualcomm shared a single DSP kernel directory (htp/) for ops kernels, HVX/HMX headers, and common helpers. This ended with Qualcomm's large PR hexagon: further improved pipeline of the core bits (L2, DMA, MM, FA) ggml-org/llama.cpp#26049 (hexagon: further improved pipeline of the core bits (L2, DMA, MM, FA) ggml-org/llama.cpp#26049, merge commit 0a50d99): after merging upstream master with this PR, JZ's default inference test produced garbled output while Qualcomm's remained normal. The root cause is that this PR moved part of the cache maintenance logic into operator implementations, making JZ's cache subsystem incompatible and causing the garbled output. To avoid chasing each upstream htp/ change, JZ forked htp/ into a new kernels/ directory pinned at baseline commit 2be3826 (where PP/TG fully exceeded Qualcomm and inference output was correct). JZ now maintains kernels/ independently; Qualcomm continues using htp/ which tracks upstream master. Selected stable upstream htp/ improvements are ported into kernels/ manually.
07/24/2026,12:30
add document:how-to-build-ggmlhexagon.md
refine document:JZ ggml-hexagon vs Qualcomm ggml-hexagon: Architecture Analysis
refine document:Why Qualcomm ggml-hexagon Cannot Offload lm-head to the DSP
07/23/2026,20:31
After resolving the stability issues on Snapdragon 8 Gen 3, JZ ggml-hexagon outperforms Qualcomm’s official ggml-hexagon in both PP and TG performance on Snapdragon 8 Gen 3 and Snapdragon 8 Elite (8 Gen 4), details can be found at https://github.com/zhouwg/ggml-hexagon/blob/self-build-jz/docs/backend/jz-ggml-hexagon/ion-mempool-vs-perbuffer-analysis-20260713.md
07/22/2026,23:47
Update docs for first-time readers
07/21/2026,20:30
On Snapdragon 8Elite(aka 8Gen4), PP and TG performance surpasses Qualcomm’s official ggml-hexagon. Latest code is available on GitHub.
07/14/2026,14:39
submit latest codes and docs to Github: fully latest codes can be found at the default branch "self-build-jz" https://github.com/zhouwg/ggml-hexagon, fully latest docs can be found at https://github.com/zhouwg/ggml-hexagon/tree/self-build-jz/docs/backend/jz-ggml-hexagon.
07/11/2026,08:25
remove dual path and trim the project, prepare for updating latest codes. now only a small number of files in the upstream master branch will be affected by JZ's ggml-hexagon modifications(PR-12326's new version: JZ's ggml-hexagon-jz.cpp & entry.c + Qualcomm's kernels):
ggml/src/ggml-hexagon/ggml-hexagon-jz.cpp
ggml/src/ggml-hexagon/CMakeLists.txt
ggml/src/ggml-hexagon/htp/dsp-ctx.h
ggml/src/ggml-hexagon/htp/entry.c
ggml/src/ggml-hexagon/htp/ggml_dsp.idl
ggml/src/ggml-hexagon/htp/Makefile
scripts/build-run-ggmlhexagon-android.sh
the reasons of ggml_dsp.idl is more simple/concise than Qualcomm's ggml_htp idl are:
from my perspective, to use an imperfect analogy: dspqueue is similar to the QNN SDK while native Fast RPC is comparable to the Hexagon SDK. of course, my understanding might not be correct.
07/10/2026,06:54
07/08/2026,23:28
here are the ideas:
under the path mulmat_algotype=29, the DSP side reuses all operators/kernels from Qualcomm’s ggml-hexagon, added a thin compatibility/translation layer for Qualcomm operators/kernels in entry.c on the DSP side and ggml-hexagon.cpp on the AP side, ggml-hexagon.cpp keeps the main data structures and core ideas from PR-12326. JZ's ggml-hexagon uses only native/pure FastRPC, without complicated dspqueue, this makes the implementation/solution more simpler and cleaner.
I hope the upstream community will accept my long-term work/effort and lift my block if JZ's ggml-hexagon gets better PP and TG performance than Qualcomm’s ggml-hexagon, this will be a great technical collaboration between me(independent contributor, aka IC) and Qualcomm’s tech expert(I provide a concise ggml-hexagon.cpp and entry.c/main.c, Qualcomm's tech expert provide highly-performance kernels).
07/02/2026,22:17
07/01/2026,21:04
07/01/2026,11:56
suddenly found there is official hexagon-sdk in https://github.com/snapdragon-toolchain/hexagon-sdk/tree/main, upgrade HEXAGON_SDK from 6.2.0.1 to 6.6.0.0 and upgrade HEXAGON_TOOLS from 8.8.06 to 19.0.07 accordingly.
got another good progress, now PP can reach 35-37 tokens per second on 8Elite phone although "consistency and stability of LLM inference test results" still exists.
06/20/2026,00:05
06/18/2026,22:39
06/01/2026
06/27/2025
performance of fp32 4096x4096 mulmat on cDSP:
before 05/27/2025: about 28 seconds
relaunched the dev activity of project ggml-hexagon since 05/27/2025
06/09/2025: about 7-8 seconds
06/25/2025: about 6-8 seconds
06/27/2025: about 3.4-4.2seconds
06/25/2025
06/09/2025
06/03/2025
05/10/2025
04/24/2025
04/17/2025
04/12/2025(April/12/2025)
04/09/2025(April/09/2025)
04/08/2025(April/08/2025)
04/07/2025(April/07/2025)
04/06/2025(April/06/2025)
04/05/2025
04/02/2025
03/31/2025
03/29/2025
03/25/2025-03/27/2025
03/19/2025---03/24/2025
03/12/2025---03/19/2025
implement a concise implementation of the special approach:"mapping the entire ggml cgraph to a single QNN graph"
01/29/2025---03/11/2025
05/28/2024---06/15/2024
04/26/2024
04/24/2024
03/29/2024---04/24/2024
03/25/2024
03/05/2024---03/16/2024
Hardware
Qualcomm Hexagon NPU
Verified devices
DataType Supports
Android
How to build ggml‐hexagon source code for Android and verify ggml-hexagon backend on Snapdragon based phone
Ubuntu 20.04,22.04,26.04 is validated and recommended as host machine(other Linux distributions might be also ok).
utilize build-run-android.sh to download Android NDK and a customized minimal Hexagon SDK automatically, the fully Qualcomm Hexagon SDK must be obtained with a Qualcomm Developer Account and cannot be downloaded automatically in the build script.
you will need an Android smartphone with adb-connected running on one of below Qualcomm SoCs:
SM8550 (Snapdragon 8 Gen 2)
SM8650 (Snapdragon 8 Gen 3，validated and verified)
SM8750-AB (Snapdragon 8 Elite)(aka Snapdragon 8 Gen 4, strongly recommend, validated and verified)
SM8850 (Snapdragon 8 Elite 5)(aka Snapdragon 8 Gen 5)
we can find that this backend works fine as expected from the log output of "adb logcat | grep ggml-hexagon". for programmers, we can use "adb logcat | grep ggml-hexagon" to help troubleshooting issues on AP side and "adb logcat | grep CDSP0" to help troubleshooting issues on DSP side.
How to do performance comparison of PP and TG between Qualcomm's ggml-hexagon and JZ's ggml-hexagon
for fair performance comparison, the same "running_params=" -ngl 99 -t 6 -n 256 --ctx-size 8192 --ubatch-size 64 --poll 1000 --no-warmup --no-mmap -fa on" " and the same prompt and same LLM model file and same 8Elite phone would be used in both inference test.
We can run automated AB tests on Snapdragon 8 Elite following the recommended steps below:
log_abtest_$(date +%Y%m%d-%H%M%S).txtWe can run non-automated AB tests on Snapdragon 8 Elite following the recommended steps below:
Known Issues & Limitations
Q&A
Feel free to submit issue reports here, even though the JZ ggml-hexagon implementation cannot be merged upstream.
Acknowledgement
[07/01/2026, July 1 2026] Thanks for Trae + GLM-5.2's great help, Qualcomm's ggml-hexagon also give me many help. GLM-5.2 and I co-create the ION-based op-batch solution after many many hours(I don't want to use Qualcomm's dspqueue in JZ's ggml-hexagon because ION-based op-batch without dspqueue is one of the highlights in JZ's ggml-hexagon), GLM-5.2 is one of the co-authors of the JZ's ggml-hexagon since June-03-2026.
[07/06/2026, July 6 2026] Thanks for Trae + DeepSeek-V4-Pro's great help. DeepSeek-V4-Pro did a good job in performance optimization.
[07/09/2026, July 9 2026] Thanks for Trae + MiniMax-M3's breakthrough help and profound insights, the PP performance has been boosted from around 180 to over 300 based on Qualcomm's new operators/kernels.
[07/09/2026, July 9 2026] GLM-5.2, DeepSeek-V4-Pro, MiniMax-M3 are both China's top AI Coding Models, sincerely thanks for the original authors of them.
[07/14/2026, July 14 2026] Kimi-K2.7 also made solid contribution on 07/13/2026 although Kimi-K2.7-Code joined this project on late evening 07-13-2026.
[07/14/2026, July 14 2026] There would be no jz-ggml-hexagon without the excellent operators/kernels implementation provided by Qualcomm.
[07/17/2026, July 17 2026] Kimi-K3 joined this project on late evening 07-17-2026.
[07/21/2026, July 21 2026] I learned a lot from the open-source community when I was a young programmer. Sincere thanks to every contributor in the wonderful open-source community ------ especially the original authors and contributors of Linux, Android, FFmpeg, ggml(another outstanding open-source project from the EU which is exactly similar to FFmpeg), llama.cpp and many other outstanding projects. I hope this project can bring modest value to the llama.cpp community, even though PR PR: Refine ggml-hexagon backend(Qualcomm Hexagon NPU backend) for latest ggml,whisper.cpp,llama.cpp ggml-org/llama.cpp#12326 has faced highly difficulties gaining acceptance and approval within the llama.cpp community.
AI usage disclosure
The core ideas originate from my fully original & fully handwritten PR-12326. Starting in June 2026, I have used AI coding agents to assist with brainstorming new ideas, writing code snippets and English tech docs (I provide questions and scope), and creating automated tests. I have already checked, tested, and fully understand all the code(excluding quantization type conversion) in ggml-hexagon-jz.cpp and entry.c. What matters most is that all decisions, directions and adjustments are made by JZ.
All reactions