Repository navigation
PP & TG Performance Comparison: dspqueue-based ggml-hexagon vs FastRPC-based ggml-hexagon #71
Replies: 14 comments 3 replies
AB Test Analysis Report - 2026-07-28Device: Qualcomm Snapdragon 8 Elite (aka 8 Gen 4) 1. How to generate this report2. Garbled Output Check
3. Performance Data (3 rounds each)JZ
QCOM
4. JZ vs QCOM Advantage
5. Regression Analysis (vs 0726 abtest baseline)
6. ConclusionNo performance regression, no garbled output. After merging upstream master and fixing the CMakeLists.txt build issue (restoring JZ slim build options: Revision History
|
AB Test Analysis Report - 2026-07-29Device: Qualcomm Snapdragon 8 Elite (aka 8 Gen 4) 1. Garbled Output Check
2. Performance Data (3 rounds each)JZ
QCOM
3. JZ vs QCOM Advantage
4. Regression Analysis (vs 0728 abtest baseline)Both JZ and QCOM in this run include the latest upstream master merge. Changes below reflect combined effect of upstream improvements + local refactor.
5. New Statistics Fields (this run)This run introduces 6. ConclusionNo performance regression, no garbled output. This AB test was conducted after merging the latest upstream master code into both JZ and QCOM implementations. After the refactor commit (renumber phases 1-12, drop Post-CIVAC / Post-copy-back debug verification blocks in hot path, add min/max_n_ops_per_call counters, refine cgraph cache comment), JZ TG improved by +3.1% (27.72 -> 28.58 tok/s) due to removal of debug verification code in the hot path. QCOM also shows improvement on both PP (+13.2%) and TG (+2.3%) from the latest upstream master merge. JZ's advantage over QCOM remains stable on Snapdragon 8 Elite (aka 8 Gen 4), with TG speedup improving from 1.12x to 1.13x. Revision History
|
|
PP&TG in Qualcomm ggml-hexagon: PP&TG in JZ ggml-hexagon: PP&TG in Qualcomm ggml-hexagon: PP&TG in JZ ggml-hexagon: |
AB Test Analysis Report - 2026-08-07 Night (Five-Model Comparison)Device: Qualcomm Snapdragon 8 Elite (aka 8 Gen 4), QCOM_HTP_V79, dsp arch 0x79, system mem 24834 MiB, VTCM=8MB, HVX=1, HMX=1, Android device id Backends:
Both backends target the same Hexagon cDSP0 on the same device. Common test config: Date: 2026-08-07 22:39 - 22:49 JZ config: Reproduction command (run from project root on Linux host with Android phone connected): ./scripts/build-run-ggmlhexagon-android.sh run_abtest_all 2>&1 | tee log_abtest_all_$(date +%Y%m%d-%H%M%S).txt1. Headline Results (3-round averages)Table-1: Five-model PP/TG summary (3-round mean)
Key takeaway: JZ wins TG on 4/5 models (max +99.0% on Qwen3.5-2B), loses TG only on qwen1.5-1.8B (the only MHA model, -28.8%). JZ wins PP on 2/5 models (Qwen3.5-2B +10.0% and gemma-4-E2B +49.5%); QCOM leads PP on the other 3. qwen1.5-1.8B is the only PP+TG double-loss model. Compared with the morning run (abtest-analysis-20260807-en.md, log Table-2: Per-token TG latency (ms/tok, averaged from eval time)
2. Per-Model Performance Data (3 rounds)2.1 Qwen3.5-2B-Q4_0.gguf (24 layers, GQA + Delta Net, prompt=52 tokens, JZ batch_calls=256)JZ
QCOM
2.2 gemma-4-E2B-it-Q4_0.gguf (35 layers, GQA 8:1, prompt=58 tokens, JZ batch_calls=256)JZ
QCOM
2.3 gemma-4-E4B_q4_0-it.gguf (42 layers, GQA 4:1, prompt=58 tokens, JZ batch_calls=256)JZ
QCOM
2.4 qwen1_5-1_8b-chat-q4_0.gguf (24 layers, MHA 1:1, prompt=51 tokens, JZ batch_calls=256)JZ
QCOM
2.5 Llama-3.2-1B-Instruct-Q4_0.gguf (16 layers, GQA 4:1, prompt=75 tokens, JZ batch_calls=257)JZ
QCOM
3. Verdict SummaryTable-3: Per-model verdict
TG pattern: JZ wins 4/5, with the single loss on qwen1.5-1.8B (the only MHA model). JZ's TG advantage comes from lm-head DSP offload (Q4_K/Q6_K -> Q4_0 tiled repack) and role-aware cache management ( PP pattern: QCOM wins 3/5 due to dspqueue async pipelining (AP-DSP overlay hides AP prep time). JZ wins PP on Qwen3.5-2B and gemma-4-E2B. The gemma-4-E2B win is explained by 35 GQA 8:1 layers letting JZ's per-layer savings accumulate past QCOM's overlap floor. The Qwen3.5-2B win is new - see section 5. Stability: TG numbers are highly reproducible across 3 runs for all models (JZ TG spread < 1.5% on 4/5 models). PP shows more variance, especially QCOM qwen1.5-1.8B (PP rises 680 -> 752 across runs). 4. Architecture-Driven AnalysisThis section explains the verdict patterns using the architectural differences documented in ion-mempool-vs-perbuffer-analysis-20260713.md and why-perbuffer-cannot-offload-lmhead-20260724-en.md. 4.1 Why JZ wins TG on 4/5 modelsTG is DRAM-bandwidth-bound: every token re-reads all weights from DRAM. The lm-head matrix is the single largest TG cost. The two architecture docs establish that:
Result: JZ TG beats QCOM on every GQA model, with the largest gap on Qwen3.5-2B (+99.0%, 26.74 vs 13.44 tok/s) where lm-head is Q6_K (151,936 vocab) and the repack+offload benefit is biggest. 4.2 Why QCOM wins PP on 3/5 (and why JZ flipped Qwen3.5-2B PP)QCOM's dspqueue (up to 16 batches in flight) overlaps AP descriptor-prep with DSP compute. In PP, where multiple ops per token can be scheduled in parallel, this async overlap hides AP prep time and gives QCOM a PP edge on shallow-GQA models (ion-mempool doc section 5, why-perbuffer doc section 9.1). JZ wins PP when its per-layer DSP savings accumulate past QCOM's fixed dspqueue overlap floor. The crossover depends on layer count and whether the graph is split (why-perbuffer doc section 9.2):
4.3 The MHA corner case: qwen1.5-1.8B double lossqwen1.5-1.8B is the only MHA (1:1) model - no KV head sharing. Its K/V matrices are [2048, 2048] per layer (vs GQA's shared, smaller K/V), causing VTCM pressure on the 8 MB DSP scratchpad. This negates JZ's per-layer DSP advantage in both PP and TG. The why-perbuffer doc (section 9.3) classifies this as a corner case: modern models use GQA where JZ shines; legacy MHA models are where QCOM's dspqueue overlap and lower VTCM pressure win. 4.4 Layer-count crossover modelThe why-perbuffer doc (section 9.2) gives the crossover formula: When 5. Delta vs Morning Run (10:24)The morning run (log Table-4: Qwen3.5-2B, morning vs night
Root cause of the PP flip: the per-head view fix in The JZ PP improvement (+14.9%) is the dominant factor in the flip; the QCOM PP drop (-5.2%) is within thermal/state variance. The other four models are stable between morning and night (within thermal/state variance), consistent with no code change affecting them. 6. Model Architecture Notes
Revision History
|
AB Test Analysis Report - 2026-08-09 Afternoon (Six-Model Comparison)Device: Qualcomm Snapdragon 8 Elite (aka 8 Gen 4), QCOM_HTP_V79, dsp arch 0x79, system mem 24834 MiB, VTCM=8MB, HVX=1, HMX=1, Android device id Backends:
Both backends target the same Hexagon cDSP0 on the same device. Common test config: Date: 2026-08-09 17:40 - 17:56 JZ config: Running params: Reproduction command (run from project root on Linux host with Android phone connected): ./scripts/build-run-ggmlhexagon-android.sh run_abtest_all 2>&1 | tee log_abtest_all_$(date +%Y%m%d-%H%M%S).txt
1. Headline ResultsTable-1: Six-model PP/TG summary (3-round mean, Qwen3.5-9B is 1-round)
Key takeaway: JZ wins TG on 4/6 models (max +105.0% on Qwen3.5-2B), loses TG on qwen1.5-1.8B (-22.6%, the only MHA model) and Qwen3.5-9B (-78.9%, submission overhead + mirror memcpy). JZ wins PP on 2/6 models (Qwen3.5-2B +7.3% and gemma-4-E2B +55.5%). Qwen3.5-9B is the first model where JZ loses both PP and TG by a 5x margin. Both backends have the same 24 Q5_K graph splits (same upstream scheduler, both reject Q5_K), but JZ pays 21.4 ms per sub-call (synchronous FastRPC + mirror) vs QCOM's ~0.1 ms (dspqueue_write), a 214x submission overhead difference. Table-2: Per-token TG latency (ms/tok, averaged from eval time)
2. Per-Model Performance Data2.1 Qwen3.5-2B-Q4_0.gguf (24 layers, GQA + Delta Net, prompt=52 tokens, JZ batch_calls=256)JZ
QCOM
2.2 gemma-4-E2B-it-Q4_0.gguf (35 layers, GQA 8:1, prompt=58 tokens, JZ batch_calls=256)JZ
QCOM
2.3 gemma-4-E4B_q4_0-it.gguf (42 layers, GQA 4:1, prompt=58 tokens, JZ batch_calls=256)JZ
QCOM
2.4 qwen1_5-1_8b-chat-q4_0.gguf (24 layers, MHA 1:1, prompt=51 tokens, JZ batch_calls=256)JZ
QCOM
2.5 Llama-3.2-1B-Instruct-Q4_0.gguf (16 layers, GQA 4:1, prompt=75 tokens, JZ batch_calls=257)JZ
QCOM
2.6 Qwen3.5-9B-Q4_0.gguf (32 layers + delta-net, GQA + Delta Net, prompt=52 tokens, 1 round only)JZ (batch_calls=6144, per-call overhead avg=23355 us)
QCOM (graphs reused=253, same 24 splits as JZ via dspqueue)
3. Verdict SummaryTable-3: Per-model verdict
TG pattern: JZ wins 4/6, with losses on qwen1.5-1.8B (MHA corner case, -22.6%) and Qwen3.5-9B (214x submission overhead + mirror, -78.9%). JZ's TG advantage on the 4 winning models comes from lm-head DSP offload (Q4_K/Q6_K -> Q4_0 tiled repack) and role-aware cache management. PP pattern: QCOM wins 4/6 PP due to dspqueue async pipelining. JZ wins PP on Qwen3.5-2B (+7.3%) and gemma-4-E2B (+55.5%) where per-layer DSP savings accumulate past QCOM's overlap floor. Stability: TG numbers are highly reproducible across 3 runs for the first 5 models (JZ TG spread <2.1%). Qwen3.5-9B has only 1 run so stability is not assessed. 4. Regression Check vs 2026-08-07 Night RunThe 2026-08-07 night run (report abtest-analysis-20260807-2320-en.md, log Table-4: PP regression check (08-07 vs 08-09, 3-round avg)
Table-5: TG regression check (08-07 vs 08-09, 3-round avg)
Regression verdict:
5. Qwen3.5-9B Deep DiveQwen3.5-9B is the newest addition to the six-model CI. It is the first model that exceeds the 4 GiB DSP VA limit (5.03 GiB) and the first to trigger a 5x performance gap in both PP and TG. 5.1 JZ batch_calls and graph splitThe 6144 batch_calls = 24 splits x 256 tokens. The 24 splits come from Q5_K weight tensors that JZ's validator rejects (Q5_K is not in the offload MUL_MAT types list). Both backends have the same 24 splits: QCOM's 5.2 Per-call overhead breakdown
The per-call overhead (avg 23355 us = 23.4 ms) is 54x higher than normal models (~435 us for Qwen3.5-2B). This is because each of the 24 sub-graph calls requires mirror memcpy for heap-fallback weights, not just FastRPC transport. 5.3 QCOM advantage: dspqueue + scatter-gather (not fewer splits)QCOM also has the same 24 graph splits as JZ (both reject Q5_K, same upstream scheduler). QCOM's advantage comes from two mechanisms that make each split cheaper, not fewer splits:
JZ's net advantage: lm-head on DSP. Despite losing overall, JZ retains one advantage: QCOM's 5.4 JZ load time penalty
JZ's longer load time reflects mirror setup: allocating heap fallback buffers, computing mirror mappings, and pre-populating DSP-accessible copies. QCOM's scatter-gather allocates per-tensor ION buffers on-demand during graph execution, reducing upfront load cost. 5.5 Output correctnessBoth JZ and QCOM produce coherent output for Qwen3.5-9B. The Detailed root cause analysis and optimization paths for Qwen3.5-9B are documented in jz-ggmlhexagon-pp-tg-optimization-analysis-zh.md, Chapter 8. 6. Architecture-Driven AnalysisThis section explains the verdict patterns using the architectural differences documented in ion-mempool-vs-perbuffer-analysis-20260713.md and why-perbuffer-cannot-offload-lmhead-20260724-en.md. 6.1 Why JZ wins TG on 4/6 modelsTG is DRAM-bandwidth-bound: every token re-reads all weights from DRAM. The lm-head matrix is the single largest TG cost. The two architecture docs establish that:
6.2 Why QCOM wins PP on 4/6 (and why JZ wins PP on Qwen3.5-2B and gemma-4-E2B)QCOM's dspqueue (up to 16 batches in flight) overlaps AP descriptor-prep with DSP compute. In PP, where multiple ops per token can be scheduled in parallel, this async overlap hides AP prep time and gives QCOM a PP edge on most models. JZ wins PP when its per-layer DSP savings accumulate past QCOM's fixed dspqueue overlap floor:
6.3 The two corner cases: qwen1.5-1.8B and Qwen3.5-9Bqwen1.5-1.8B is the only MHA (1:1) model. Its K/V matrices are [2048, 2048] per layer, causing VTCM pressure on the 8 MB DSP scratchpad. This negates JZ's per-layer DSP advantage in both PP and TG, but the gap is moderate (-22.6% TG, -24.0% PP). Qwen3.5-9B is the first >4 GiB model. Both backends face the same Q5_K graph split (24 sub-graphs per token), but JZ is hit by two additional barriers that QCOM avoids:
JZ retains one net advantage: lm-head on DSP (Q6_K -> Q4_0 repack), saving ~80 ms/token vs QCOM's CPU-bound lm_head (32768 guard). But this is overwhelmed by the 535 ms/token submission overhead. The combined effect: JZ TG is 4.7x slower than QCOM (1.40 vs 6.64 tok/s). 6.4 The <4 GiB vs >4 GiB dividing lineThe six models form a clear pattern based on model size relative to the 4 GiB DSP VA limit:
gemma-4-E4B (4.9 GiB) is the borderline case: it exceeds 4 GiB but has no Q5_K weights, so there are no graph splits (batch_calls=256). JZ's mirror mechanism handles the overflow weights, and the TG advantage (+46.5%) is preserved because the mirror cost is amortized across 42 layers without per-token split overhead. Qwen3.5-9B (5.0 GiB) is only slightly larger but has Q5_K weights, triggering 24 graph splits on both backends. The splits themselves are not the differentiator - it is the 214x per-submission overhead (JZ synchronous FastRPC vs QCOM dspqueue) and the mirror memcpy (JZ heap fallback vs QCOM scatter-gather) that create the 5x gap. 7. Model Architecture Notes
8. Concluding Remark
This shows that JZ's architectural advantages (lm-head offload + mempool contiguous IOVA) are fully unleashed in the < 4 GiB no-split scenario, but suppressed by dual barriers in the > 4 GiB scenario. In other words, JZ's ggml-hexagon backend don't support 4+GiB ion memory pool in a single FastRPC session due to Qualcomm's limitation, this is the biggest technical challenge in JZ's ggml-hexagon. The six-model data confirms this dividing line: JZ wins TG on 4/5 models under 4 GiB (max +105.0%), wins TG on the 4.9 GiB borderline model (+46.5%, no Q5_K split), but loses TG by -78.9% on the 5.03 GiB model where Q5_K graph split (214x submission overhead) and mirror memcpy (78.6% of TG time) create a 5x gap. The < 4 GiB vs > 4 GiB boundary is the critical inflection point for JZ's backend. Revision History
|
AB Test Performance Comparison Report - 2026-08-22 PM (Eight-Model Comparison)Device: Qualcomm Snapdragon 8 Elite (a.k.a. 8 Gen 4), QCOM_HTP_V79, dsp arch 0x79, system memory 24834 MiB, VTCM=8MB, HVX=1, HMX=1, Android device ID Backends:
Both backends target the same Hexagon cDSP0 on the same device. Common test configuration: Date: 2026-08-22 17:13 - 17:32 fastrpc config: Run parameters: Table-0: Supported Model Aliases
Reproduction command (after connecting the Android phone from a Linux host, run from the project root): ./scripts/build-run-ggmlhexagon-android.sh run_abtest_all 2>&1 | tee log_abtest_all_$(date +%Y%m%d-%H%M%S).txt
1. Core ResultsTable-1: Eight-Model PP/TG Summary (3-round average, Qwen3.5-9B single run)
Key findings: fastrpc wins TG on 5/8 models (minicpm5-1b +43.8%, Llama-3.2-1B +47.1%, Qwen3.5-2B +80.5%, gemma-4-E2B +8.0%, gemma-4-E4B +22.1%), and loses on 3/8: qwen1.5-1.8B (-29.6%, MHA edge case), Nanbeige-3B (-11.9%, Q4_0 lm_head + large vocab), Qwen3.5-9B (-14.3%, cpu-backend fallback). The 5/8 TG win pattern is identical to 08-15, 08-16, 08-17, 08-18, and 08-20 — the sixth consecutive run with the same win/loss distribution. Compared to 08-20, fastrpc TG shifted on two notable models: qwen1.5-1.8B dropped -8.2% (18.65 -> 17.12), a genuine regression with PP also dropping -4.4% (545.54 -> 521.36). Conversely, Qwen3.5-9B improved +10.5% (5.25 -> 5.80) with load time decreasing -17.1% (5568 -> 4617 ms), suggesting the p3 anomaly from 08-20 (396,759 us, 127x increase) has been fixed in version 0.99.7.5. dspqueue TG is very stable across all 8 models (changes within ±4% vs 08-20), confirming the dspqueue library change (+5288 bytes) did not introduce regressions. PP pattern: fastrpc wins PP on 4/8 (minicpm5-1b +14.3%, Qwen3.5-2B +11.5%, gemma-4-E2B +36.3%, Nanbeige-3B +63.9%), loses on 4/8 (qwen1.5-1.8B -29.4%, Llama-3.2-1B -5.4%, gemma-4-E4B -5.4%, Qwen3.5-9B -90.1%). Compared to 08-20 (4 wins, 4 losses), the PP pattern is unchanged. The notable shift: qwen1.5-1.8B fastrpc PP dropped from 545.54 to 521.36 (-4.4%), widening the PP gap from -24.3% to -29.4%. Stability: fastrpc TG is highly reproducible across 3 rounds for 6 of the first 7 models (spread <1.0%). The exceptions are minicpm5-1b (1/3 runs degenerate) and gemma-4-E4B (spread 2.5%: 14.53-14.90). dspqueue TG is stable for most models (spread <2%), except gemma-4-E4B (spread 3.0%: 11.88-12.24) and Qwen3.5-2B (spread 3.7%: 14.73-15.29). |
AB Test Performance Comparison Report - 2026-08-22 PM (Eight-Model Comparison)Device: Qualcomm Snapdragon 8 Elite (a.k.a. 8 Gen 4), QCOM_HTP_V79, dsp arch 0x79, system memory 24834 MiB, VTCM=8MB, HVX=1, HMX=1, Android device ID Backends:
Both backends target the same Hexagon cDSP0 on the same device. Common test configuration: Date: 2026-08-22 17:13 - 17:32 fastrpc config: Run parameters: Table-0: Supported Model Aliases
Reproduction command (after connecting the Android phone from a Linux host, run from the project root): ./scripts/build-run-ggmlhexagon-android.sh run_abtest_all 2>&1 | tee log_abtest_all_$(date +%Y%m%d-%H%M%S).txt
1. Core ResultsTable-1: Eight-Model PP/TG Summary (3-round average, Qwen3.5-9B single run)
Key findings: fastrpc wins TG on 5/8 models (minicpm5-1b +43.8%, Llama-3.2-1B +47.1%, Qwen3.5-2B +80.5%, gemma-4-E2B +8.0%, gemma-4-E4B +22.1%), and loses on 3/8: qwen1.5-1.8B (-29.6%, MHA edge case), Nanbeige-3B (-11.9%, Q4_0 lm_head + large vocab), Qwen3.5-9B (-14.3%, cpu-backend fallback). The 5/8 TG win pattern is identical to 08-15, 08-16, 08-17, 08-18, and 08-20 — the sixth consecutive run with the same win/loss distribution. Compared to 08-20, fastrpc TG shifted on two notable models: qwen1.5-1.8B dropped -8.2% (18.65 -> 17.12), a genuine regression with PP also dropping -4.4% (545.54 -> 521.36). Conversely, Qwen3.5-9B improved +10.5% (5.25 -> 5.80) with load time decreasing -17.1% (5568 -> 4617 ms), suggesting the p3 anomaly from 08-20 (396,759 us, 127x increase) has been fixed in version 0.99.7.5. dspqueue TG is very stable across all 8 models (changes within ±4% vs 08-20), confirming the dspqueue library change (+5288 bytes) did not introduce regressions. PP pattern: fastrpc wins PP on 4/8 (minicpm5-1b +14.3%, Qwen3.5-2B +11.5%, gemma-4-E2B +36.3%, Nanbeige-3B +63.9%), loses on 4/8 (qwen1.5-1.8B -29.4%, Llama-3.2-1B -5.4%, gemma-4-E4B -5.4%, Qwen3.5-9B -90.1%). Compared to 08-20 (4 wins, 4 losses), the PP pattern is unchanged. The notable shift: qwen1.5-1.8B fastrpc PP dropped from 545.54 to 521.36 (-4.4%), widening the PP gap from -24.3% to -29.4%. Stability: fastrpc TG is highly reproducible across 3 rounds for 6 of the first 7 models (spread <1.0%). The exceptions are minicpm5-1b (1/3 runs degenerate) and gemma-4-E4B (spread 2.5%: 14.53-14.90). dspqueue TG is stable for most models (spread <2%), except gemma-4-E4B (spread 3.0%: 11.88-12.24) and Qwen3.5-2B (spread 3.7%: 14.73-15.29). |
|
PP&TG in Qualcomm's dspqueue-based ggml-hexagon 
PP&TG in FastRPC-based ggml-hexagon: 
|
|
after merge Qualcomm's large PR-26501, PP & TG in Qualcomm's dspqueue-based ggml-hexagon: 
PP & TG in fastrpc-based ggml-hexagon: |
AB Test Performance Comparison Report - 2026-08-28 PM (Eight-Model Comparison)Device: Qualcomm Snapdragon 8 Elite (a.k.a. 8 Gen 4), QCOM_HTP_V79, dsp arch 0x79, system memory 24834 MiB, VTCM=8MB, HVX=1, HMX=1, Android device ID Backends:
Both backends target the same Hexagon cDSP0 on the same device. Common test configuration: Date: 2026-08-28 12:48 - 13:05 fastrpc config: Run parameters: Table-0: Supported Model Aliases
Reproduction command (after connecting the Android phone from a Linux host, run from the project root): ./scripts/build-run-ggmlhexagon-android.sh run_abtest_all 2>&1 | tee log_abtest_all_$(date +%Y%m%d-%H%M%S).txt1. Core ResultsTable-1: Eight-Model PP/TG Summary (3-round average, Qwen3.5-9B single run)
Key findings: fastrpc wins TG on 5/8 models (minicpm5-1b +42.6%, Llama-3.2-1B +37.2%, Qwen3.5-2B +25.3%, gemma-4-E2B +13.5%, gemma-4-E4B +21.5%), and loses on 3/8: qwen1.5-1.8B (-31.6%, MHA edge case), Nanbeige-3B (-3.7%, Q4_0 lm_head + large vocab), Qwen3.5-9B (-33.7%, cpu-backend fallback). The 5/8 TG win pattern is identical to 08-15, 08-16, 08-17, 08-18, 08-20, and 08-22 — the seventh consecutive run with the same win/loss distribution. However, the magnitude of fastrpc's TG advantages narrowed significantly due to the dspqueue library improvement (+628888 bytes, +11.1%):
Conversely, Qwen3.5-9B fastrpc regressed significantly: TG dropped from 5.80 (08-22) to 4.28 (-26.2%), and load time increased from 4617 ms to 6699 ms (+45.1%). The p3 anomaly from 08-20 did not recur (p3=3,930 us, normal). Instead, dsp_exec increased +16.3% (13.35 s → 15.53 s), directly causing the TG regression. PP pattern: fastrpc wins PP on 4/8 (minicpm5-1b +4.4%, Qwen3.5-2B +5.0%, gemma-4-E2B +38.5%, Nanbeige-3B +61.1%), loses on 4/8 (qwen1.5-1.8B -29.5%, Llama-3.2-1B -6.1%, gemma-4-E4B -10.8%, Qwen3.5-9B -92.6%). Compared to 08-22 (4 wins, 4 losses), the PP pattern is unchanged, but qwen1.5-1.8B fastrpc PP recovered from 521.36 to 543.68 (+4.3%), narrowing the PP gap from -29.4% to -29.5% (essentially flat, as dspqueue PP also improved +4.5%). Stability: fastrpc TG is highly reproducible across 3 rounds for 5 of the first 7 models (spread <1.0%). The exceptions are minicpm5-1b (1/3 runs degenerate) and Llama-3.2-1B (2/3 runs early EOS at 8 tokens). dspqueue TG is stable for most models (spread <2%), except Qwen3.5-2B (spread 3.6%: 20.06-21.11) and gemma-4-E4B (spread 3.0%: 11.78-12.75). 2. Summary of FindingsTable-2: Per-Model Verdict
TG pattern: fastrpc wins 5/8, loses 3/8 — identical win/loss pattern to 08-15, 08-16, 08-17, 08-18, 08-20, and 08-22. This is the seventh consecutive run with the same 5/8 TG win distribution. However, the margins shifted dramatically due to the dspqueue library improvement (+628888 bytes, +11.1%):
If dspqueue continues improving at this rate, the 5/8 TG win pattern could break on Qwen3.5-2B (advantage down to +25.3% from +80.5% in one run). PP pattern: fastrpc wins PP on 4/8 (minicpm5-1b +4.4%, Qwen3.5-2B +5.0%, gemma-4-E2B +38.5%, Nanbeige-3B +61.1%), loses on 4/8 (qwen1.5-1.8B -29.5%, Llama-3.2-1B -6.1%, gemma-4-E4B -10.8%, Qwen3.5-9B -92.6%). Compared to 08-22 (4 wins, 4 losses), the PP pattern is unchanged. dspqueue library improvement: The dspqueue host library increase of 628888 bytes (+11.1%) is the largest single-run library change across all test runs. It brought major TG improvements on 4 models (Qwen3.5-2B +38.5%, Llama-3.2-1B +14.3%, qwen1.5-1.8B +13.2%, minicpm5-1b +5.1%), and minor regressions on 3 models (Nanbeige-3B -5.4%, gemma-4-E2B -3.2%, Qwen3.5-9B -4.6%). gemma-4-E4B was essentially unchanged (+2.0%). The improvement is concentrated on models with GQA attention and Q4_K/Q6_K lm_head, while models with Q4_0 lm_head (Nanbeige-3B, Qwen3.5-9B) did not benefit. Table-3: Cross-Model AP Phase Comparison (p1-p10, 3-round average; Run 1 for Qwen3.5-9B)
Table-4: Cross-Model RPC, DSP, and HMX Metrics (3-round average; Run 1 for Qwen3.5-9B; Run 2 for minicpm5-1b and Llama-3.2-1B)
DSP profiling insights: The new HMX metrics reveal significant architectural differences across models:
3. Regression Check Against 2026-08-22 PM RunThe 08-22 PM run (report Table-5: PP Regression Check (08-22 vs 08-28, 3-round average)
Table-6: TG Regression Check (08-22 vs 08-28, 3-round average)
Regression verdict:
|
This comment was marked as low quality.
This comment was marked as low quality.
|
Thank you for your interest in this project and for sharing your valuable experiments on HTP v81 (SM8850 / Snapdragon 8 Elite Gen5). You are absolutely correct on your observations:
|
|
Date: 2026-09-02 1. Executive SummaryThis campaign introduces a sparse-MoE control workload to the Qualcomm Hexagon HTP investigation. The purpose is to separate the cost of active matrix computation from the additional costs of MoE routing, expert dispatch, data movement, synchronization, and host/DSP orchestration during batch-1 decode. The tested workload was Marco-Nano-Instruct, converted from The clean result was: The dense Qwen reference from the same campaign was: The comparison is: Metric | Dense Qwen MTP1 | Marco-Nano sparse MoE -- | -- | -- End-to-end decode | 11.26 tok/s | 26.72 tok/s Time/token | 88.8 ms | 37.4 ms DSP MUL_MAT / token | ~65 ms | ~8.7 msThe lower confidence values intentionally reflect the need for direct event-level wall-clock attribution. 28. Preserved ArtifactsHost model directory: Primary files: Q4_0 size: Report: Validation: Raw PMU/profile logs are preserved for re-analysis. |
|
Thanks for your time. Could you please perform cross‑validation on your Snapdragon 8 Gen 5 phone and create a benchmark report similar to this one: You may use GLM‑5.2 or another LLM to analyze |
|
1.Finally, the FastRPC-based ggml-hexagon works fine with Spark‑X2.5‑4B.gguf. Thanks so much to team at USTC:
Also thanks to Xiaomi Mimo V2.5: for the inference issue with Spark-X2.5-4B.gguf, Mimo V2.5 identified the root cause and delivered a working fix in collaboration with me.
The reason for slow PP & TG on Spark-4B is similar to Qwen3.5‑9B‑Q4_0.gguf: many graph splits in FastRPC-based ggml-hexagon are caused by the 4 GiB mempool limitation. Hexagon V79 user-mode has a 32-bit virtual address space (4 GiB):"The Hexagon processor has a 32-bit byte-addressable memory address space. The user application may access the entire linear address space." Hexagon V79 Programmer's Reference Manual, Memory Section This means the DSP can only mmap/access <= 4 GiB at any time. Models larger than 4 GiB (e.g. Qwen3.5-9B, Spark-4B) cannot fit entirely in a single mempool. The FastRPC-based ggml-hexagon falls back to heap allocation with mirror memcpy for overflow weights, incurring significant PP&TG overhead. In short: many graph splits in FastRPC-based ggml-hexagon are caused by the 4 GiB mempool limitation. 
3.1 PP&TG in FastRPC-based ggml-hexagon: 3.2 PP&TG in dspqueue-based ggml-hexagon(aka Qualcomm's official ggml-hexagon): 
|
================================================================================
|
|
Thanks for sharing the test report and thanks for your time. test reportReport content lacks proper formatting. or follow the template in #83 Thank you for your help and collaboration. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
How to perform PP & TG benchmark comparisons between dspqueue-based ggml-hexagon and FastRPC-based ggml-hexagon
for fair performance comparison, the same "running_params=" -ngl 99 -t 6 -n 256 --ctx-size 8192 --ubatch-size 64 --poll 1000 --no-warmup --no-mmap -fa on" " and the same prompt and same LLM model file and same 8Elite phone would be used in both inference test.
We can run automated AB tests on Snapdragon 8 Elite following the recommended steps below:
log_abtest_$(date +%Y%m%d-%H%M%S).txtWe can run non-automated AB tests on Snapdragon 8 Elite following the recommended steps below:
Notice
Feel free to share test results from different Qualcomm chip-based Android phones. For developers and experts from China, posts written in Chinese are permitted.Developers and experts on GitHub, please support me in getting JZ's ggml-hexagon accepted by the Qualcomm and upstream llama.cpp community
Test results
Snapdragon 8 Elite(aka 8Gen4), Vendor: OnePlus
PP&TG in JZ's ggml-hexagon:
PP&TG in Qualcomm's ggml-hexagon
Snapdragon 8Gen3, Vendor: Xiaomi
LLM inference test
llama-bench test
PP&TG in Qualcomm ggml-hexagon:
PP&TG in JZ ggml-hexagon:
All reactions