Performance of llama.cpp with FastRPC/mempool-based ggml-hexagon

[zhouwg]

Overview

Qualcomm Hexagon SDK exposes two distinct RPC transport mechanisms: native FastRPC and dspqueue.

Project ggml‑hexagon introduces a FastRPC/mempool‑based ggml‑hexagon backend. It coexists with the existing dspqueue‑based (aka Qualcomm's official) ggml‑hexagon implementation and demonstrates that the FastRPC-based ggml-hexagon greatly outperforms Qualcomm's official ggml-hexagon on some modern models.

This dual‑backend pattern mirrors existing patterns inside llama.cpp:

• ggml‑openvino vs ggml‑sycl (multiple backends targeting same hardware)
• ggml‑cuda vs ggml‑hip (re‑use of existing device kernels across host layers)

This is similar to the Apple Silicon benchmark thread, but for FastRPC-based ggml-hexagon! We'll be testing the qwen3-2b qwen3-4b spark-1b gemma4-e2b nanbeige-3b gemma4-e4b qwen3-9b model like the other thread to keep things consistent, the build script (scripts/build-run-ggmlhexagon-android.sh) automatically downloads 8 models on first run.

Attention

For developers and experts from China, please note:

The build script (scripts/build-run-ggmlhexagon-android.sh) automatically downloads 8 models on its first run. Please ensure you can access to Hugging Face and have sufficient network bandwidth to download LLM models from Hugging Face.

Instructions(template-1)

git clone https://github.com/ggml-hexagon/ggml-hexagon.git
git checkout self-build-jz
./scripts/build-run-ggmlhexagon-android.sh build (build the mempool/FastRPC-invoke ggml-hexagon backend for performance comparision)
./scripts/build-run-ggmlhexagon-android.sh build_dspqueue (build the dspqueue ggml-hexagon backend for performance comparison, aka Qualcomm's official ggml-hexagon)
./scripts/build-run-ggmlhexagon-android.sh run_llamaversion (display llama-cpp version information, e.g. version: 0.2.0-dev (build 11120, commit 9f03708a9), built with Clang 21.0.0 for Android aarch64)
./scripts/build-run-ggmlhexagon-android.sh run_abtest_all 2>&1 | tee log_abtest_all_$(date +%Y%m%d-%H%M%S).txt

Feel free to use your preferred LLM to analyze log_abtest_all_xxxx.txt, create the test report following the template below.

---

PP & TG Performance Comparison: dspqueue-based ggml-hexagon vs FastRPC-based ggml-hexagon
Device: Qualcomm Snapdragon 8 Elite (aka 8 Gen 4)
Date: All AB tests complete 2026-09-15 11:49:13
llama.cpp version:
version: 0.4.1-dev (build 11594, commit 57364ab)
built with Clang 21.0.0 for Android aarch64

Table: Eight-Model PP/TG Summary (3-round average, Qwen3.5-9B single run)

Model	Layers	Model Size	fastrpc PP	dspqueue PP	PP Diff	fastrpc TG	dspqueue TG	TG Diff
minicpm5-1b	24	635MiB	981.54	824.59	+19.0%	52.54	37.99	+38.3%
Qwen3.5-2B	24	1.2GiB	384.16	348.96	+10.1%	22.39	15.76	+42.1%
Qwen3.5-4B	32	2.5GiB	183.69	142.80	+28.6%	10.47	9.19	+13.9%
Spark-X2.5-1.7B	28	3.2GiB	355.35	10.45	+3300.5%	15.00	5.78	+159.5%
gemma-4-E2B	35	2.9GiB	557.48	383.21	+45.5%	27.82	22.51	+23.6%
Nanbeige-3B	22	2.4GiB	286.56	166.08	+72.5%	7.94	8.60	-7.7%
gemma-4-E4B	42	4.9GiB	334.86	338.37	-1.0%	14.72	10.03	+46.8%
Qwen3.5-9B	32	5.1GiB	11.05	99.35	-88.9%	3.91	5.95	-34.3%

---

Another template(template-2)

git clone https://github.com/ggml-hexagon/ggml-hexagon.git
git checkout self-build-jz
./scripts/build-run-ggmlhexagon-android.sh build (build the mempool/FastRPC-invoke ggml-hexagon backend for performance comparision)
./scripts/build-run-ggmlhexagon-android.sh build_dspqueue (build the dspqueue ggml-hexagon backend for performance comparison, aka Qualcomm's official ggml-hexagon)
./scripts/build-run-ggmlhexagon-android.sh run_llamaversion (display llama-cpp version information, e.g. version: 0.2.0-dev (build 11120, commit 9f03708a9), built with Clang 21.0.0 for Android aarch64)
./scripts/build-run-ggmlhexagon-android.sh update_fastrpc_libs (push mempool/FastRPC runtime .so from out/ab-test/ to device, for build)
./scripts/build-run-ggmlhexagon-android.sh run_llamabench gemma4-e2b
./scripts/build-run-ggmlhexagon-android.sh update_dspqueue_libs (push dspqueue runtime .so from out/ab-test/ to device, for build_dspqueue)
./scripts/build-run-ggmlhexagon-android.sh run_llamabench gemma4-e2b

[zhouwg]

PP & TG Performance Comparison: dspqueue-based ggml-hexagon vs FastRPC-based ggml-hexagon
Device: Qualcomm Snapdragon 8 Elite (aka 8 Gen 4)
Date: All AB tests complete 2026-09-17
llama.cpp version:
version: 0.4.1-dev (build 11643, commit 0984881)
built with Clang 21.0.0 for Android aarch64

Table: Eight-Model PP/TG Summary (3-round average, Qwen3.5-9B single run)

Model	Layers	Model Size	fastrpc PP	dspqueue PP	PP Diff	fastrpc TG	dspqueue TG	TG Diff
minicpm5-1b	24	635MiB	956.92	1154.23	-17.1%	50.79	61.88	-17.9%
Qwen3.5-2B	24	1.2GiB	400.30	496.39	-19.4%	28.12	29.33	-4.1%
Qwen3.5-4B	32	2.5GiB	203.05	152.05	+33.5%	13.29	12.68	+4.8%
Spark-X2.5-1.7B	28	3.2GiB	357.32	10.60	+3270.9%	15.43	6.04	+155.5%
gemma-4-E2B	35	2.9GiB	587.78	418.70	+40.4%	26.49	28.39	-6.7%
Nanbeige-3B	22	2.4GiB	281.00	379.71	-26.0%	8.19	10.55	-22.4%
gemma-4-E4B	42	4.9GiB	341.63	445.46	-23.3%	14.60	15.17	-3.8%
Qwen3.5-9B	32	5.1GiB	15.94	105.82	-84.9%	4.74	7.79	-39.2%

[zhouwg]

So it seems native Q4_K and Q6_K hexagon kernel show significant impact.

[zhouwg]

PP & TG Performance Comparison report on 2026-09-27

Device: Qualcomm Snapdragon 8 Elite (aka 8 Gen 4)
Model: 15 models("minicpm5-1b" "qwen3-2b" "qwen3-4b" "spark-1b" "gemma4-e2b" "gemma4-e4b" "nanbeige-3b" "qwen3-9b" "llama3" "qwen1" "nanbeige-3b-q80" "minicpm5-1b-q80" "minicpm5-1b-q4-k-m" "minicpm5-2b-q80" "spark-4b"), default are gemma-4-E2B-it-Q4_0.gguf(aka gemma4-e2b) and spark-1b(aka Spark-X2.5-1.7B.gguf)
Date: All standalone tests and AB tests complete 2026-09-27
Context: After merging upstream master latest code
llama.cpp version: version: 0.5.0-dev (build 11849, commit 1fa6441, behind ggml-org#29502) built with Clang 21.0.0 for Android aarch64
FastRPC/mempool-based ggml-hexagon version: version 0.4.8

1. Standalone tests

PP&TG in dspqueue-based ggml-hexagon(aka Qualcomm's official ggml-hexagon):

PP&TG in FastRPC/mempool-based ggml-hexagon:

2. Eight-Model AB tests PP/TG Summary (3-round average, Qwen3.5-9B single run)

Eight-model PP/TG Comparison Analysis (log_abtest_all_20260927-113422.txt), generated by GLM-5.2

Model	Layers	Model Size	fastrpc PP	dspqueue PP	PP Diff	fastrpc TG	dspqueue TG	TG Diff
minicpm5-1b	24	635MiB	1027.95	1021.36	+0.6%	56.60	65.73	-13.9%
Qwen3.5-2B	24	1.2GiB	535.50	500.27	+7.0%	29.14	29.22	-0.3%
Qwen3.5-4B	32	2.5GiB	265.58	220.90	+20.2%	13.37	14.33	-6.7%
Spark-X2.5-1.7B	28	3.2GiB	361.46	10.80	+3246.9%	14.58	5.86	+148.8%
gemma-4-E2B	35	2.9GiB	573.12	311.37	+84.1%	26.41	26.56	-0.6%
Nanbeige-3B	22	2.4GiB	284.63	260.86	+9.1%	8.42	10.34	-18.6%
gemma-4-E4B	42	4.9GiB	339.36	328.77	+3.2%	14.37	14.12	+1.8%
Qwen3.5-9B	32	5.1GiB	7.32	188.52	-96.1%	4.92	8.36	-41.1%