ggml-hexagon v0.98(v20250625)

[zhouwg]

Overview

the ggml-hexagon v0.98(ggml-hexagon.cpp v1.13 + ggml-dsp v0.98) is released on 20250625(June 25 2025).

fully source code of ggml-hexagon.cpp: https://github.com/zhouwg/ggml-hexagon/blob/self-build/ggml/src/ggml-hexagon/ggml-hexagon.cpp

reference source code of ggml-dsp: https://github.com/zhouwg/ggml-hexagon/tree/self-build/ggml/src/ggml-hexagon/kernels

test phone is a Snapdragon 8Elite based Android phone and a Snapdragon 8Gen3 based Android phone, test model is qwen1_5-1_8b-chat-q4_0.gguf. QNN SDK is v2.35.0.250530, Hexagon SDK is v6.2.0.1, Android NDK is android-ndk-r28.

computation results in the following test cases can be reproduced by third-party or AI experts otherwise the benchmark data is meaningless.

Supported Qualcomm mobile SoC

for simplify workflow, only support following two HTP archs at the moment.

#v75 --- Snapdragon 8 Gen3
#v79 --- Snapdragon 8 Elite

Android phone equipped with Qualcomm Snapdragon 8Elite is strongly recommended, because

• sometimes the same dsp codes can running well as expected on Snapdragon 8Elite based phone, but can't works as expected on other Snapdragon based phone(e.g. 8Gen3).
• 8Elite's memory subsystem support for LP-DDR5x memory, up to 5300 MHz; 8Gen3's memory subsystem support for LP-DDR5x memory, up to 4800 MHz.
• DSP clock rate on 8Gen3 is slower than DSP clock rate on 8Elite.

for 8Gen3(0x8c75):

for 8Elite(0x8c79):

Highlights

• support HVX-enabled llamafile-based mulmat on cDSP side
• add mulmat_algotype to develop/verify various mulmat algorithms/optimization approaches on cDSP side, currently support 0,1,2,3,4,5,6,32(0x20),33(0x21). developers/domain experts/AI experts around the world can help to optimize them accordingly in the future.

How to verify on Android phone

a Linux machine is required for following steps, Android phone equipped with Qualcomm Snapdragon 8Elite is strongly recommended.
the following steps have verified in a fresh&clean ggml-hexagon project with 8Elite based Android phone.

• make sure Android phone equipped with 8Elite is connected properly and can be accessed from Linux via "adb ls /sdcard"
• fetch source code(default branch in this project)
  https://github.com/zhouwg/ggml-hexagon
• setup dev envs(this script will setup local dev envs automatically. directly access to Github and Huggingface is required for this step.)
  ./scripts/build-run-android.sh

• modify following in the scripts/build-run-android.sh for your phone accordingly(used to make sure the appropriate libggmldsp-skel.so matches the test phone:libggmldsp-skelv75.so for 8Gen3, libggmldsp-skelv79.so for 8Elite).
  https://github.com/zhouwg/ggml-hexagon/blob/self-build/scripts/build-run-android.sh#L103-L104
  

• build
  ./scripts/build-run-android.sh build

• upload QNN runtime libs to phone for compare performance between QNN-CPU, QNN-GPU, QNN-NPU, Hexagon-cDSP, ggml
  ./scripts/build-run-android.sh updateqnnlib

• run test-backend-ops on Snapdragon based phone
  ./scripts/build-run-android.sh run_testops

• run LLM inference via cDSP on Snapdragon based phone(the default LLM model is qwen1_5-1_8b-chat-q4_0.gguf)
  ./scripts/build-run-android.sh run_llamacli 3 (the following benchmark data might-be depend on workload of Android OS)

• run ggmlhexagon-benchmark on Snapdragon based phone
  ./scripts/build-run-android.sh run_benchmark ADD 0 (benchmark of QNN-CPU)
  ./scripts/build-run-android.sh run_benchmark ADD 1 (benchmark of QNN-GPU)
  ./scripts/build-run-android.sh run_benchmark ADD 2 (benchmark of QNN-NPU)
  ./scripts/build-run-android.sh run_benchmark ADD 3 (benchmark of Hexagon-cDSP)
  ./scripts/build-run-android.sh run_benchmark ADD 4 (benchmark of ggml)
  ./scripts/build-run-android.sh run_benchmark MUL_MAT 0 (benchmark of QNN-CPU)
  ./scripts/build-run-android.sh run_benchmark MUL_MAT 1 (benchmark of QNN-GPU)
  ./scripts/build-run-android.sh run_benchmark MUL_MAT 2 (benchmark of QNN-NPU)
  ./scripts/build-run-android.sh run_benchmark MUL_MAT 3 (benchmark of Hexagon-cDSP)
  ./scripts/build-run-android.sh run_benchmark MUL_MAT 4 (benchmark of ggml)

• verify speed/performance of mulmat on cDSP side
  ./scripts/build-run-android.sh run_benchmark MUL_MAT 3 0/1/2/3/4/5/6/32/33

• verify accuracy/computation result of mulmat on cDSP side
  ./scripts/build-run-android.sh run_testop MUL_MAT 0/1/2/3/4/5/6/32/33

• observe logs on ARM AP side

adb logcat | grep ggml-hexagon

• observe logs on cDSP side
  • create farf file on phone to observe logs on cDSP side
  • observe logs on cDSP side

adb shell "echo 0x1f >  /data/local/tmp/ggmlhexagon-benchmark.farf"
adb shell cat /data/local/tmp/ggmlhexagon-benchmark.farf
adb logcat -s adsprpc

Todo

• sometimes the same dsp codes can running well as expected on Snapdragon 8Elite based phone, but can't works as expected on other Snapdragon based phone(e.g. 8Gen3).
• I didn't have enough resource to troubleshooting various weird issues on other Snapdragon based phone(e.g. 8Gen3) at the moment although I'll try my best to fix them in the future(v0.98.x).
• performance of fp32 mulmat on cDSP side is still much slower than QNN-CPU, QNN-NPU, the default ggml backend(ARM neon), port ik_mulmat to this project(the author of ik_mulmat is the original author of quantize subsystem in the upstream llama.cpp project).
• it seems there is segment fault with "./scripts/build-run-android.sh run_threadsafety". this is a potential bug in ggml-hexagon.cpp(not happened in the previous test and I don't know how to fix it at themoment, might-be refer to ggml-metal backend:metal : fix thread-safety ggml-org/llama.cpp#14300. this potential bug was introduced/detected by a PR/feature in upstream: llama : add thread safety test ggml-org/llama.cpp#14035
  

[zhouwg]

first time running consistently under 6 seconds(computation result is correct)

[zhouwg]

first time running consistently under 4 seconds(computation result is correct)