ggml-hexagon v0.97(v20250609)

[zhouwg]

Overview

the ggml-hexagon v0.97(ggml-hexagon.cpp v1.10 + ggml-dsp v0.97) is released on 20250609(June 09 2025).

fully source code of ggml-hexagon.cpp: https://github.com/zhouwg/ggml-hexagon/blob/self-build/ggml/src/ggml-hexagon/ggml-hexagon.cpp

reference source code of ggml-dsp: https://github.com/zhouwg/ggml-hexagon/tree/self-build/ggml/src/ggml-hexagon/kernels

test phone is a Snapdragon 8Elite based Android phone and a Snapdragon 8Gen3 based Android phone, test model is qwen1_5-1_8b-chat-q4_0.gguf. QNN SDK is v2.35.0.250530, Hexagon SDK is v6.2.0.1, Android NDK is android-ndk-r28.

computation results in the following test cases are correct(or can be reproduced by third-party or AI experts) otherwise the benchmark data is meaningless.

Supported Qualcomm mobile SoC

#v68 --- Snapdragon 888
#v69 --- Snapdragon 8 Gen1
#v73 --- Snapdragon 8 Gen2
#v75 --- Snapdragon 8 Gen3
#v79 --- Snapdragon 8 Elite(aka Gen4)

• Android phone equipped with Qualcomm Snapdragon 8Elite is strongly recommended, at least 8Gen3 is strongly recommended(although the performance of mulmat on 8Gen3 is really bad at the moment).

• make sure the appropriate libggmldsp-skel.so matches the test phone(libggmldsp-skelv75.so for 8Gen3, libggmldsp-skelv79.so for 8Elite......).

Highlights

• corner case of GGML_OP_ADD in real scenarios(command line Android program or standard Android APP).

• fp32 4096 x 4096 matrix addition on 8Elite: from 15 milliseconds to 6-7 millisecond(self-made optimization approach, faster than my previous self-made optimization approach v0.96, much faster than the example assemble codes in Hexagon SDK), much faster than QNN-CPU, QNN-GPU and QNN-NPU(need to be confirmed by the third-party). is it also faster than the default ggml backend(highly-optimized by AI experts and domain experts), I'm not sure, feedback is greatly welcomed and appreciated(please leave a comment).
  
  
  
  
  
  
  (after add running stamp for the default ggml backend)
  

• fp32 4096 x 4096 matrix mulmat on 8Elite: from 28 seconds to 7-8 seconds(self-made optimization approach, much faster than the example assemble codes in Hexagon SDK). much slower than ggml and QNN-NPU, I personally guess the reason is because of "lack of HMX(Hexagon Matrix eXtension) instructions" and I think I can confirm that the HMX(prebuilt binary libhmx.a in the libggmldsp-skel.so) enabled matrix mulmat is much faster than the HVX enabled matrix mulmat.

• works with llama-cli(the official command line tool), works with test-backend-ops(the official command line tool)

• works with whisper.cpp in a standard Android APP, works with llama.cpp in a standard Android APP

• probe cDSP clock rate when initialize cDSP
  for 8Gen3(0x8c75):

for 8Elite(0x8c79):

PR in upstream llama.cpp

I'd like to make some contributions to the upstream llama.cpp because I learned/got too much from the great project llama.cpp. related source codes in PR-12326 will be updated later.

How to verify

a Linux machine is required for following steps.

• fetch source code(default branch in this project)
  https://github.com/zhouwg/ggml-hexagon

• setup dev env(this script will setup local dev automatically. directly access to Github is required for this step.)
  ./scripts/build-run-android.sh

• modify following in the scripts/build-run-android.sh for your phone accordingly
  https://github.com/zhouwg/ggml-hexagon/blob/self-build/scripts/build-run-android.sh#L69-L76
  

• build
  ./scripts/build-run-android.sh build

• run test-backend-ops on Snapdragon based phone
  ./scripts/build-run-android.sh run_testops

• run LLM inference on Snapdragon based phone(the default LLM model is qwen1_5-1_8b-chat-q4_0.gguf)
  ./scripts/build-run-android.sh run_llamacli

• upload QNN runtime libs to phone for compare performance between QNN-CPU, QNN-GPU, QNN-NPU, Hexagon-cDSP, ggml
  ./scripts/build-run-android.sh updateqnnlib

• run ggmlhexagon-benchmark on Snapdragon based phone
  ./scripts/build-run-android.sh run_benchmark ADD 0 (benchmark of QNN-CPU)
  ./scripts/build-run-android.sh run_benchmark ADD 1 (benchmark of QNN-GPU)
  ./scripts/build-run-android.sh run_benchmark ADD 2 (benchmark of QNN-NPU)
  ./scripts/build-run-android.sh run_benchmark ADD 3 (benchmark of Hexagon-cDSP)
  ./scripts/build-run-android.sh run_benchmark ADD 4 (benchmark of ggml)
  ./scripts/build-run-android.sh run_benchmark MUL_MAT 0 (benchmark of QNN-CPU)
  ./scripts/build-run-android.sh run_benchmark MUL_MAT 1 (benchmark of QNN-GPU)
  ./scripts/build-run-android.sh run_benchmark MUL_MAT 2 (benchmark of QNN-NPU)
  ./scripts/build-run-android.sh run_benchmark MUL_MAT 3 (benchmark of Hexagon-cDSP)
  ./scripts/build-run-android.sh run_benchmark MUL_MAT 4 (benchmark of ggml)

• observe logs on ARM AP side
  adb logcat | grep ggml-hexagon

• create farf file on phone to observe logs on cDSP side

adb shell cat /data/local/tmp/ggmlhexagon-benchmark.farf
0x1f

• observe logs on cDSP side
  adb logcat -s adsprpc

Todo

• sometimes the same dsp codes can running well as expected on Snapdragon 8Elite based phone, but can't works as expected on other Snapdragon based phone(e.g. 8Gen3).
• I didn't have enough resource to troubleshooting various weird issues on other Snapdragon based phone(e.g. 8Gen3) at the moment although I'll try my best to fix them in the future(v0.98).