semi-fully version of discussions-14356(https://github.com/ggml-org/llama.cpp/discussions/14356)(backuped on Jul 12 2025) #64

zhouwg · 2025-07-12T09:29:12Z

zhouwg
Jul 12, 2025
Maintainer

Introduction

Focus on LLM inference on Android Phone/Pad/TV/STB/PC/Intelligent Cockpit Domain in Intelligent Electric Vehicle, especially focus on llama.cpp + Android phone equipped with Qualcomm Snapdragon high-end mobile SoC, such as Snapdragon 8Gen3, 8Elite, 8Elite2(might-be released on 09/2025 or 10/2025)....

Maintain an up-to-date discussion to track the latest development of fundamental technologies, state-of-the-art on-device LLM models, Android APPs, future directions in this rapidly evolving field.

Part 1: Research Papers

on-device inference

HeteroLLM: Accelerating Large Language Model Inference on Mobile SoCs platform with Heterogeneous AI Accelerators [Paper]
LLM prefilling with mllm-NPU [Paper] , https://github.com/UbiquitousLearning/mllm
PowerInfer-2: Fast Large Language Model Inference on a Smartphone [Paper] , https://github.com/SJTU-IPADS/PowerInfer
T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge [Paper] , https://github.com/microsoft/T-MAC
MobileNetV4 -- Universal Models for the Mobile Ecosystem [Paper]
TinyLlama: An Open-Source Small Language Model [Paper]

core technologies

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning [Paper]
Efficient Memory Management for Large Language Model Serving with PagedAttention [Paper]
Mamba: Linear-Time Sequence Modeling with Selective State Spaces [Paper]
QLoRA: Efficient Finetuning of Quantized LLMs [Paper]
GPT-4 Technical Report [Paper]
LLaMA: Open and Efficient Foundation Language Models [Paper]
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness [Paper]
Training language models to follow instructions with human feedback [Paper] ![](https://img.shields.io/badge/arXiv-2022.03-red
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models [Paper]
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale [Paper]
Deep Reinforcement Learning from Human Preferences [Paper]
Distilling the Knowledge in a Neural Network [Paper]

fundamental technologies

Attention Is All You Need [Paper]
Deep Residual Learning for Image Recognition [Paper]
Sequence to Sequence Learning with Neural Networks [Paper]
Efficient Estimation of Word Representations in Vector Space [Paper]
ImageNet Classification with Deep Convolutional Neural Networks [Paper]

Part 2: On-Device Inference framework

Part 3: Hardware acceleration

OpenCL

ggml-opencl: https://github.com/ggml-org/llama.cpp/tree/master/ggml/src/ggml-opencl

Vulkan

ggml-vulkan: https://github.com/ggml-org/llama.cpp/tree/master/ggml/src/ggml-vulkan

Qualcomm Hexagon NPU

QNN solution

aka End-to-End solution, which is similar to the Qualcomm's official AI-Hub solution.

Fig-1: cited from Qualcomm's QNN SDK Manual

QNN solution based fully/semi open source projects:

llama.cpp solution

a specified ggml backend for llama.cpp on Android. Theoretically speaking, model preparation/conversion is not needed in this solution, which is exactly similar to ggml-cuda,ggml-metal,ggml-sycl,ggml-opencl,ggml-cann,ggml-vulkan.

Fig-2: cited from Qualcomm's Hexagon SDK Programmers_Reference_Manual

libggml-hexagons.so(running on ARM CPU side): ggml-hexagon.cpp + stub.c

libggmldsp_skel.so(running on DSP side): ggmldsp.c + skel.c + add.c + mulmat.c + ... ( developers and AI experts can operate hardware directly on DSP side)

block-beta
columns 1


block:ggml_backend
  ggml_backend["libggml-hexagon.so"]
  style ggml_backend    fill:#3c3,color:#000,stroke:#000

 block:ggmlbackends
   ggml_cpu["ggml-hexagon.cpp"]
   ggml_metal["stub.c"]
   
   style ggml_cpu       fill:#888,color:#000,stroke:#000
   style ggml_metal     fill:#888,color:#000,stroke:#000

  end
end

block:ggml_backendsubsystem
  ggml_backendsubsystem["ARM CPU"]
  style ggml_backendsubsystem fill:#3c3,color:#000,stroke:#000
end


block:ggml_backend1
ggml_backend1["libggmlop-skel.so"]
  style ggml_backend1    fill:#3c3,color:#000,stroke:#000

block:ggmlbackends1
 ggml_cpu1["ggml-dsp.c"]
   ggml_metal1["skel.c"]
   ggml_sycl1["add.c"]
   ggml_cuda1["mulmat.c"]
   ggml_ops["entry.c"]
   ggml_ops["..."]

   style ggml_cpu1       fill:#888,color:#000,stroke:#000
   style ggml_metal1     fill:#888,color:#000,stroke:#000
   style ggml_sycl1      fill:#888,color:#000,stroke:#000
   style ggml_cuda1      fill:#888,color:#000,stroke:#000
 style ggml_ops      fill:#888,color:#000,stroke:#000
 ggml_metal --> ggml_metal1
  end
end


block:ggml_backendsubsystem1
  ggml_backendsubsystem1["Hexagon NPU"]
  style ggml_backendsubsystem1 fill:#3c3,color:#000,stroke:#000

ggml_backendsubsystem --> ggml_backendsubsystem1
ggml_backendsubsystem1 --> ggml_backendsubsystem
end

Fig-3: self-made

llama.cpp solution based open source projects:

https://github.com/zhouwg/ggml-hexagon, the first original llama.cpp solution which launched on 03/2024(the initial version was reverse engineered from Qualcomm's codes in executorch), this llama.cpp-based solution is a simple, concise, easy-to-extend, easy-to-troubleshoot, easy-to-maintain solution. PR:PR: Refine ggml-hexagon backend(Qualcomm Hexagon NPU backend) for latest ggml,whisper.cpp,llama.cpp ggml-org/llama.cpp#12326
https://github.com/chraac/llama.cpp (hard-forked from zhouwg's initial version). PR:[WIP]backend: Integrating QNN (Qualcomm AI Engine Direct) as a dedicated backend for Qualcomm NPUs ggml-org/llama.cpp#12063

Part 4: LLamacpp based Android APPs

llama.cpp based Androd APPs

https://github.com/a-ghorbani/pocketpal-ai
https://github.com/Vali-98/ChatterUI
https://github.com/shubham0204/SmolChat-Android
https://github.com/kantv-ai/kantv (this is an on-device AI learning&hobby project, focus on Android phone)

Part 5: State-of-the-art on-device LLM models

Google gemma-3n:

gemma-3n-E4B-it-litert-preview: https://huggingface.co/google/gemma-3n-E4B-it-litert-preview
gemma-3n-E2B-it: https://huggingface.co/google/gemma-3n-E2B-it
gemma-3n-E4B-it: https://huggingface.co/google/gemma-3n-E4B-it
gemma-3n-E2B-it GGUF model: https://huggingface.co/ggml-org/gemma-3n-E2B-it-GGUF/tree/main (verified on Snapdragon 8Elite and 8Gen3, speed in 8Elite is good, speed in 8Gen3 is not good)
gemma-3n-E4B-it GGUF model: https://huggingface.co/ggml-org/gemma-3n-E4B-it-GGUF/tree/main

HuggingFace SmolVLM:

SmolVLM-256M-Instruct:https://huggingface.co/HuggingFaceTB/SmolVLM2-256M-Video-Instruct
SmolVLM-256M-Instruct GGUF model: https://huggingface.co/ggml-org/SmolVLM2-256M-Video-Instruct-GGUF/tree/main (verified on Snapdragon 8Gen3 and Snapdragon 8Elite, unbelievable experience on Android phone)

Part 6: Mobile SoC References

Qualcomm: https://docs.qualcomm.com/bundle/publicresource/topics/80-63442-50/introduction.html
MTK: https://neuropilot.mediatek.com/resources/public/npexpress/en/docs/npexpress
Samsung Exynos: TBD
Huawei Hisilicon: TBD
UniSoC: TBD
Xiaomi Xuanjie: TBD
Google Tensor: TBD

Part 7: Acknowledgement

GGML(as a programmer whom believe the philosophy of "simple is beautiful" which comes from the great Unix, I personally/always think it's another FFmpeg since 03/2024 and I like it very much although there are many AI inference frameworks from big IT tech giants)

ASR engine whisper.cpp
LLM engine llama.cpp
CV engine opencv-mobile
MTMD(multimodal) engine MTMD subsystem in llama.cpp

help on 04/04/2024 from a software engineer at Quic(Qualcomm Innovation Center)
reminder/help on 03/19/2025 from a staff tech expert at Qualcomm

"QNN is not the right solution here"

https://mp.weixin.qq.com/s/1lUSlc0tvEWLuOFOP0WkUA (zh)
https://github.com/ngxson/smolvlm-realtime-webcam

Disclaimer

I'm an experienced full-stack(from kernel, framework to APP) Android programmer, but I don't know anything about hardcore AI tech.

Correction and guidance from developers/domain experts/AI experts are greatly welcomed and appreciated.

zhouwg · 2025-07-12T09:30:22Z

zhouwg
Jul 12, 2025
Maintainer Author

[Time: Pls see timestamp in the screenshots]

Benchmark data of project ggml-hexagon

benchmark data of ggml-hexagon(PR in the llama.cpp community:ggml-org#12326) on Android phone equipped with Qualcomm Snapdragon 8Elite(one of the most advanced mobile SoC on our planet at the moment):

./scripts/build-run-android.sh run_benchmark ADD 0/1/2/3/4

./scripts/build-run-android.sh run_benchmark MUL_MAT 0/1/2/3/4

unfortunately the following computational result is incorrect.

How to verify on Android phone

Overview

a Linux machine is required for following steps, Android phone equipped with Qualcomm Snapdragon 8Elite is strongly recommended.
Android phone equipped with Qualcomm Snapdragon 8Elite is strongly recommended, because

sometimes the same dsp codes can running well as expected on Snapdragon 8Elite based phone, but can't works as expected on other Snapdragon based phone(e.g. 8Gen3).
8Elite's memory subsystem support for LP-DDR5x memory, up to 5300 MHz; 8Gen3's memory subsystem support for LP-DDR5x memory, up to 4800 MHz.
DSP clock rate on 8Gen3 is slower than DSP clock rate on 8Elite.

for 8Gen3(0x8c75):

for 8Elite(0x8c79):

Steps

the following steps have verified in a fresh&clean ggml-hexagon clone with 8Elite based Android phone.

make sure Android phone equipped with 8Elite is connected properly and can be accessed from Linux via "adb ls /sdcard"
fetch source code(default branch in this project)
https://github.com/zhouwg/ggml-hexagon
setup dev envs(this script will setup local dev envs automatically. directly access to Github and Huggingface is required for this step.)
./scripts/build-run-android.sh

modify following in the scripts/build-run-android.sh for your phone accordingly(used to make sure the appropriate libggmldsp-skel.so matches the test phone:libggmldsp-skelv75.so for 8Gen3, libggmldsp-skelv79.so for 8Elite).
https://github.com/zhouwg/ggml-hexagon/blob/self-build/scripts/build-run-android.sh#L103-L104
build
./scripts/build-run-android.sh build
upload QNN runtime libs to phone for compare performance between QNN-CPU, QNN-GPU, QNN-NPU, Hexagon-cDSP, ggml
./scripts/build-run-android.sh updateqnnlib
run test-backend-ops on Snapdragon based phone
./scripts/build-run-android.sh run_testops
run LLM inference via cDSP on Snapdragon based phone(the default LLM model is qwen1_5-1_8b-chat-q4_0.gguf)
./scripts/build-run-android.sh run_llamacli 3 (the following benchmark data might-be depend on workload of Android OS)

run ggmlhexagon-benchmark on Snapdragon based phone
./scripts/build-run-android.sh run_benchmark ADD 0 (benchmark of QNN-CPU)
./scripts/build-run-android.sh run_benchmark ADD 1 (benchmark of QNN-GPU)
./scripts/build-run-android.sh run_benchmark ADD 2 (benchmark of QNN-NPU)
./scripts/build-run-android.sh run_benchmark ADD 3 (benchmark of Hexagon-cDSP)
./scripts/build-run-android.sh run_benchmark ADD 4 (benchmark of ggml)
./scripts/build-run-android.sh run_benchmark MUL_MAT 0 (benchmark of QNN-CPU)
./scripts/build-run-android.sh run_benchmark MUL_MAT 1 (benchmark of QNN-GPU)
./scripts/build-run-android.sh run_benchmark MUL_MAT 2 (benchmark of QNN-NPU)
./scripts/build-run-android.sh run_benchmark MUL_MAT 3 (benchmark of Hexagon-cDSP)
./scripts/build-run-android.sh run_benchmark MUL_MAT 4 (benchmark of ggml)
verify speed/performance of mulmat on cDSP side
./scripts/build-run-android.sh run_benchmark MUL_MAT 3 0/1/2/3/4/5/6/32/33
verify accuracy/computation result of mulmat on cDSP side
./scripts/build-run-android.sh run_testop MUL_MAT 0/1/2/3/4/5/6/32/33
observe logs on ARM AP side

adb logcat | grep ggml-hexagon

observe logs on cDSP side
- create farf file on phone to observe logs on cDSP side
- observe logs on cDSP side

adb shell "echo 0x1f >  /data/local/tmp/ggmlhexagon-benchmark.farf"
adb shell cat /data/local/tmp/ggmlhexagon-benchmark.farf
adb logcat -s adsprpc

What does 0/1/2/3/4 mean in the script?

0: offload ggml op to QNN_CPU via QNN SDK API(highly-designed and uniform high-level API provided by Qualcomm)
1: offload ggml op to QNN_GPU via QNN SDK API(highly-designed and uniform high-level API provided by Qualcomm)
2: offload ggml op to QNN_NPU via QNN SDK API(highly-designed and uniform high-level API provided by Qualcomm)
3: offload ggml op to cDSP(aka Hexagon NPU) directly via Hexagon SDK API(low-level API provided by Qualcomm, can operate hardware directly)
4: the default ggml backend, or the ARM neon codes in the llama.cpp.

HVX and HMX

HVX: Hexagon Vector eXtension
HMX: Hexagon Matrix eXtension (HMX instructions is not opened at the moment)

I'm confused about QNN-NPU and Hexagon-NPU, Are they different implementations of the same hardware?

there some subtle differences between QNN-NPU,QNN-HTP,Hexagon-NPU,Hexagon-cDSP, pls refer to:

Fig:cited from Qualcomm's official website

as my personal understanding, in the all:
QNN-HTP = cDSP + HVX + HMX, provided in Qualcomm's QNN SDK, can be found in PR-12326
QNN-NPU = QNN-HTP,
QNN-DSP = cDSP + HVX, provided in Qualcomm's QNN SDK
Hexagon-cDSP = cDSP + HVX(one of the approaches in PR-12326)
Hexagon-NPU = QNN-HTP

both QNN-HTP, QNN-NPU, QNN-DSP, Hexagon-cDSP, Hexagon-NPU are all NPU-based solution.

Technical merits in PR-12326:

implementation in PR-12326 or my forked llama.cpp project is simple, concise, easy-to-extend(AI experts can add other GGML operators very easily), easy-to-maintain, easy-to-troubleshoot. the key-point here is that the tiny ggml-dsp on Hexagon cDSP side(aka the existing implementation of framework of hexagon kernels on cDSP side) is completely ported/borrowed from the original ggml, which I already explained/clarified in PR-12326 on 04/2025. in other words, the approach/implementation in PR-12326 or my forked llama.cpp project is standing on the shoulders of giants.
PR-12326 or my forked llama.cpp project not only support QNN-based hardware acceleration (that's QNN-CPU, QNN-GPU, QNN-NPU) but also support offload op to Hexagon-NPU directly(that's Hexagon-cDSP). at the same time, it can easily compare performance between QNN-CPU,QNN-GPU,QNN-NPU, Hexagon-cDSP, the default ggml backend.
benchmark data and steps of reproduce the benchmark data in PR-12326 or my forked llama.cpp project can be easily reproduced by third-party on a 8Elite based Android phone.
compare to ggml-opencl, ggml-vulkan on Snapdragon based Android phone, PR-12326 or my forked llama.cpp project is another feasible backend although it only support GGML_OP_ADD and GGML_OP_MULMAT and the performance of mulmat is not the best at the moment.

Updated benchmark data of fp32 mulmat on Hexagon cDSP(actually updated on 06/27/2025, merged in this comment on Jun 8 2025)

benchmark data of ggml-hexagon(PR in the llama.cpp community:ggml-org#12326) on Android phone equipped with Qualcomm Snapdragon 8Elite(one of the most advanced mobile SoC on our planet at the moment):

how to verify on Android phone: same as above.

performance of fp32 4096x4096 mulmat on cDSP:

before 05/27/2025: about 28 seconds.
after relaunched the dev activity of project ggml-hexagon since 05/27/2025
06/09/2025: about 7-8 seconds
06/25/2025: about 6-8 seconds
06/27/2025: about 3.4-4.2seconds

0 replies

zhouwg · 2025-07-12T09:30:49Z

zhouwg
Jul 12, 2025
Maintainer Author

[Time: Pls see timestamp in the screenshots]

Overall evaluation of ggml-opencl,ggml-vulkan,ggml-hexagon and the default ggml backend on Android phone

Introduction

As well known, there are ggml-opencl,ggml-vulkan backend in the llama.cpp project. In this technical post, I'd like to introduce an overall evaluation of ggml-opencl, ggml-vulkan, ggml-hexagon in PR-12326 (which launched on 03/2024 and re-launched since 01/29/2025 and re-re-launched since 05/27/2025), and the default ggml backend on Snapdragon 8Elite based Android phone. Some background knowledge about ggml-hexagon can be found in the entry point of this discussion and the above technical post.

In order to provide a fair and straightforward comparison of the performance of these backends, this evaluation will use llama-cli instead of llama-bench which is preferred by AI experts.

As well known, Android phone is a typcial resource constrained device. Accordingly, qwen1_5-1_8b.gguf will be used in this evaluation because the weights in qwen1_5-1_8b.gguf is suitable for all these 4 backends on Android phone.

For simplify workflow of developing/improving/testing ggml-opencl, ggml-vulkan and ggml-hexagon on Android phone(especially Snapdragon high-end mobile SoC based Android phone), following build scripts are provided:

How to build ggml-opencl backend for Android device on Linux

https://github.com/zhouwg/ggml-hexagon/blob/self-build/scripts/build-run-ggmlopencl-android.sh

Details

#!/usr/bin/env bash
#
# Copyright (c) Qualcomm(https://www.qualcomm.com/developer/blog/2024/11/introducing-new-opn-cl-gpu-backend-llama-cpp-for-qualcomm-adreno-gpu, original-author)
# Copyright (c) rmatif(https://github.com/rmatif, core-author)
# Copyright (c) zhouwg(https://github.com/zhouwg, co-author)
#
# build llama.cpp + ggml-opencl backend on Linux for Android phone
# this script will setup local dev envs automatically
#
set -e

######## part-1: don't modify contents in this part ########

PWD=`pwd`
PROJECT_HOME_PATH=`pwd`
PROJECT_ROOT_PATH=${PROJECT_HOME_PATH}
HOST_CPU_COUNTS=`cat /proc/cpuinfo | grep "processor" | wc | awk '{print int($1)}'`

#running path on Android phone
REMOTE_PATH=/data/local/tmp

#Android NDK can be found at:
#https://developer.android.com/ndk/downloads
ANDROID_PLATFORM=android-34
ANDROID_NDK_VERSION=r28
ANDROID_NDK_NAME=android-ndk-${ANDROID_NDK_VERSION}
ANDROID_NDK_FULLNAME=${ANDROID_NDK_NAME}-linux.zip
ANDROID_NDK=${PROJECT_ROOT_PATH}/prebuilts/${ANDROID_NDK_NAME}

# --- Define NDK paths based on the absolute SDK path ---
NDK_TOOLCHAIN_SYSROOT_INCLUDE_PATH="${ANDROID_NDK}/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include"
NDK_TOOLCHAIN_SYSROOT_ARM64_LIB_PATH="${ANDROID_NDK}/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/aarch64-linux-android"

#OpenCL Headers can be found at:
#https://https://github.com/KhronosGroup/OpenCL-Headers
OPENCL_SDK_URL=https://github.com/KhronosGroup/OpenCL-Headers
OPENCL_SDK_PATH=${PROJECT_ROOT_PATH}/prebuilts/OpenCL_SDK
OPENCL_HEADERS_PATH=${OPENCL_SDK_PATH}/OpenCL-Headers

#running_params=" -ngl 99 -t 4 -n 256 --no-warmup -fa 1 "
running_params=" -ngl 99 -t 4 -n 256 --no-warmup "

######## part-2: contents in this part can be modified ########
PROMPT_STRING="introduce the movie Once Upon a Time in America briefly.\n"
#for llama-cli, 1.1 GiB, will be downloaded automatically via this script when running this script at the first time
TEST_MODEL_NAME=/sdcard/qwen1_5-1_8b-chat-q4_0.gguf
#for llama-cli, 4.5 GiB, will be downloadded automatically via this script when running this script at the first time
TEST_MODEL_NAME=/sdcard/gemma-3n-E2B-it-Q8_0.gguf

#for llama-bench, 4.5 GiB, will be downloadded automatically via this script when running this script at the first time
GGUF_MODEL_NAME=/sdcard/gemma-3n-E2B-it-Q8_0.gguf
#for llama-bench, 1.12 GiB, will be downloadded automatically via this script when running this script at the first time
GGUF_MODEL_NAME=/sdcard/qwen1_5-1_8b-chat-q4_0.gguf

######## part-3: utilities and functions ########

function dump_vars()
{
    echo -e "ANDROID_NDK:          ${ANDROID_NDK}"
    echo -e "OPENCL_SDK_PATH:      ${OPENCL_SDK_PATH}"
}


function show_pwd()
{
    echo -e "current working path:$(pwd)\n"
}


function check_command_in_host()
{
    set +e
    cmd=$1
    ls /usr/bin/${cmd}
    if [ $? -eq 0 ]; then
        #printf "${cmd} already exist on host machine\n"
        echo ""
    else
        printf "${cmd} not exist on host machine, pls install command line utility ${cmd} firstly and accordingly\n"
        exit 1
    fi
    set -e
}


function check_commands_in_host()
{
    check_command_in_host wget
    check_command_in_host git
    check_command_in_host ninja
}


function check_android_phone()
{
    adb shell ls /bin/ls
    if [ ! $? -eq 0 ]; then
        printf "pls check Android phone is connected properly\n"
        exit 1
    fi
}


function check_and_download_opencl_sdk()
{
    is_opencl_sdk_exist=1

    if [ ! -d ${OPENCL_SDK_PATH} ]; then
        echo -e "OPENCL_SDK_PATH ${OPENCL_SDK_PATH} not exist, download it from ${OPENCL_SDK_URL}...\n"
        is_opencl_sdk_exist=0
    fi
    if [ ! -f ${NDK_TOOLCHAIN_SYSROOT_ARM64_LIB_PATH}/libOpenCL.so ]; then
        echo -e "${NDK_TOOLCHAIN_SYSROOT_ARM64_LIB_PATH}/libOpenCL.so not exist...\n"
        is_opencl_sdk_exist=0
    fi

    if [ ${is_opencl_sdk_exist} -eq 0 ]; then
        mkdir -p ${OPENCL_SDK_PATH}
        cd ${OPENCL_SDK_PATH}

        if [ ! -d OpenCL-Headers ]; then
            echo "Cloning OpenCL-Headers..."
            git clone https://github.com/KhronosGroup/OpenCL-Headers
            if [ $? -ne 0 ]; then
                printf "failed to download OpenCL-Headers to %s \n" "${OPENCL_SDK_PATH}"
                exit 1
            fi
        fi
        cd ${PROJECT_ROOT_PATH}/prebuilts/OpenCL_SDK/OpenCL-Headers
        printf "Copying OpenCL Headers to Android NDK sysroot include: ${NDK_TOOLCHAIN_SYSROOT_INCLUDE_PATH}"
        mkdir -p ${NDK_TOOLCHAIN_SYSROOT_INCLUDE_PATH}
        /bin/cp -r -fv CL ${NDK_TOOLCHAIN_SYSROOT_INCLUDE_PATH}

        cd ${PROJECT_ROOT_PATH}/prebuilts/OpenCL_SDK
        if [ ! -d OpenCL-ICD-Loader ]; then
            echo "Cloning OpenCL-ICD-Loader..."
            git clone https://github.com/KhronosGroup/OpenCL-ICD-Loader
            if [ $? -ne 0 ]; then
                printf "failed to download OpenCL-ICD-Loader to %s \n" "${OPENCL_SDK_PATH}"
                exit 1
            fi
        fi
        cd ${PROJECT_ROOT_PATH}/prebuilts/OpenCL_SDK/OpenCL-ICD-Loader
        mkdir -p build
        cd build
        cmake .. -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_TOOLCHAIN_FILE=${ANDROID_NDK}/build/cmake/android.toolchain.cmake -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=latest -DANDROID_STL=c++_shared -DOPENCL_ICD_LOADER_HEADERS_DIR=${NDK_TOOLCHAIN_SYSROOT_INCLUDE_PATH}
        echo "Building OpenCL-ICD-Loader with ninjia..."
        ninja
        if [ $? -ne 0 ]; then
            printf "failed to build OpenCL-ICD-Loader\n"
            exit 1
        fi
        mkdir -p ${NDK_TOOLCHAIN_SYSROOT_ARM64_LIB_PATH}
        /bin/cp -fv libOpenCL.so ${NDK_TOOLCHAIN_SYSROOT_ARM64_LIB_PATH}

        echo "OpenCL components setup complete"
        echo "OpenCL Headers are in: ${NDK_TOOLCHAIN_SYSROOT_INCLUDE_PATH}/CL"
        echo "libOpenCL.so is in:    ${NDK_TOOLCHAIN_SYSROOT_ARM64_LIB_PATH}/libOpenCL.so"

        cd ${PROJECT_ROOT_PATH}
    else
        printf "OpenCL SDK already exist:    ${OPENCL_SDK_PATH} \n\n"
    fi
}


function check_and_download_ndk()
{
    is_android_ndk_exist=1

    if [ ! -d ${ANDROID_NDK} ]; then
        is_android_ndk_exist=0
    fi

    if [ ! -f ${ANDROID_NDK}/build/cmake/android.toolchain.cmake ]; then
        is_android_ndk_exist=0
    fi

    if [ ${is_android_ndk_exist} -eq 0 ]; then

        if [ ! -f ${PROJECT_ROOT_PATH}/prebuilts/${ANDROID_NDK_FULLNAME} ]; then
            wget --no-config --quiet --show-progress -O ${PROJECT_ROOT_PATH}/prebuilts/${ANDROID_NDK_FULLNAME} https://dl.google.com/android/repository/${ANDROID_NDK_FULLNAME}
        fi

        cd ${PROJECT_ROOT_PATH}/prebuilts/
        unzip ${ANDROID_NDK_FULLNAME}

        if [ $? -ne 0 ]; then
            printf "failed to download Android NDK to %s \n" "${ANDROID_NDK}"
            exit 1
        fi
        cd ${PROJECT_ROOT_PATH}

        printf "Android NDK saved to ${ANDROID_NDK} \n\n"
    else
        printf "Android NDK already exist:   ${ANDROID_NDK} \n\n"
    fi
}


function build_arm64
{
    cmake -H. -B./out/ggmlopencl-android -DCMAKE_BUILD_TYPE=Release -DGGML_OPENMP=OFF -DCMAKE_TOOLCHAIN_FILE=${ANDROID_NDK}/build/cmake/android.toolchain.cmake -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=latest -DCMAKE_C_FLAGS=-march=armv8.7-a -DGGML_OPENCL=ON -DLLAMA_CURL=OFF
    cd out/ggmlopencl-android
    make -j${HOST_CPU_COUNTS}
    show_pwd

    cd -
}


function build_arm64_debug
{
    cmake -H. -B./out/ggmlopencl-android -DCMAKE_BUILD_TYPE=Debug -DGGML_OPENMP=OFF -DCMAKE_TOOLCHAIN_FILE=${ANDROID_NDK}/build/cmake/android.toolchain.cmake -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=latest -DCMAKE_C_FLAGS=-march=armv8.7-a -DGGML_OPENCL=ON -DLLAMA_CURL=OFF
    cd out/ggmlopencl-android
    make -j${HOST_CPU_COUNTS}
    show_pwd

    cd -
}


function remove_temp_dir()
{
    if [ -d out/ggmlopencl-android ]; then
        echo "remove out/ggmlopencl-android directory in `pwd`"
        rm -rf out/ggmlopencl-android
    fi
}


function build_ggml_opencl()
{
    show_pwd
    check_and_download_ndk
    check_and_download_opencl_sdk
    dump_vars
    remove_temp_dir
    build_arm64
}


function build_ggml_opencl_debug()
{
    show_pwd
    check_and_download_ndk
    check_and_download_opencl_sdk
    dump_vars
    remove_temp_dir
    build_arm64_debug
}


function check_and_download_model()
{
    set +e

    model_name=$1
    model_url=$2

    adb shell ls /sdcard/${model_name}
    if [ $? -eq 0 ]; then
        printf "the prebuild LLM model ${model_name} already exist on Android phone\n"
    else
        printf "the prebuild LLM model ${model_name} not exist on Android phone\n"
        wget --no-config --quiet --show-progress -O ${PROJECT_ROOT_PATH}/models/${model_name} ${model_url}
        adb push ${PROJECT_ROOT_PATH}/models/${model_name} /sdcard/
    fi

    set -e
}


function check_prebuilt_models()
{
    set +e

    check_and_download_model qwen1_5-1_8b-chat-q4_0.gguf https://huggingface.co/Qwen/Qwen1.5-1.8B-Chat-GGUF/resolve/main/qwen1_5-1_8b-chat-q4_0.gguf
    check_and_download_model gemma-3n-E2B-it-Q8_0.gguf https://huggingface.co/ggml-org/gemma-3n-E2B-it-GGUF/resolve/main/gemma-3n-E2B-it-Q8_0.gguf

    set -e
}


function prepare_run_on_phone()
{
    if [ $# != 1 ]; then
        print "invalid param"
        return
    fi
    program=$1

    check_prebuilt_models

    if [ -f ./out/ggmlopencl-android/bin/libggml-cpu.so ]; then
        adb push ./out/ggmlopencl-android/bin/*.so ${REMOTE_PATH}/
    fi
    adb push ./out/ggmlopencl-android/bin/${program} ${REMOTE_PATH}/

    adb shell ls -l ${REMOTE_PATH}/libggml-*.so

    adb shell chmod +x ${REMOTE_PATH}/${program}
}


function run_llamacli()
{
    prepare_run_on_phone llama-cli

    echo "${REMOTE_PATH}/llama-cli ${running_params} -no-cnv -m ${TEST_MODEL_NAME} -p \"${PROMPT_STRING}\""
    adb shell "cd ${REMOTE_PATH} \
               && export LD_LIBRARY_PATH=${REMOTE_PATH} \
               && ${REMOTE_PATH}/llama-cli ${running_params} -no-cnv -m ${TEST_MODEL_NAME} -p \"${PROMPT_STRING}\""

}


function run_llamabench()
{
    prepare_run_on_phone llama-bench

    echo "adb shell \"cd ${REMOTE_PATH} \
               && export LD_LIBRARY_PATH=${REMOTE_PATH} \
               && ${REMOTE_PATH}/llama-bench ${running_params} -m ${GGUF_MODEL_NAME}\""
    echo "${REMOTE_PATH}/llama-bench ${running_params} -m ${GGUF_MODEL_NAME}"

    adb shell "cd ${REMOTE_PATH} \
               && export LD_LIBRARY_PATH=${REMOTE_PATH} \
               && ${REMOTE_PATH}/llama-bench ${running_params} -m ${GGUF_MODEL_NAME}"

}


function run_threadsafety()
{
    prepare_run_on_phone test-thread-safety

    echo "${REMOTE_PATH}/test-thread-safety -np 2 -m ${GGUF_MODEL_NAME} -p \"hello,world\" -n 256 -ngl 99 "
    adb shell "cd ${REMOTE_PATH} \
               && export LD_LIBRARY_PATH=${REMOTE_PATH} \
               && ${REMOTE_PATH}/test-thread-safety -np 1 -m ${GGUF_MODEL_NAME} -p \"hello,world\" -n 256 -ngl 99 "

}


function run_test-ops()
{
    prepare_run_on_phone test-backend-ops

    adb shell "cd ${REMOTE_PATH} \
               && export LD_LIBRARY_PATH=${REMOTE_PATH} \
               && ${REMOTE_PATH}/test-backend-ops test"

}


function run_test-op()
{
    prepare_run_on_phone test-backend-ops

    echo "adb shell cd ${REMOTE_PATH} \
               && export LD_LIBRARY_PATH=${REMOTE_PATH} \
               && ${REMOTE_PATH}/test-backend-ops test -o ${opname}"

    echo "\n"
    adb shell "cd ${REMOTE_PATH} \
               && export LD_LIBRARY_PATH=${REMOTE_PATH} \
               && ${REMOTE_PATH}/test-backend-ops test -o ${opname}"

}


function run_perf-op()
{
    prepare_run_on_phone test-backend-ops

    echo "adb shell cd ${REMOTE_PATH} \
               && export LD_LIBRARY_PATH=${REMOTE_PATH} \
               && ${REMOTE_PATH}/test-backend-ops perf -o ${opname}"

    echo "\n"
    adb shell "cd ${REMOTE_PATH} \
               && export LD_LIBRARY_PATH=${REMOTE_PATH} \
               && ${REMOTE_PATH}/test-backend-ops perf -o ${opname}"

}


function show_usage()
{
    echo -e "\n\n\n"
    echo "Usage:"
    echo "  $0 help"
    echo "  $0 build"
    echo "  $0 build_debug"
    echo "  $0 run_testops"
    echo "  $0 run_testop     ADD/MUL_MAT                                                    (verify accuracy    of ADD/MUL_MAT)"
    echo "  $0 run_perfop     ADD/MUL_MAT                                                    (verify performance of ADD/MUL_MAT)"
    echo "  $0 run_llamacli"
    echo "  $0 run_llamabench"
    echo "  $0 run_threadsafety"
    echo -e "\n\n\n"
}


######## part-4: entry point  ########

show_pwd

check_commands_in_host
check_android_phone
check_and_download_ndk
check_and_download_opencl_sdk
check_prebuilt_models

if [ $# == 0 ]; then
    show_usage
    exit 1
elif [ $# == 1 ]; then
    if [ "$1" == "-h" ]; then
        show_usage
        exit 1
    elif [ "$1" == "help" ]; then
        show_usage
        exit 1
    elif [ "$1" == "build" ]; then
        build_ggml_opencl
        exit 0
    elif [ "$1" == "build_debug" ]; then
        build_ggml_opencl_debug
        exit 0
    elif [ "$1" == "run_testops" ]; then
        run_test-ops
        exit 0
    elif [ "$1" == "run_llamacli" ]; then
        run_llamacli
        exit 0
    elif [ "$1" == "run_llamabench" ]; then
        run_llamabench
        exit 0
    elif [ "$1" == "run_threadsafety" ]; then
        run_threadsafety
        exit 0
    else
        show_usage
        exit 1
    fi
elif [ $# == 2 ]; then
    if [ "$1" == "run_testop" ]; then
        opname=$2
        run_test-op
        exit 0
    elif [ "$1" == "run_perfop" ]; then
        opname=$2
        run_perf-op
        exit 1
    fi
else
    show_usage
    exit 1
fi

How to build ggml-vulkan backend for Android device on Linux

https://github.com/zhouwg/ggml-hexagon/blob/self-build/scripts/build-run-ggmlvulkan-android.sh

Details

#!/usr/bin/env bash
#
#
# build llama.cpp + ggml-vulkan backend on Linux for Android phone
# this script will setup local dev envs automatically
#
set -e

######## part-1: don't modify contents in this part ########

PWD=`pwd`
PROJECT_HOME_PATH=`pwd`
PROJECT_ROOT_PATH=${PROJECT_HOME_PATH}
HOST_CPU_COUNTS=`cat /proc/cpuinfo | grep "processor" | wc | awk '{print int($1)}'`

#running path on Android phone
REMOTE_PATH=/data/local/tmp

#Android NDK can be found at:
#https://developer.android.com/ndk/downloads
ANDROID_PLATFORM=android-34
ANDROID_NDK_VERSION=r28
ANDROID_NDK_NAME=android-ndk-${ANDROID_NDK_VERSION}
ANDROID_NDK_FULLNAME=${ANDROID_NDK_NAME}-linux.zip
ANDROID_NDK=${PROJECT_ROOT_PATH}/prebuilts/${ANDROID_NDK_NAME}

#Vulkan-Headers can be found at:
#https://github.com/KhronosGroup/Vulkan-Headers
VULKAN_SDK_URL=https://github.com/KhronosGroup/Vulkan-Headers
VULKAN_SDK_PATH=${PROJECT_ROOT_PATH}/prebuilts/Vulkan_SDK
VULKAN_HEADERS_PATH=${VULKAN_SDK_PATH}/Vulkan-Headers

#running_params=" -ngl 99 -t 4 -n 256 --no-warmup -fa 1 "
running_params=" -ngl 99 -t 4 -n 256 --no-warmup "

######## part-2: contents in this part can be modified ########
PROMPT_STRING="introduce the movie Once Upon a Time in America briefly.\n"
#for llama-cli, 1.1 GiB, will be downloaded automatically via this script when running this script at the first time
TEST_MODEL_NAME=/sdcard/qwen1_5-1_8b-chat-q4_0.gguf
#for llama-cli, 4.5 GiB, will be downloadded automatically via this script when running this script at the first time
TEST_MODEL_NAME=/sdcard/gemma-3n-E2B-it-Q8_0.gguf

#for llama-bench, 4.5 GiB, will be downloadded automatically via this script when running this script at the first time
GGUF_MODEL_NAME=/sdcard/gemma-3n-E2B-it-Q8_0.gguf
#for llama-bench, 1.12 GiB, will be downloadded automatically via this script when running this script at the first time
GGUF_MODEL_NAME=/sdcard/qwen1_5-1_8b-chat-q4_0.gguf

######## part-3: utilities and functions ########

function dump_vars()
{
    echo -e "ANDROID_NDK:          ${ANDROID_NDK}"
    echo -e "VULKAN_SDK_PATH:      ${VULKAN_SDK_PATH}"
}


function show_pwd()
{
    echo -e "current working path:$(pwd)\n"
}


function check_command_in_host()
{
    set +e
    cmd=$1
    ls /usr/bin/${cmd}
    if [ $? -eq 0 ]; then
        #printf "${cmd} already exist on host machine\n"
        echo ""
    else
        printf "${cmd} not exist on host machine, pls install command line utility ${cmd} firstly and accordingly\n"
        exit 1
    fi
    set -e
}


function check_commands_in_host()
{
    check_command_in_host wget
    check_command_in_host git
}


function check_android_phone()
{
    adb shell ls /bin/ls
    if [ ! $? -eq 0 ]; then
        printf "pls check Android phone is connected properly\n"
        exit 1
    fi
}


function check_and_download_vulkan_sdk()
{
    is_vulkan_sdk_exist=1

    if [ ! -d ${VULKAN_SDK_PATH} ]; then
        echo -e "VULKAN_SDK_PATH ${VULKAN_SDK_PATH} not exist, download it from ${VULKAN_SDK_URL}...\n"
        is_vulkan_sdk_exist=0
    fi

    if [ ${is_vulkan_sdk_exist} -eq 0 ]; then
        mkdir -p ${VULKAN_SDK_PATH}
        cd ${VULKAN_SDK_PATH}

        if [ ! -d Vulkan-Headers ]; then
            echo "Cloning Vulkan-Headers..."
            git clone https://github.com/KhronosGroup/Vulkan-Headers
            if [ $? -ne 0 ]; then
                printf "failed to download Vulkan-Headers to %s \n" "${VULKAN_SDK_PATH}"
                exit 1
            fi
        fi

        echo "Vulkan components setup complete"
        echo "Vulkan Headers are in: ${VULKAN_HEADERS_PATH}"

        cd ${PROJECT_ROOT_PATH}
    else
        printf "Vulkan SDK already exist:    ${VULKAN_SDK_PATH} \n\n"
    fi
}


function check_and_download_ndk()
{
    is_android_ndk_exist=1

    if [ ! -d ${ANDROID_NDK} ]; then
        is_android_ndk_exist=0
    fi

    if [ ! -f ${ANDROID_NDK}/build/cmake/android.toolchain.cmake ]; then
        is_android_ndk_exist=0
    fi

    if [ ${is_android_ndk_exist} -eq 0 ]; then

        if [ ! -f ${PROJECT_ROOT_PATH}/prebuilts/${ANDROID_NDK_FULLNAME} ]; then
            wget --no-config --quiet --show-progress -O ${PROJECT_ROOT_PATH}/prebuilts/${ANDROID_NDK_FULLNAME} https://dl.google.com/android/repository/${ANDROID_NDK_FULLNAME}
        fi

        cd ${PROJECT_ROOT_PATH}/prebuilts/
        unzip ${ANDROID_NDK_FULLNAME}

        if [ $? -ne 0 ]; then
            printf "failed to download Android NDK to %s \n" "${ANDROID_NDK}"
            exit 1
        fi
        /bin/cp -fv ${ANDROID_NDK}/shader-tools/linux-x86_64/glslc ${ANDROID_NDK}/shader-tools/linux-x86_64/glsls
        cd ${PROJECT_ROOT_PATH}

        printf "Android NDK saved to ${ANDROID_NDK} \n\n"
    else
        printf "Android NDK already exist:   ${ANDROID_NDK} \n\n"
    fi
}


function build_arm64
{
    cmake -H. -B./out/ggmlvulkan-android -DCMAKE_BUILD_TYPE=Release -DGGML_OPENMP=OFF -DCMAKE_TOOLCHAIN_FILE=${ANDROID_NDK}/build/cmake/android.toolchain.cmake -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=latest -DCMAKE_C_FLAGS=-march=armv8.7-a -DGGML_VULKAN=ON -DLLAMA_CURL=OFF -DGGML_LLAMAFILE=OFF -DVulkan_GLSLC_EXECUTABLE=${ANDROID_NDK}/shader-tools/linux-x86_64/glsls -DVulkan_INCLUDE_DIR=${VULKAN_HEADERS_PATH}/include
    cd out/ggmlvulkan-android
    make -j${HOST_CPU_COUNTS}
    show_pwd

    cd -
}


function build_arm64_debug
{
    cmake -H. -B./out/ggmlvulkan-android -DCMAKE_BUILD_TYPE=Debug -DGGML_OPENMP=OFF -DCMAKE_TOOLCHAIN_FILE=${ANDROID_NDK}/build/cmake/android.toolchain.cmake -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=latest -DCMAKE_C_FLAGS=-march=armv8.7-a -DGGML_VULKAN=ON -DGGML_VULKAN_DEBUG=ON -DLLAMA_CURL=OFF -DGGML_LLAMAFILE=OFF -DVulkan_GLSLC_EXECUTABLE=${ANDROID_NDK}/shader-tools/linux-x86_64/glsls -DVulkan_INCLUDE_DIR=${VULKAN_HEADERS_PATH}/include
    cd out/ggmlvulkan-android
    make -j${HOST_CPU_COUNTS}
    show_pwd

    cd -
}


function remove_temp_dir()
{
    if [ -d out/ggmlvulkan-android ]; then
        echo "remove out/ggmlvulkan-android directory in `pwd`"
        rm -rf out/ggmlvulkan-android
    fi
}


function build_ggml_vulkan()
{
    show_pwd
    check_and_download_ndk
    check_and_download_vulkan_sdk
    dump_vars
    remove_temp_dir
    build_arm64
}


function build_ggml_vulkan_debug()
{
    show_pwd
    check_and_download_ndk
    check_and_download_vulkan_sdk
    dump_vars
    remove_temp_dir
    build_arm64_debug
}


function check_and_download_model()
{
    set +e

    model_name=$1
    model_url=$2

    adb shell ls /sdcard/${model_name}
    if [ $? -eq 0 ]; then
        printf "the prebuild LLM model ${model_name} already exist on Android phone\n"
    else
        printf "the prebuild LLM model ${model_name} not exist on Android phone\n"
        wget --no-config --quiet --show-progress -O ${PROJECT_ROOT_PATH}/models/${model_name} ${model_url}
        adb push ${PROJECT_ROOT_PATH}/models/${model_name} /sdcard/
    fi

    set -e
}


function check_prebuilt_models()
{
    set +e

    check_and_download_model qwen1_5-1_8b-chat-q4_0.gguf https://huggingface.co/Qwen/Qwen1.5-1.8B-Chat-GGUF/resolve/main/qwen1_5-1_8b-chat-q4_0.gguf
    check_and_download_model gemma-3n-E2B-it-Q8_0.gguf https://huggingface.co/ggml-org/gemma-3n-E2B-it-GGUF/resolve/main/gemma-3n-E2B-it-Q8_0.gguf

    set -e
}


function prepare_run_on_phone()
{
    if [ $# != 1 ]; then
        print "invalid param"
        return
    fi
    program=$1

    check_prebuilt_models

    if [ -f ./out/ggmlvulkan-android/bin/libggml-cpu.so ]; then
        adb push ./out/ggmlvulkan-android/bin/*.so ${REMOTE_PATH}/
    fi
    adb push ./out/ggmlvulkan-android/bin/${program} ${REMOTE_PATH}/
    adb push ./out/ggmlvulkan-android/bin/vulkan-shaders-gen ${REMOTE_PATH}/

    adb shell ls -l ${REMOTE_PATH}/libggml-*.so

    adb shell chmod +x ${REMOTE_PATH}/${program}
}


function run_llamacli()
{
    prepare_run_on_phone llama-cli

    echo "${REMOTE_PATH}/llama-cli ${running_params} -no-cnv -m ${TEST_MODEL_NAME} -p \"${PROMPT_STRING}\""
    adb shell "cd ${REMOTE_PATH} \
               && export LD_LIBRARY_PATH=${REMOTE_PATH} \
               && ${REMOTE_PATH}/llama-cli ${running_params} -no-cnv -m ${TEST_MODEL_NAME} -p \"${PROMPT_STRING}\""

}


function run_llamabench()
{
    prepare_run_on_phone llama-bench

    echo "adb shell \"cd ${REMOTE_PATH} \
               && export LD_LIBRARY_PATH=${REMOTE_PATH} \
               && ${REMOTE_PATH}/llama-bench ${running_params} -m ${GGUF_MODEL_NAME}\""
    echo "${REMOTE_PATH}/llama-bench ${running_params} -m ${GGUF_MODEL_NAME}"

    adb shell "cd ${REMOTE_PATH} \
               && export LD_LIBRARY_PATH=${REMOTE_PATH} \
               && ${REMOTE_PATH}/llama-bench ${running_params} -m ${GGUF_MODEL_NAME}"

}


function run_threadsafety()
{
    prepare_run_on_phone test-thread-safety

    echo "${REMOTE_PATH}/test-thread-safety -np 2 -m ${GGUF_MODEL_NAME} -p \"hello,world\" -n 256 -ngl 99 "
    adb shell "cd ${REMOTE_PATH} \
               && export LD_LIBRARY_PATH=${REMOTE_PATH} \
               && ${REMOTE_PATH}/test-thread-safety -np 1 -m ${GGUF_MODEL_NAME} -p \"hello,world\" -n 256 -ngl 99 "

}


function run_test-ops()
{
    prepare_run_on_phone test-backend-ops

    adb shell "cd ${REMOTE_PATH} \
               && export LD_LIBRARY_PATH=${REMOTE_PATH} \
               && ${REMOTE_PATH}/test-backend-ops test"

}


function run_test-op()
{
    prepare_run_on_phone test-backend-ops

    echo "adb shell cd ${REMOTE_PATH} \
               && export LD_LIBRARY_PATH=${REMOTE_PATH} \
               && ${REMOTE_PATH}/test-backend-ops test -o ${opname}"

    echo "\n"
    adb shell "cd ${REMOTE_PATH} \
               && export LD_LIBRARY_PATH=${REMOTE_PATH} \
               && ${REMOTE_PATH}/test-backend-ops test -o ${opname}"

}


function run_perf-op()
{
    prepare_run_on_phone test-backend-ops

    echo "adb shell cd ${REMOTE_PATH} \
               && export LD_LIBRARY_PATH=${REMOTE_PATH} \
               && ${REMOTE_PATH}/test-backend-ops perf -o ${opname}"

    echo "\n"
    adb shell "cd ${REMOTE_PATH} \
               && export LD_LIBRARY_PATH=${REMOTE_PATH} \
               && ${REMOTE_PATH}/test-backend-ops perf -o ${opname}"

}


function show_usage()
{
    echo -e "\n\n\n"
    echo "Usage:"
    echo "  $0 help"
    echo "  $0 build"
    echo "  $0 build_debug"
    echo "  $0 run_testops"
    echo "  $0 run_testop     ADD/MUL_MAT                                                    (verify accuracy    of ADD/MUL_MAT)"
    echo "  $0 run_perfop     ADD/MUL_MAT                                                    (verify performance of ADD/MUL_MAT)"
    echo "  $0 run_llamacli"
    echo "  $0 run_llamabench"
    echo "  $0 run_threadsafety"
    echo -e "\n\n\n"
}


######## part-4: entry point  ########

show_pwd

check_commands_in_host
check_android_phone
check_and_download_ndk
check_and_download_vulkan_sdk
check_prebuilt_models

if [ $# == 0 ]; then
    show_usage
    exit 1
elif [ $# == 1 ]; then
    if [ "$1" == "-h" ]; then
        show_usage
        exit 1
    elif [ "$1" == "help" ]; then
        show_usage
        exit 1
    elif [ "$1" == "build" ]; then
        build_ggml_vulkan
        exit 0
    elif [ "$1" == "build_debug" ]; then
        build_ggml_vulkan_debug
        exit 0
    elif [ "$1" == "run_testops" ]; then
        run_test-ops
        exit 0
    elif [ "$1" == "run_llamacli" ]; then
        run_llamacli
        exit 0
    elif [ "$1" == "run_llamabench" ]; then
        run_llamabench
        exit 0
    elif [ "$1" == "run_threadsafety" ]; then
        run_threadsafety
        exit 0
    else
        show_usage
        exit 1
    fi
elif [ $# == 2 ]; then
    if [ "$1" == "run_testop" ]; then
        opname=$2
        run_test-op
        exit 0
    elif [ "$1" == "run_perfop" ]; then
        opname=$2
        run_perf-op
        exit 1
    fi
else
    show_usage
    exit 1
fi

How to build ggml-hexagon backend for Android device on Linux

https://github.com/zhouwg/ggml-hexagon/blob/self-build/scripts/build-run-ggmlhexagon-android.sh (supports performance comparison between QNN-CPU, QNN-GPU, QNN-NPU, Hexagon-cDSP and the default ggml backend (ARM NEON)).

Benchmark data

test-backend-ops with GGML_OP_ADD via ggml-vulkan on 8Elite phone, works fine

Details

adb shell cd /data/local/tmp                && export LD_LIBRARY_PATH=/data/local/tmp                && /data/local/tmp/test-backend-ops test -o ADD
\n
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Adreno (TM) 830 (Qualcomm Technologies Inc. Adreno Vulkan Driver) | uma: 1 | fp16: 1 | warp size: 64 | shared memory: 32768 | int dot: 0 | matrix cores: none
Testing 2 devices

Backend 1/2: Vulkan0
  Device description: Adreno (TM) 830
  Device memory: 15106 MB (15106 MB free)

  ADD(type=f16,ne=[1,1,8,1],nr=[1,1,1,1]): OK
  ADD(type=f16,ne=[1,1,1,1],nr=[32,1,1,1]): OK
  ADD(type=f16,ne=[1,1,320,320],nr=[1,1,1,1]): OK
  ADD(type=f16,ne=[10,5,1,1],nr=[1,1,1,1]): OK
  ADD(type=f16,ne=[10,5,4,1],nr=[1,1,1,1]): OK
  ADD(type=f16,ne=[10,5,4,3],nr=[1,1,1,1]): OK
  ADD(type=f16,ne=[10,5,4,3],nr=[2,1,1,1]): OK
  ADD(type=f16,ne=[10,5,4,3],nr=[1,2,1,1]): OK
  ADD(type=f16,ne=[10,5,4,3],nr=[1,1,2,1]): OK
  ADD(type=f16,ne=[10,5,4,3],nr=[1,1,1,2]): OK
  ADD(type=f16,ne=[10,5,4,3],nr=[1,1,2,2]): OK
  ADD(type=f16,ne=[10,5,4,3],nr=[1,2,2,2]): OK
  ADD(type=f16,ne=[10,5,4,3],nr=[2,2,2,2]): OK
  ADD(type=f16,ne=[1280,1,1,1],nr=[1,1,1,1]): OK
  ADD(type=f16,ne=[1280,1,1,1],nr=[1,16,16,1]): OK
  ADD(type=f16,ne=[1280,16,16,1],nr=[1,1,1,1]): OK
  ADD(type=f16,ne=[1280,1,1,1],nr=[1,256,1,1]): OK
  ADD(type=f16,ne=[1,1,1280,1],nr=[16,16,1,1]): OK
  ADD(type=f16,ne=[16,16,1280,1],nr=[1,1,1,1]): OK
  ADD(type=f16,ne=[1,1,1920,1],nr=[16,16,1,1]): OK
  ADD(type=f16,ne=[1,1,2560,1],nr=[16,16,1,1]): OK
  ADD(type=f16,ne=[1,1,1280,1],nr=[32,32,1,1]): OK
  ADD(type=f16,ne=[1,1,1920,1],nr=[32,32,1,1]): OK
  ADD(type=f16,ne=[1,1,640,1],nr=[32,32,1,1]): OK
  ADD(type=f16,ne=[5120,1,1,1],nr=[1,256,1,1]): OK
  ADD(type=f16,ne=[640,1,1,1],nr=[1,1,1,1]): OK
  ADD(type=f32,ne=[1,1,8,1],nr=[1,1,1,1]): OK
  ADD(type=f32,ne=[1,1,1,1],nr=[32,1,1,1]): OK
  ADD(type=f32,ne=[1,1,320,320],nr=[1,1,1,1]): OK
  ADD(type=f32,ne=[10,5,1,1],nr=[1,1,1,1]): OK
  ADD(type=f32,ne=[10,5,4,1],nr=[1,1,1,1]): OK
  ADD(type=f32,ne=[10,5,4,3],nr=[1,1,1,1]): OK
  ADD(type=f32,ne=[10,5,4,3],nr=[2,1,1,1]): OK
  ADD(type=f32,ne=[10,5,4,3],nr=[1,2,1,1]): OK
  ADD(type=f32,ne=[10,5,4,3],nr=[1,1,2,1]): OK
  ADD(type=f32,ne=[10,5,4,3],nr=[1,1,1,2]): OK
  ADD(type=f32,ne=[10,5,4,3],nr=[1,1,2,2]): OK
  ADD(type=f32,ne=[10,5,4,3],nr=[1,2,2,2]): OK
  ADD(type=f32,ne=[10,5,4,3],nr=[2,2,2,2]): OK
  ADD(type=f32,ne=[1280,1,1,1],nr=[1,1,1,1]): OK
  ADD(type=f32,ne=[1280,1,1,1],nr=[1,16,16,1]): OK
  ADD(type=f32,ne=[1280,16,16,1],nr=[1,1,1,1]): OK
  ADD(type=f32,ne=[1280,1,1,1],nr=[1,256,1,1]): OK
  ADD(type=f32,ne=[1,1,1280,1],nr=[16,16,1,1]): OK
  ADD(type=f32,ne=[16,16,1280,1],nr=[1,1,1,1]): OK
  ADD(type=f32,ne=[1,1,1920,1],nr=[16,16,1,1]): OK
  ADD(type=f32,ne=[1,1,2560,1],nr=[16,16,1,1]): OK
  ADD(type=f32,ne=[1,1,1280,1],nr=[32,32,1,1]): OK
  ADD(type=f32,ne=[1,1,1920,1],nr=[32,32,1,1]): OK
  ADD(type=f32,ne=[1,1,640,1],nr=[32,32,1,1]): OK
  ADD(type=f32,ne=[5120,1,1,1],nr=[1,256,1,1]): OK
  ADD(type=f32,ne=[640,1,1,1],nr=[1,1,1,1]): OK
  6485/6485 tests passed
  Backend Vulkan0: OK
Backend 2/2: CPU
  Skipping CPU backend
2/2 backends passed
OK

test-backend-ops with GGML_OP_MUL_MAT via ggml-vulkan on 8Elite phone, works but many errors

Details

zhouwg:$ ./scripts/build-run-ggmlvulkan-android.sh run_testop MUL_MAT
current working path:/home/zhouwg/kantvai/llama.cpp

/usr/bin/wget

/usr/bin/git

/bin/ls
Android NDK already exist:   /home/zhouwg/kantvai/llama.cpp/prebuilts/android-ndk-r28 

Vulkan SDK already exist:    /home/zhouwg/kantvai/llama.cpp/prebuilts/Vulkan_SDK 

/sdcard/qwen1_5-1_8b-chat-q4_0.gguf
the prebuild LLM model qwen1_5-1_8b-chat-q4_0.gguf already exist on Android phone
/sdcard/gemma-3n-E2B-it-Q8_0.gguf
the prebuild LLM model gemma-3n-E2B-it-Q8_0.gguf already exist on Android phone
/sdcard/qwen1_5-1_8b-chat-q4_0.gguf
the prebuild LLM model qwen1_5-1_8b-chat-q4_0.gguf already exist on Android phone
/sdcard/gemma-3n-E2B-it-Q8_0.gguf
the prebuild LLM model gemma-3n-E2B-it-Q8_0.gguf already exist on Android phone
./out/ggmlvulkan-android/bin/libggml-base.so: 1 file pushed. 18.2 MB/s (6962256 bytes in 0.365s)
./out/ggmlvulkan-android/bin/libggml-cpu.so: 1 file pushed. 17.7 MB/s (2881360 bytes in 0.155s)
./out/ggmlvulkan-android/bin/libggml.so: 1 file pushed. 17.6 MB/s (1764240 bytes in 0.096s)
./out/ggmlvulkan-android/bin/libggml-vulkan.so: 1 file pushed. 19.1 MB/s (25159984 bytes in 1.257s)
./out/ggmlvulkan-android/bin/libllama.so: 1 file pushed. 18.9 MB/s (20892992 bytes in 1.053s)
./out/ggmlvulkan-android/bin/libmtmd.so: 1 file pushed. 17.9 MB/s (4863024 bytes in 0.259s)
6 files pushed. 18.7 MB/s (62523856 bytes in 3.189s)
./out/ggmlvulkan-android/bin/test-backend-ops: 1 file pushed. 17.2 MB/s (5415160 bytes in 0.299s)
./out/ggmlvulkan-android/bin/vulkan-shaders-gen: 1 file pushed. 15.0 MB/s (197176 bytes in 0.013s)
-rwxrwxrwx 1 shell shell  6962256 2025-07-06 22:54 /data/local/tmp/libggml-base.so
-rwxrwxrwx 1 shell shell  2881360 2025-07-06 22:54 /data/local/tmp/libggml-cpu.so
-rwxrwxrwx 1 shell shell  5848736 2025-07-06 10:56 /data/local/tmp/libggml-hexagon.so
-rwxrwxrwx 1 shell shell  1476880 2025-07-06 12:42 /data/local/tmp/libggml-opencl.so
-rwxrwxrwx 1 shell shell 25159984 2025-07-06 22:55 /data/local/tmp/libggml-vulkan.so
adb shell cd /data/local/tmp                && export LD_LIBRARY_PATH=/data/local/tmp                && /data/local/tmp/test-backend-ops test -o MUL_MAT
\n
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Adreno (TM) 830 (Qualcomm Technologies Inc. Adreno Vulkan Driver) | uma: 1 | fp16: 1 | warp size: 64 | shared memory: 32768 | int dot: 0 | matrix cores: none
Testing 2 devices

Backend 1/2: Vulkan0
  Device description: Adreno (TM) 830
  Device memory: 15106 MB (15106 MB free)

  MUL_MAT(type_a=f32,type_b=f32,m=16,n=1,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=f32,type_b=f32,m=16,n=2,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=f32,type_b=f32,m=16,n=3,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=f32,type_b=f32,m=16,n=4,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=f32,type_b=f32,m=16,n=5,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=f32,type_b=f32,m=16,n=6,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=f32,type_b=f32,m=16,n=7,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=f32,type_b=f32,m=16,n=8,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=f32,type_b=f32,m=16,n=9,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=1,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=2,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=3,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=4,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=5,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=6,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=7,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=8,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=9,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=bf16,type_b=f32,m=16,n=1,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=bf16,type_b=f32,m=16,n=2,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=bf16,type_b=f32,m=16,n=3,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=bf16,type_b=f32,m=16,n=4,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=bf16,type_b=f32,m=16,n=5,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=bf16,type_b=f32,m=16,n=6,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=bf16,type_b=f32,m=16,n=7,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=bf16,type_b=f32,m=16,n=8,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=bf16,type_b=f32,m=16,n=9,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=1,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=2,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=3,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=4,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=5,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=6,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=7,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=8,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
[MUL_MAT] NMSE = 0.962842217 > 0.000500000   MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=9,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): FAIL
  MUL_MAT(type_a=q4_1,type_b=f32,m=16,n=1,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=q4_1,type_b=f32,m=16,n=2,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK```

</details>
  MUL_MAT(type_a=q4_1,type_b=f32,m=16,n=3,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=q4_1,type_b=f32,m=16,n=4,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=q4_1,type_b=f32,m=16,n=5,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=q4_1,type_b=f32,m=16,n=6,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=q4_1,type_b=f32,m=16,n=7,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=q4_1,type_b=f32,m=16,n=8,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
[MUL_MAT] NMSE = 1.111166107 > 0.000500000   MUL_MAT(type_a=q4_1,type_b=f32,m=16,n=9,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): FAIL
  MUL_MAT(type_a=q5_0,type_b=f32,m=16,n=1,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=q5_0,type_b=f32,m=16,n=2,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=q5_0,type_b=f32,m=16,n=3,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=q5_0,type_b=f32,m=16,n=4,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=q5_0,type_b=f32,m=16,n=5,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=q5_0,type_b=f32,m=16,n=6,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=q5_0,type_b=f32,m=16,n=7,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=q5_0,type_b=f32,m=16,n=8,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=q5_0,type_b=f32,m=16,n=9,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=q5_1,type_b=f32,m=16,n=1,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=q5_1,type_b=f32,m=16,n=2,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=q5_1,type_b=f32,m=16,n=3,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=q5_1,type_b=f32,m=16,n=4,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=q5_1,type_b=f32,m=16,n=5,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=q5_1,type_b=f32,m=16,n=6,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=q5_1,type_b=f32,m=16,n=7,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=q5_1,type_b=f32,m=16,n=8,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
  MUL_MAT(type_a=q5_1,type_b=f32,m=16,n=9,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
[MUL_MAT] NMSE = 2.307256596 > 0.000500000   MUL_MAT(type_a=q8_0,type_b=f32,m=16,n=1,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): FAIL
[MUL_MAT] NMSE = 2.016235316 > 0.000500000   MUL_MAT(type_a=q8_0,type_b=f32,m=16,n=2,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): FAIL
[MUL_MAT] NMSE = 2.225127068 > 0.000500000   MUL_MAT(type_a=q8_0,type_b=f32,m=16,n=3,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): FAIL
[MUL_MAT] NMSE = 1.158545781 > 0.000500000   MUL_MAT(type_a=q8_0,type_b=f32,m=16,n=4,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): FAIL
[MUL_MAT] NMSE = 1.599564986 > 0.000500000   MUL_MAT(type_a=q8_0,type_b=f32,m=16,n=5,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): FAIL
[MUL_MAT] NMSE = 1.682749565 > 0.000500000   MUL_MAT(type_a=q8_0,type_b=f32,m=16,n=6,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): FAIL
[MUL_MAT] NMSE = 1.883349541 > 0.000500000   MUL_MAT(type_a=q8_0,type_b=f32,m=16,n=7,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): FAIL
[MUL_MAT] NMSE = 2.006733186 > 0.000500000   MUL_MAT(type_a=q8_0,type_b=f32,m=16,n=8,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): FAIL
[MUL_MAT] NMSE = 1.799521972 > 0.000500000   MUL_MAT(type_a=q8_0,type_b=f32,m=16,n=9,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): FAIL
[MUL_MAT] NMSE = 0.672237012 > 0.000500000   MUL_MAT(type_a=q2_K,type_b=f32,m=16,n=1,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): FAIL
[MUL_MAT] NMSE = 0.477019036 > 0.000500000   MUL_MAT(type_a=q2_K,type_b=f32,m=16,n=2,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): FAIL
[MUL_MAT] NMSE = 0.689437405 > 0.000500000   MUL_MAT(type_a=q2_K,type_b=f32,m=16,n=3,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): FAIL
[MUL_MAT] NMSE = 0.912660646 > 0.000500000   MUL_MAT(type_a=q2_K,type_b=f32,m=16,n=4,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): FAIL
[MUL_MAT] NMSE = 0.699199288 > 0.000500000   MUL_MAT(type_a=q2_K,type_b=f32,m=16,n=5,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): FAIL
[MUL_MAT] NMSE = 0.892200715 > 0.000500000   MUL_MAT(type_a=q2_K,type_b=f32,m=16,n=6,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): FAIL
[MUL_MAT] NMSE = 1.094863304 > 0.000500000   MUL_MAT(type_a=q2_K,type_b=f32,m=16,n=7,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): FAIL
[MUL_MAT] NMSE = 1.144789085 > 0.000500000   MUL_MAT(type_a=q2_K,type_b=f32,m=16,n=8,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): FAIL
  MUL_MAT(type_a=q2_K,type_b=f32,m=16,n=9,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
[MUL_MAT] NMSE = 0.992872044 > 0.000500000   MUL_MAT(type_a=q3_K,type_b=f32,m=16,n=1,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): FAIL
[MUL_MAT] NMSE = 0.425093494 > 0.000500000   MUL_MAT(type_a=q3_K,type_b=f32,m=16,n=2,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): FAIL
[MUL_MAT] NMSE = 1.100980204 > 0.000500000   MUL_MAT(type_a=q3_K,type_b=f32,m=16,n=3,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): FAIL
[MUL_MAT] NMSE = 1.265905747 > 0.000500000   MUL_MAT(type_a=q3_K,type_b=f32,m=16,n=4,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): FAIL
[MUL_MAT] NMSE = 1.061933538 > 0.000500000   MUL_MAT(type_a=q3_K,type_b=f32,m=16,n=5,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): FAIL
[MUL_MAT] NMSE = 0.940871716 > 0.000500000   MUL_MAT(type_a=q3_K,type_b=f32,m=16,n=6,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): FAIL
[MUL_MAT] NMSE = 0.924015348 > 0.000500000   MUL_MAT(type_a=q3_K,type_b=f32,m=16,n=7,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): FAIL
[MUL_MAT] NMSE = 0.821410041 > 0.000500000   MUL_MAT(type_a=q3_K,type_b=f32,m=16,n=8,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): FAIL
  MUL_MAT(type_a=q3_K,type_b=f32,m=16,n=9,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0): OK
ggml_vulkan: Compute pipeline creation failed for mul_mat_vec_q4_k_f32_f32_1
ggml_vulkan: vk::Device::createComputePipeline: ErrorUnknown
Segmentation fault

llama-cli via ggml-vulkan on 8Elite phone, can't works

Details

zhouwg:$ ./scripts/build-run-ggmlvulkan-android.sh run_llamacli
current working path:/home/zhouwg/kantvai/llama.cpp

/usr/bin/wget

/usr/bin/git

/bin/ls
Android NDK already exist:   /home/zhouwg/kantvai/llama.cpp/prebuilts/android-ndk-r28 

Vulkan SDK already exist:    /home/zhouwg/kantvai/llama.cpp/prebuilts/Vulkan_SDK 

/sdcard/qwen1_5-1_8b-chat-q4_0.gguf
the prebuild LLM model qwen1_5-1_8b-chat-q4_0.gguf already exist on Android phone
/sdcard/gemma-3n-E2B-it-Q8_0.gguf
the prebuild LLM model gemma-3n-E2B-it-Q8_0.gguf already exist on Android phone
/sdcard/qwen1_5-1_8b-chat-q4_0.gguf
the prebuild LLM model qwen1_5-1_8b-chat-q4_0.gguf already exist on Android phone
/sdcard/gemma-3n-E2B-it-Q8_0.gguf
the prebuild LLM model gemma-3n-E2B-it-Q8_0.gguf already exist on Android phone
./out/ggmlvulkan-android/bin/libggml-base.so: 1 file pushed. 19.7 MB/s (6962256 bytes in 0.338s)
./out/ggmlvulkan-android/bin/libggml-cpu.so: 1 file pushed. 17.0 MB/s (2881360 bytes in 0.162s)
./out/ggmlvulkan-android/bin/libggml.so: 1 file pushed. 16.5 MB/s (1764240 bytes in 0.102s)
./out/ggmlvulkan-android/bin/libggml-vulkan.so: 1 file pushed. 17.5 MB/s (25159984 bytes in 1.369s)
./out/ggmlvulkan-android/bin/libllama.so: 1 file pushed. 18.7 MB/s (20892992 bytes in 1.063s)
./out/ggmlvulkan-android/bin/libmtmd.so: 1 file pushed. 17.0 MB/s (4863024 bytes in 0.273s)
6 files pushed. 18.0 MB/s (62523856 bytes in 3.311s)
./out/ggmlvulkan-android/bin/llama-cli: 1 file pushed. 17.2 MB/s (27637424 bytes in 1.529s)
./out/ggmlvulkan-android/bin/vulkan-shaders-gen: 1 file pushed. 13.2 MB/s (197176 bytes in 0.014s)
-rwxrwxrwx 1 shell shell  6962256 2025-07-06 22:54 /data/local/tmp/libggml-base.so
-rwxrwxrwx 1 shell shell  2881360 2025-07-06 22:54 /data/local/tmp/libggml-cpu.so
-rwxrwxrwx 1 shell shell  5848736 2025-07-06 10:56 /data/local/tmp/libggml-hexagon.so
-rwxrwxrwx 1 shell shell  1476880 2025-07-06 12:42 /data/local/tmp/libggml-opencl.so
-rwxrwxrwx 1 shell shell 25159984 2025-07-06 22:55 /data/local/tmp/libggml-vulkan.so
/data/local/tmp/llama-cli  -ngl 99 -t 4 -n 256 --no-warmup  -no-cnv -m /sdcard/gemma-3n-E2B-it-Q8_0.gguf -p ""
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Adreno (TM) 830 (Qualcomm Technologies Inc. Adreno Vulkan Driver) | uma: 1 | fp16: 1 | warp size: 64 | shared memory: 32768 | int dot: 0 | matrix cores: none
build: 6032 (cc826618e) with Android (12896553, +pgo, +bolt, +lto, +mlgo, based on r530567c) clang version 19.0.0 (https://android.googlesource.com/toolchain/llvm-project 97a699bf4812a18fb657c2779f5296a4ab2694d2) for x86_64-unknown-linux-gnu
main: llama backend init
main: load the model and apply lora adapter, if any
llama_model_load_from_file_impl: using device Vulkan0 (Adreno (TM) 830) - 15106 MiB free
llama_model_loader: loaded meta data with 42 key-value pairs and 727 tensors from /sdcard/gemma-3n-E2B-it-Q8_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = gemma3n
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                         general.size_label str              = 4.5B
llama_model_loader: - kv   3:                            general.license str              = gemma
llama_model_loader: - kv   4:                   general.base_model.count u32              = 1
llama_model_loader: - kv   5:                  general.base_model.0.name str              = Gemma 3n E4b It
llama_model_loader: - kv   6:          general.base_model.0.organization str              = Google
llama_model_loader: - kv   7:              general.base_model.0.repo_url str              = https://huggingface.co/google/gemma-3...
llama_model_loader: - kv   8:                               general.tags arr[str,5]       = ["automatic-speech-recognition", "aut...
llama_model_loader: - kv   9:                     gemma3n.context_length u32              = 32768
llama_model_loader: - kv  10:                   gemma3n.embedding_length u32              = 2048
llama_model_loader: - kv  11:                        gemma3n.block_count u32              = 30
llama_model_loader: - kv  12:                gemma3n.feed_forward_length u32              = 8192
llama_model_loader: - kv  13:               gemma3n.attention.head_count u32              = 8
llama_model_loader: - kv  14:   gemma3n.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  15:               gemma3n.attention.key_length u32              = 256
llama_model_loader: - kv  16:             gemma3n.attention.value_length u32              = 256
llama_model_loader: - kv  17:                     gemma3n.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  18:           gemma3n.attention.sliding_window u32              = 512
llama_model_loader: - kv  19:            gemma3n.attention.head_count_kv u32              = 2
llama_model_loader: - kv  20:                   gemma3n.altup.active_idx u32              = 0
llama_model_loader: - kv  21:                   gemma3n.altup.num_inputs u32              = 4
llama_model_loader: - kv  22:   gemma3n.embedding_length_per_layer_input u32              = 256
llama_model_loader: - kv  23:         gemma3n.attention.shared_kv_layers f32              = 10.000000
llama_model_loader: - kv  24:          gemma3n.activation_sparsity_scale arr[f32,30]      = [1.644853, 1.644853, 1.644853, 1.6448...
llama_model_loader: - kv  25:   gemma3n.attention.sliding_window_pattern arr[bool,30]     = [true, true, true, true, false, true,...
llama_model_loader: - kv  26:                    tokenizer.chat_template str              = {{ bos_token }}\n{%- if messages[0]['r...
llama_model_loader: - kv  27:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  28:                         tokenizer.ggml.pre str              = default
llama_model_loader: - kv  29:                      tokenizer.ggml.tokens arr[str,262144]  = ["<pad>", "<eos>", "<bos>", "<unk>", ...
llama_model_loader: - kv  30:                      tokenizer.ggml.scores arr[f32,262144]  = [-1000.000000, -1000.000000, -1000.00...
llama_model_loader: - kv  31:                  tokenizer.ggml.token_type arr[i32,262144]  = [3, 3, 3, 3, 3, 4, 3, 3, 3, 3, 3, 3, ...
llama_model_loader: - kv  32:                tokenizer.ggml.bos_token_id u32              = 2
llama_model_loader: - kv  33:                tokenizer.ggml.eos_token_id u32              = 1
llama_model_loader: - kv  34:            tokenizer.ggml.unknown_token_id u32              = 3
llama_model_loader: - kv  35:            tokenizer.ggml.padding_token_id u32              = 0
llama_model_loader: - kv  36:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  37:               tokenizer.ggml.add_sep_token bool             = false
llama_model_loader: - kv  38:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  39:            tokenizer.ggml.add_space_prefix bool             = false
llama_model_loader: - kv  40:               general.quantization_version u32              = 2
llama_model_loader: - kv  41:                          general.file_type u32              = 7
llama_model_loader: - type  f32:  362 tensors
llama_model_loader: - type  f16:   93 tensors
llama_model_loader: - type q8_0:  272 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q8_0
print_info: file size   = 4.45 GiB (8.59 BPW) 
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: special tokens cache size = 6414
load: token to piece cache size = 1.9446 MB
print_info: arch             = gemma3n
print_info: vocab_only       = 0
print_info: n_ctx_train      = 32768
print_info: n_embd           = 2048
print_info: n_layer          = 30
print_info: n_head           = 8
print_info: n_head_kv        = 2
print_info: n_rot            = 256
print_info: n_swa            = 512
print_info: is_swa_any       = 1
print_info: n_embd_head_k    = 256
print_info: n_embd_head_v    = 256
print_info: n_gqa            = 4
print_info: n_embd_k_gqa     = 512
print_info: n_embd_v_gqa     = 512
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-06
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 1.0e+00
print_info: n_ff             = 8192
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 2
print_info: rope scaling     = linear
print_info: freq_base_train  = 1000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 32768
print_info: rope_finetuned   = unknown
print_info: model type       = E2B
print_info: model params     = 4.46 B
print_info: general.name     = n/a
print_info: vocab type       = SPM
print_info: n_vocab          = 262144
print_info: n_merges         = 0
print_info: BOS token        = 2 '<bos>'
print_info: EOS token        = 1 '<eos>'
print_info: EOT token        = 106 '<end_of_turn>'
print_info: UNK token        = 3 '<unk>'
print_info: PAD token        = 0 '<pad>'
print_info: LF token         = 248 '<0x0A>'
print_info: EOG token        = 1 '<eos>'
print_info: EOG token        = 106 '<end_of_turn>'
print_info: max token length = 48
load_tensors: loading model tensors, this can take a while... (mmap = true)
ggml_vulkan: Device memory allocation of size 2139095040 failed.
ggml_vulkan: Requested buffer size exceeds device memory allocation limit: ErrorOutOfDeviceMemory
alloc_tensor_range: failed to allocate Vulkan0 buffer of size 2139095040
llama_model_load: error loading model: unable to allocate Vulkan0 buffer
llama_model_load_from_file_impl: failed to load model
common_init_from_params: failed to load model '/sdcard/gemma-3n-E2B-it-Q8_0.gguf'
main: error: unable to load model

Details

zhouwg:$ ./scripts/build-run-ggmlvulkan-android.sh run_llamacli
current working path:/home/zhouwg/kantvai/llama.cpp

/usr/bin/wget

/usr/bin/git

/bin/ls
Android NDK already exist:   /home/zhouwg/kantvai/llama.cpp/prebuilts/android-ndk-r28 

Vulkan SDK already exist:    /home/zhouwg/kantvai/llama.cpp/prebuilts/Vulkan_SDK 

/sdcard/qwen1_5-1_8b-chat-q4_0.gguf
the prebuild LLM model qwen1_5-1_8b-chat-q4_0.gguf already exist on Android phone
/sdcard/gemma-3n-E2B-it-Q8_0.gguf
the prebuild LLM model gemma-3n-E2B-it-Q8_0.gguf already exist on Android phone
/sdcard/qwen1_5-1_8b-chat-q4_0.gguf
the prebuild LLM model qwen1_5-1_8b-chat-q4_0.gguf already exist on Android phone
/sdcard/gemma-3n-E2B-it-Q8_0.gguf
the prebuild LLM model gemma-3n-E2B-it-Q8_0.gguf already exist on Android phone
./out/ggmlvulkan-android/bin/libggml-base.so: 1 file pushed. 17.5 MB/s (6962256 bytes in 0.380s)
./out/ggmlvulkan-android/bin/libggml-cpu.so: 1 file pushed. 17.1 MB/s (2881360 bytes in 0.160s)
./out/ggmlvulkan-android/bin/libggml.so: 1 file pushed. 17.0 MB/s (1764240 bytes in 0.099s)
./out/ggmlvulkan-android/bin/libggml-vulkan.so: 1 file pushed. 17.7 MB/s (25159984 bytes in 1.356s)
./out/ggmlvulkan-android/bin/libllama.so: 1 file pushed. 18.8 MB/s (20892992 bytes in 1.060s)
./out/ggmlvulkan-android/bin/libmtmd.so: 1 file pushed. 16.7 MB/s (4863024 bytes in 0.277s)
6 files pushed. 17.9 MB/s (62523856 bytes in 3.336s)
./out/ggmlvulkan-android/bin/llama-cli: 1 file pushed. 18.0 MB/s (27637424 bytes in 1.466s)
./out/ggmlvulkan-android/bin/vulkan-shaders-gen: 1 file pushed. 12.9 MB/s (197176 bytes in 0.015s)
-rwxrwxrwx 1 shell shell  6962256 2025-07-07 22:26 /data/local/tmp/libggml-base.so
-rwxrwxrwx 1 shell shell  2881360 2025-07-07 22:26 /data/local/tmp/libggml-cpu.so
-rwxrwxrwx 1 shell shell  5847712 2025-07-07 22:35 /data/local/tmp/libggml-hexagon.so
-rwxrwxrwx 1 shell shell  1476880 2025-07-07 22:19 /data/local/tmp/libggml-opencl.so
-rwxrwxrwx 1 shell shell 25159984 2025-07-07 22:27 /data/local/tmp/libggml-vulkan.so
/data/local/tmp/llama-cli  -ngl 99 -t 4 -n 256 --no-warmup  -no-cnv -m /sdcard/qwen1_5-1_8b-chat-q4_0.gguf -p "introduce the movie Once Upon a Time in America briefly.\n"
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Adreno (TM) 830 (Qualcomm Technologies Inc. Adreno Vulkan Driver) | uma: 1 | fp16: 1 | warp size: 64 | shared memory: 32768 | int dot: 0 | matrix cores: none
build: 6038 (12ac0f350) with Android (12896553, +pgo, +bolt, +lto, +mlgo, based on r530567c) clang version 19.0.0 (https://android.googlesource.com/toolchain/llvm-project 97a699bf4812a18fb657c2779f5296a4ab2694d2) for x86_64-unknown-linux-gnu
main: llama backend init
main: load the model and apply lora adapter, if any
llama_model_load_from_file_impl: using device Vulkan0 (Adreno (TM) 830) - 15106 MiB free
llama_model_loader: loaded meta data with 21 key-value pairs and 291 tensors from /sdcard/qwen1_5-1_8b-chat-q4_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen2
llama_model_loader: - kv   1:                               general.name str              = Qwen1.5-1.8B-Chat-AWQ-fp16
llama_model_loader: - kv   2:                          qwen2.block_count u32              = 24
llama_model_loader: - kv   3:                       qwen2.context_length u32              = 32768
llama_model_loader: - kv   4:                     qwen2.embedding_length u32              = 2048
llama_model_loader: - kv   5:                  qwen2.feed_forward_length u32              = 5504
llama_model_loader: - kv   6:                 qwen2.attention.head_count u32              = 16
llama_model_loader: - kv   7:              qwen2.attention.head_count_kv u32              = 16
llama_model_loader: - kv   8:     qwen2.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv   9:                       qwen2.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  10:                qwen2.use_parallel_residual bool             = true
llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,151936]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  13:                  tokenizer.ggml.token_type arr[i32,151936]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  14:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv  15:                tokenizer.ggml.eos_token_id u32              = 151645
llama_model_loader: - kv  16:            tokenizer.ggml.padding_token_id u32              = 151643
llama_model_loader: - kv  17:                tokenizer.ggml.bos_token_id u32              = 151643
llama_model_loader: - kv  18:                    tokenizer.chat_template str              = {% for message in messages %}{{'<|im_...
llama_model_loader: - kv  19:               general.quantization_version u32              = 2
llama_model_loader: - kv  20:                          general.file_type u32              = 2
llama_model_loader: - type  f32:  121 tensors
llama_model_loader: - type q4_0:  169 tensors
llama_model_loader: - type q6_K:    1 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_0
print_info: file size   = 1.04 GiB (4.85 BPW) 
load: missing pre-tokenizer type, using: 'default'
load:                                             
load: ************************************        
load: GENERATION QUALITY WILL BE DEGRADED!        
load: CONSIDER REGENERATING THE MODEL             
load: ************************************        
load:                                             
load: special tokens cache size = 293
load: token to piece cache size = 0.9338 MB
print_info: arch             = qwen2
print_info: vocab_only       = 0
print_info: n_ctx_train      = 32768
print_info: n_embd           = 2048
print_info: n_layer          = 24
print_info: n_head           = 16
print_info: n_head_kv        = 16
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 1
print_info: n_embd_k_gqa     = 2048
print_info: n_embd_v_gqa     = 2048
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-06
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 5504
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: causal attn      = 1
print_info: pooling type     = -1
print_info: rope type        = 2
print_info: rope scaling     = linear
print_info: freq_base_train  = 1000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 32768
print_info: rope_finetuned   = unknown
print_info: model type       = 1B
print_info: model params     = 1.84 B
print_info: general.name     = Qwen1.5-1.8B-Chat-AWQ-fp16
print_info: vocab type       = BPE
print_info: n_vocab          = 151936
print_info: n_merges         = 151387
print_info: BOS token        = 151643 '<|endoftext|>'
print_info: EOS token        = 151645 '<|im_end|>'
print_info: EOT token        = 151645 '<|im_end|>'
print_info: PAD token        = 151643 '<|endoftext|>'
print_info: LF token         = 198 'Ċ'
print_info: EOG token        = 151643 '<|endoftext|>'
print_info: EOG token        = 151645 '<|im_end|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: offloading 24 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 25/25 layers to GPU
load_tensors:   CPU_Mapped model buffer size =   166.92 MiB
load_tensors:      Vulkan0 model buffer size =   895.75 MiB
................................................................
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 4096
llama_context: n_ctx_per_seq = 4096
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = 0
llama_context: freq_base     = 1000000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_per_seq (4096) < n_ctx_train (32768) -- the full capacity of the model will not be utilized
llama_context: Vulkan_Host  output buffer size =     0.58 MiB
llama_kv_cache_unified:    Vulkan0 KV buffer size =   768.00 MiB
llama_kv_cache_unified: size =  768.00 MiB (  4096 cells,  24 layers,  1 seqs), K (f16):  384.00 MiB, V (f16):  384.00 MiB
llama_kv_cache_unified: LLAMA_SET_ROWS=0, using old ggml_cpy() method for backwards compatibility
llama_context:    Vulkan0 compute buffer size =   300.75 MiB
llama_context: Vulkan_Host compute buffer size =    12.01 MiB
llama_context: graph nodes  = 942
llama_context: graph splits = 2
common_init_from_params: setting dry_penalty_last_n to ctx_size = 4096
main: llama threadpool init, n_threads = 4

system_info: n_threads = 4 (n_threads_batch = 4) / 8 | CPU : NEON = 1 | ARM_FMA = 1 | MATMUL_INT8 = 1 | DOTPROD = 1 | REPACK = 1 | 

sampler seed: 2452246687
sampler params: 
	repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
	dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = 4096
	top_k = 40, top_p = 0.950, min_p = 0.050, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, top_n_sigma = -1.000, temp = 0.800
	mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
sampler chain: logits -> logit-bias -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist 
generate: n_ctx = 4096, n_batch = 2048, n_predict = 256, n_keep = 0

introduce the movie Once Upon a Time in America briefly.
download需虫他对他对他对在他的工作在他的他对他对满在他的工作在他的工作在他的__在他的_________________________________________________虫_安心_______安心___虫__虫_安心__安心_ load_安心_虫 � hom虫有人_安心_安心_安心_虫_被人虫有人安心_安心_质量和__安心_有人被人_质量和/home 无辜被人_质量和_安心__质量和_安心_有人有人有人在外_质量和_安心_被人_质量和_安心_质量和_质量和_安心_善意满_在他的质量和_安心_质量和_安心_当他善意_质量和_安心_当他 uninolocation__镖__质量和_当他_质量和_当他_当他善意_质量和_安心_质量和_安心_质量和_善意_当他善意_质量和_安心_善意_质量和_善意_在他的_质量和_安心_在他的_和服务在他的_当他善意_在他的_当他善意_当他善意_在他的_在他的_当他善意_在他的_在他的_在他的

llama_perf_sampler_print:    sampling time =     198.34 ms /   269 runs   (    0.74 ms per token,  1356.25 tokens per second)
llama_perf_context_print:        load time =    3492.56 ms
llama_perf_context_print: prompt eval time =    2066.98 ms /    13 tokens (  159.00 ms per token,     6.29 tokens per second)
llama_perf_context_print:        eval time =    7861.17 ms /   255 runs   (   30.83 ms per token,    32.44 tokens per second)
llama_perf_context_print:       total time =   11681.79 ms /   268 tokens

llama-cli via ggml-opencl on 8Elite phone

Details

zhouwg:$ ./scripts/build-run-ggmlopencl-android.sh run_llamacli
current working path:/home/zhouwg/kantvai/llama.cpp

/usr/bin/wget

/usr/bin/git

/usr/bin/ninja

/bin/ls
Android NDK already exist:   /home/zhouwg/kantvai/llama.cpp/prebuilts/android-ndk-r28 

OpenCL SDK already exist:    /home/zhouwg/kantvai/llama.cpp/prebuilts/OpenCL_SDK 

/sdcard/qwen1_5-1_8b-chat-q4_0.gguf
the prebuild LLM model qwen1_5-1_8b-chat-q4_0.gguf already exist on Android phone
/sdcard/gemma-3n-E2B-it-Q8_0.gguf
the prebuild LLM model gemma-3n-E2B-it-Q8_0.gguf already exist on Android phone
/sdcard/qwen1_5-1_8b-chat-q4_0.gguf
the prebuild LLM model qwen1_5-1_8b-chat-q4_0.gguf already exist on Android phone
/sdcard/gemma-3n-E2B-it-Q8_0.gguf
the prebuild LLM model gemma-3n-E2B-it-Q8_0.gguf already exist on Android phone
./out/ggmlopencl-android/bin/libggml-base.so: 1 file pushed. 17.6 MB/s (6962256 bytes in 0.377s)
./out/ggmlopencl-android/bin/libggml-cpu.so: 1 file pushed. 17.6 MB/s (3407440 bytes in 0.184s)
./out/ggmlopencl-android/bin/libggml-opencl.so: 1 file pushed. 16.5 MB/s (1476880 bytes in 0.086s)
./out/ggmlopencl-android/bin/libggml.so: 1 file pushed. 16.6 MB/s (1764248 bytes in 0.101s)
./out/ggmlopencl-android/bin/libllama.so: 1 file pushed. 19.9 MB/s (24163448 bytes in 1.158s)
./out/ggmlopencl-android/bin/libmtmd.so: 1 file pushed. 18.6 MB/s (4863024 bytes in 0.249s)
6 files pushed. 18.8 MB/s (42637296 bytes in 2.159s)
./out/ggmlopencl-android/bin/llama-cli: 1 file pushed. 18.6 MB/s (27712544 bytes in 1.425s)
-rwxrwxrwx 1 shell shell  6962256 2025-07-07 22:19 /data/local/tmp/libggml-base.so
-rwxrwxrwx 1 shell shell  3407440 2025-07-07 22:19 /data/local/tmp/libggml-cpu.so
-rwxrwxrwx 1 shell shell  5848720 2025-07-07 22:02 /data/local/tmp/libggml-hexagon.so
-rwxrwxrwx 1 shell shell  1476880 2025-07-07 22:19 /data/local/tmp/libggml-opencl.so
-rwxrwxrwx 1 shell shell 25159984 2025-07-07 22:27 /data/local/tmp/libggml-vulkan.so
/data/local/tmp/llama-cli  -ngl 99 -t 4 -n 256 --no-warmup  -no-cnv -m /sdcard/qwen1_5-1_8b-chat-q4_0.gguf -p "introduce the movie Once Upon a Time in America briefly.\n"
ggml_opencl: selected platform: 'QUALCOMM Snapdragon(TM)'

ggml_opencl: device: 'QUALCOMM Adreno(TM) 830 (OpenCL 3.0 Adreno(TM) 830)'
ggml_opencl: OpenCL driver: OpenCL 3.0 QUALCOMM build: commit unknown Compiler E031.47.18.23
ggml_opencl: vector subgroup broadcast support: true
ggml_opencl: device FP16 support: true
ggml_opencl: mem base addr align: 128
ggml_opencl: max mem alloc size: 1024 MB
ggml_opencl: SVM coarse grain buffer support: true
ggml_opencl: SVM fine grain buffer support: true
ggml_opencl: SVM fine grain system support: false
ggml_opencl: SVM atomics support: true
ggml_opencl: flattening quantized weights representation as struct of arrays (GGML_OPENCL_SOA_Q)
ggml_opencl: using kernels optimized for Adreno (GGML_OPENCL_USE_ADRENO_KERNELS)
ggml_opencl: loading OpenCL kernels....................................................
ggml_opencl: default device: 'QUALCOMM Adreno(TM) 830 (OpenCL 3.0 Adreno(TM) 830)'
build: 6038 (12ac0f350) with Android (12896553, +pgo, +bolt, +lto, +mlgo, based on r530567c) clang version 19.0.0 (https://android.googlesource.com/toolchain/llvm-project 97a699bf4812a18fb657c2779f5296a4ab2694d2) for x86_64-unknown-linux-gnu
main: llama backend init
main: load the model and apply lora adapter, if any
llama_model_load_from_file_impl: using device GPUOpenCL (QUALCOMM Adreno(TM) 830) - 0 MiB free
llama_model_loader: loaded meta data with 21 key-value pairs and 291 tensors from /sdcard/qwen1_5-1_8b-chat-q4_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen2
llama_model_loader: - kv   1:                               general.name str              = Qwen1.5-1.8B-Chat-AWQ-fp16
llama_model_loader: - kv   2:                          qwen2.block_count u32              = 24
llama_model_loader: - kv   3:                       qwen2.context_length u32              = 32768
llama_model_loader: - kv   4:                     qwen2.embedding_length u32              = 2048
llama_model_loader: - kv   5:                  qwen2.feed_forward_length u32              = 5504
llama_model_loader: - kv   6:                 qwen2.attention.head_count u32              = 16
llama_model_loader: - kv   7:              qwen2.attention.head_count_kv u32              = 16
llama_model_loader: - kv   8:     qwen2.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv   9:                       qwen2.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  10:                qwen2.use_parallel_residual bool             = true
llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,151936]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  13:                  tokenizer.ggml.token_type arr[i32,151936]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  14:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv  15:                tokenizer.ggml.eos_token_id u32              = 151645
llama_model_loader: - kv  16:            tokenizer.ggml.padding_token_id u32              = 151643
llama_model_loader: - kv  17:                tokenizer.ggml.bos_token_id u32              = 151643
llama_model_loader: - kv  18:                    tokenizer.chat_template str              = {% for message in messages %}{{'<|im_...
llama_model_loader: - kv  19:               general.quantization_version u32              = 2
llama_model_loader: - kv  20:                          general.file_type u32              = 2
llama_model_loader: - type  f32:  121 tensors
llama_model_loader: - type q4_0:  169 tensors
llama_model_loader: - type q6_K:    1 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_0
print_info: file size   = 1.04 GiB (4.85 BPW) 
load: missing pre-tokenizer type, using: 'default'
load:                                             
load: ************************************        
load: GENERATION QUALITY WILL BE DEGRADED!        
load: CONSIDER REGENERATING THE MODEL             
load: ************************************        
load:                                             
load: special tokens cache size = 293
load: token to piece cache size = 0.9338 MB
print_info: arch             = qwen2
print_info: vocab_only       = 0
print_info: n_ctx_train      = 32768
print_info: n_embd           = 2048
print_info: n_layer          = 24
print_info: n_head           = 16
print_info: n_head_kv        = 16
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 1
print_info: n_embd_k_gqa     = 2048
print_info: n_embd_v_gqa     = 2048
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-06
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 5504
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: causal attn      = 1
print_info: pooling type     = -1
print_info: rope type        = 2
print_info: rope scaling     = linear
print_info: freq_base_train  = 1000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 32768
print_info: rope_finetuned   = unknown
print_info: model type       = 1B
print_info: model params     = 1.84 B
print_info: general.name     = Qwen1.5-1.8B-Chat-AWQ-fp16
print_info: vocab type       = BPE
print_info: n_vocab          = 151936
print_info: n_merges         = 151387
print_info: BOS token        = 151643 '<|endoftext|>'
print_info: EOS token        = 151645 '<|im_end|>'
print_info: EOT token        = 151645 '<|im_end|>'
print_info: PAD token        = 151643 '<|endoftext|>'
print_info: LF token         = 198 'Ċ'
print_info: EOG token        = 151643 '<|endoftext|>'
print_info: EOG token        = 151645 '<|im_end|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: offloading 24 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 25/25 layers to GPU
load_tensors:   CPU_Mapped model buffer size =   166.92 MiB
load_tensors:       OpenCL model buffer size =   895.75 MiB
................................................................
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 4096
llama_context: n_ctx_per_seq = 4096
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = 0
llama_context: freq_base     = 1000000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_per_seq (4096) < n_ctx_train (32768) -- the full capacity of the model will not be utilized
llama_context:        CPU  output buffer size =     0.58 MiB
llama_kv_cache_unified:     OpenCL KV buffer size =   768.00 MiB
llama_kv_cache_unified: size =  768.00 MiB (  4096 cells,  24 layers,  1 seqs), K (f16):  384.00 MiB, V (f16):  384.00 MiB
llama_kv_cache_unified: LLAMA_SET_ROWS=0, using old ggml_cpy() method for backwards compatibility
llama_context:     OpenCL compute buffer size =   300.75 MiB
llama_context:        CPU compute buffer size =    12.01 MiB
llama_context: graph nodes  = 942
llama_context: graph splits = 2
common_init_from_params: setting dry_penalty_last_n to ctx_size = 4096
main: llama threadpool init, n_threads = 4

system_info: n_threads = 4 (n_threads_batch = 4) / 8 | CPU : NEON = 1 | ARM_FMA = 1 | MATMUL_INT8 = 1 | DOTPROD = 1 | REPACK = 1 | 

introduce the movie Once Upon a Time in America briefly.
sampler seed: 710723739
sampler params: 
	repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
	dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = 4096
	top_k = 40, top_p = 0.950, min_p = 0.050, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, top_n_sigma = -1.000, temp = 0.800
	mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
sampler chain: logits -> logit-bias -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist 
generate: n_ctx = 4096, n_batch = 2048, n_predict = 256, n_keep = 0

Once Upon a Time in America is a 1988 American crime film directed by Francis Ford Coppola. The film is a crime drama that tells the story of two New York City detectives, Sam Marlow (played by Robert De Niro) and David Frost (played by Al Pacino), who are assigned to investigate the murder of a wealthy businessman named John Reed (played by Al Pacino).
The film takes place in New York City in the late 1950s and early 1960s, during the height of the Cold War. The film follows Marlow and Frost as they delve into the complex web of secrets and corruption that runs through the city's elite society, including the high society, the underworld, and the political establishment.
Marlow and Frost are initially assigned to work on the murder of Reed, but they soon realize that he is not the only target of their investigation. Reed's business partner, a rival businessman named Arthur Goldstein (played by Robert De Niro), has also been murdered and his body is found in a nearby park.
As Marlow and Frost uncover the truth about Reed's death, they discover a complex web of relationships, hidden agendas, and power struggles that threatens to destroy the city's social fabric

llama_perf_sampler_print:    sampling time =     131.08 ms /   269 runs   (    0.49 ms per token,  2052.13 tokens per second)
llama_perf_context_print:        load time =    2395.93 ms
llama_perf_context_print: prompt eval time =     104.11 ms /    13 tokens (    8.01 ms per token,   124.87 tokens per second)
llama_perf_context_print:        eval time =    8510.46 ms /   255 runs   (   33.37 ms per token,    29.96 tokens per second)
llama_perf_context_print:       total time =   11124.95 ms /   268 tokens

llama-cli via the default ggml backend on 8Elite phone

Details

zhouwg:$ ./scripts/build-run-ggmlhexagon-android.sh run_llamacli 4
current working path:/home/zhouwg/kantvai/llama.cpp

/usr/bin/wget

/usr/bin/xzcat

/bin/ls
Android NDK already exist:         /home/zhouwg/kantvai/llama.cpp/prebuilts/android-ndk-r28 

Qualcomm QNN SDK already exist:    /home/zhouwg/kantvai/llama.cpp/prebuilts/QNN_SDK/qairt/2.36.0.250627/ 

Qualcomm Hexagon SDK already exist:/home/zhouwg/kantvai/llama.cpp/prebuilts/Hexagon_SDK/6.2.0.1 

/sdcard/t5-very-small-random-F32.gguf
the prebuild LLM model t5-very-small-random-F32.gguf already exist on Android phone
/sdcard/qwen1_5-1_8b-chat-q4_0.gguf
the prebuild LLM model qwen1_5-1_8b-chat-q4_0.gguf already exist on Android phone
/sdcard/gemma-3n-E2B-it-Q8_0.gguf
the prebuild LLM model gemma-3n-E2B-it-Q8_0.gguf already exist on Android phone
/data/local/tmp/libQnnCpu.so
/data/local/tmp/libQnnGpu.so
/data/local/tmp/libQnnHtp.so
QNN runtime libs already exist on Android phone
./scripts/ggml-hexagon.cfg: 1 file pushed. 0.6 MB/s (3270 bytes in 0.005s)
/sdcard/t5-very-small-random-F32.gguf
the prebuild LLM model t5-very-small-random-F32.gguf already exist on Android phone
/sdcard/qwen1_5-1_8b-chat-q4_0.gguf
the prebuild LLM model qwen1_5-1_8b-chat-q4_0.gguf already exist on Android phone
/sdcard/gemma-3n-E2B-it-Q8_0.gguf
the prebuild LLM model gemma-3n-E2B-it-Q8_0.gguf already exist on Android phone
./out/ggmlhexagon-android/bin/libggml-base.so: 1 file pushed. 30.3 MB/s (6656456 bytes in 0.210s)
./out/ggmlhexagon-android/bin/libggml-cpu.so: 1 file pushed. 29.2 MB/s (3580440 bytes in 0.117s)
./out/ggmlhexagon-android/bin/libggmldsp-skel.so: 1 file pushed. 3.5 MB/s (34080 bytes in 0.009s)
./out/ggmlhexagon-android/bin/libggmldsp-skelv79.so: 1 file pushed. 3.4 MB/s (34080 bytes in 0.010s)
./out/ggmlhexagon-android/bin/libggml-hexagon.so: 1 file pushed. 30.8 MB/s (5847712 bytes in 0.181s)
./out/ggmlhexagon-android/bin/libggml.so: 1 file pushed. 29.6 MB/s (1761624 bytes in 0.057s)
./out/ggmlhexagon-android/bin/libllama.so: 1 file pushed. 30.4 MB/s (20835656 bytes in 0.653s)
./out/ggmlhexagon-android/bin/libmtmd.so: 1 file pushed. 29.9 MB/s (4859056 bytes in 0.155s)
8 files pushed. 29.8 MB/s (43609104 bytes in 1.395s)
./out/ggmlhexagon-android/bin/llama-cli: 1 file pushed. 30.9 MB/s (27625608 bytes in 0.853s)
-rwxrwxrwx 1 shell shell  6656456 2025-07-08 10:33 /data/local/tmp/libggml-base.so
-rwxrwxrwx 1 shell shell  3580440 2025-07-08 10:33 /data/local/tmp/libggml-cpu.so
-rwxrwxrwx 1 shell shell  5847712 2025-07-08 10:33 /data/local/tmp/libggml-hexagon.so
-rwxrwxrwx 1 shell shell  1476880 2025-07-08 10:30 /data/local/tmp/libggml-opencl.so
-rwxrwxrwx 1 shell shell 25159984 2025-07-08 10:32 /data/local/tmp/libggml-vulkan.so
./scripts/ggml-hexagon-for-binary-lib.cfg: 1 file pushed. 1.0 MB/s (3381 bytes in 0.003s)
adb push /home/zhouwg/kantvai/llama.cpp/prebuilts/ggml-dsp/20250627/libggmldsp-skelv79.so /data/local/tmp/libggmldsp-skel.so
/home/zhouwg/kantvai/llama.cpp/prebuilts/ggml-dsp/20250627/libggmldsp-skelv79.so: 1 file pushed. 30.0 MB/s (1099976 bytes in 0.035s)
/data/local/tmp/llama-cli  -ngl 99 -t 4 -n 256 --no-warmup  -mg 4 -no-cnv -m /sdcard/qwen1_5-1_8b-chat-q4_0.gguf -p "introduce the movie Once Upon a Time in America briefly.\n"
backend 4
warning: no usable GPU found, --gpu-layers option will be ignored
warning: one possible reason is that llama.cpp was compiled without GPU support
warning: consult docs/build.md for compilation instructions
warning: llama.cpp was compiled without support for GPU offload. Setting the main GPU has no effect.
build: 6045 (cff201cf3) with Android (12896553, +pgo, +bolt, +lto, +mlgo, based on r530567c) clang version 19.0.0 (https://android.googlesource.com/toolchain/llvm-project 97a699bf4812a18fb657c2779f5296a4ab2694d2) for x86_64-unknown-linux-gnu
main: llama backend init
main: load the model and apply lora adapter, if any
llama_model_loader: loaded meta data with 21 key-value pairs and 291 tensors from /sdcard/qwen1_5-1_8b-chat-q4_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen2
llama_model_loader: - kv   1:                               general.name str              = Qwen1.5-1.8B-Chat-AWQ-fp16
llama_model_loader: - kv   2:                          qwen2.block_count u32              = 24
llama_model_loader: - kv   3:                       qwen2.context_length u32              = 32768
llama_model_loader: - kv   4:                     qwen2.embedding_length u32              = 2048
llama_model_loader: - kv   5:                  qwen2.feed_forward_length u32              = 5504
llama_model_loader: - kv   6:                 qwen2.attention.head_count u32              = 16
llama_model_loader: - kv   7:              qwen2.attention.head_count_kv u32              = 16
llama_model_loader: - kv   8:     qwen2.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv   9:                       qwen2.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  10:                qwen2.use_parallel_residual bool             = true
llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,151936]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  13:                  tokenizer.ggml.token_type arr[i32,151936]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  14:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv  15:                tokenizer.ggml.eos_token_id u32              = 151645
llama_model_loader: - kv  16:            tokenizer.ggml.padding_token_id u32              = 151643
llama_model_loader: - kv  17:                tokenizer.ggml.bos_token_id u32              = 151643
llama_model_loader: - kv  18:                    tokenizer.chat_template str              = {% for message in messages %}{{'<|im_...
llama_model_loader: - kv  19:               general.quantization_version u32              = 2
llama_model_loader: - kv  20:                          general.file_type u32              = 2
llama_model_loader: - type  f32:  121 tensors
llama_model_loader: - type q4_0:  169 tensors
llama_model_loader: - type q6_K:    1 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_0
print_info: file size   = 1.04 GiB (4.85 BPW) 
load: missing pre-tokenizer type, using: 'default'
load:                                             
load: ************************************        
load: GENERATION QUALITY WILL BE DEGRADED!        
load: CONSIDER REGENERATING THE MODEL             
load: ************************************        
load:                                             
load: special tokens cache size = 293
load: token to piece cache size = 0.9338 MB
print_info: arch             = qwen2
print_info: vocab_only       = 0
print_info: n_ctx_train      = 32768
print_info: n_embd           = 2048
print_info: n_layer          = 24
print_info: n_head           = 16
print_info: n_head_kv        = 16
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 1
print_info: n_embd_k_gqa     = 2048
print_info: n_embd_v_gqa     = 2048
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-06
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 5504
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: causal attn      = 1
print_info: pooling type     = -1
print_info: rope type        = 2
print_info: rope scaling     = linear
print_info: freq_base_train  = 1000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 32768
print_info: rope_finetuned   = unknown
print_info: model type       = 1B
print_info: model params     = 1.84 B
print_info: general.name     = Qwen1.5-1.8B-Chat-AWQ-fp16
print_info: vocab type       = BPE
print_info: n_vocab          = 151936
print_info: n_merges         = 151387
print_info: BOS token        = 151643 '<|endoftext|>'
print_info: EOS token        = 151645 '<|im_end|>'
print_info: EOT token        = 151645 '<|im_end|>'
print_info: PAD token        = 151643 '<|endoftext|>'
print_info: LF token         = 198 'Ċ'
print_info: EOG token        = 151643 '<|endoftext|>'
print_info: EOG token        = 151645 '<|im_end|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors:   CPU_Mapped model buffer size =  1062.67 MiB
load_tensors:   CPU_REPACK model buffer size =   651.38 MiB
................................................................
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 4096
llama_context: n_ctx_per_seq = 4096
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = 0
llama_context: freq_base     = 1000000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_per_seq (4096) < n_ctx_train (32768) -- the full capacity of the model will not be utilized
llama_context:        CPU  output buffer size =     0.58 MiB
llama_kv_cache_unified:        CPU KV buffer size =   768.00 MiB
llama_kv_cache_unified: size =  768.00 MiB (  4096 cells,  24 layers,  1 seqs), K (f16):  384.00 MiB, V (f16):  384.00 MiB
llama_kv_cache_unified: LLAMA_SET_ROWS=0, using old ggml_cpy() method for backwards compatibility
llama_context:        CPU compute buffer size =   300.75 MiB
llama_context: graph nodes  = 942
llama_context: graph splits = 1
common_init_from_params: setting dry_penalty_last_n to ctx_size = 4096
main: llama threadpool init, n_threads = 4
introduce the movie Once Upon a Time in America briefly.

system_info: n_threads = 4 (n_threads_batch = 4) / 8 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | MATMUL_INT8 = 1 | DOTPROD = 1 | REPACK = 1 | 

sampler seed: 471864632
sampler params: 
	repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
	dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = 4096
	top_k = 40, top_p = 0.950, min_p = 0.050, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, top_n_sigma = -1.000, temp = 0.800
	mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
sampler chain: logits -> logit-bias -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist 
generate: n_ctx = 4096, n_batch = 2048, n_predict = 256, n_keep = 0

Once Upon a Time in America is a 1984 American crime film directed by Elie Sperber and starring Robert DeNiro, Alan Rickman, and Billy Crystal. The film is a crime drama that tells the story of a Jewish mob boss named Jack Sackman (played by DeNiro) and his family's struggles with crime, corruption, and the struggle for identity in the 1920s and 1930s in New York City.
The movie opens with Jack Sackman, a charismatic and ruthless mob boss, operating his criminal empire in the 1920s. He and his family, including his wife, Rachel (played by Alan Rickman), his son, Jacob (played by Billy Crystal), and his daughter, Ruth (played by Rosanna De'Vino), are the driving forces of the mob, but their actions have consequences that ripple through the city's criminal underworld.
The film explores themes of family, loyalty, and the corruption of power. As Jack's family becomes increasingly intertwined with the criminal world, they must navigate their own personal lives as well as the complexities of the mob's operations. They must confront their own fears and desires, as well as the moral dilemmas they face

llama_perf_sampler_print:    sampling time =      20.82 ms /   269 runs   (    0.08 ms per token, 12917.79 tokens per second)
llama_perf_context_print:        load time =     840.45 ms
llama_perf_context_print: prompt eval time =      93.43 ms /    13 tokens (    7.19 ms per token,   139.14 tokens per second)
llama_perf_context_print:        eval time =    6907.60 ms /   255 runs   (   27.09 ms per token,    36.92 tokens per second)
llama_perf_context_print:       total time =    7785.98 ms /   268 tokens

llama-cli via the candidate ggml-hexagon on 8Elite phone (for avoid misleading, this testcase only directly offload GGML_OP_ADD to cDSP(aka Hexagon NPU), I already try my best to optimize GGML_OP_ADD on cDSP side and the performance of fp32 4096x4096 matrix addition is better than QNN-CPU,QNN-GPU,QNN-NPU, the default ggml backend at the moment)

Details

zhouwg:$ ./scripts/build-run-ggmlhexagon-android.sh run_llamacli 3
current working path:/home/zhouwg/kantvai/llama.cpp

/usr/bin/wget

/usr/bin/xzcat

/bin/ls
Android NDK already exist:         /home/zhouwg/kantvai/llama.cpp/prebuilts/android-ndk-r28 

Qualcomm QNN SDK already exist:    /home/zhouwg/kantvai/llama.cpp/prebuilts/QNN_SDK/qairt/2.36.0.250627/ 

Qualcomm Hexagon SDK already exist:/home/zhouwg/kantvai/llama.cpp/prebuilts/Hexagon_SDK/6.2.0.1 

/sdcard/t5-very-small-random-F32.gguf
the prebuild LLM model t5-very-small-random-F32.gguf already exist on Android phone
/sdcard/qwen1_5-1_8b-chat-q4_0.gguf
the prebuild LLM model qwen1_5-1_8b-chat-q4_0.gguf already exist on Android phone
/sdcard/gemma-3n-E2B-it-Q8_0.gguf
the prebuild LLM model gemma-3n-E2B-it-Q8_0.gguf already exist on Android phone
/data/local/tmp/libQnnCpu.so
/data/local/tmp/libQnnGpu.so
/data/local/tmp/libQnnHtp.so
QNN runtime libs already exist on Android phone
./scripts/ggml-hexagon.cfg: 1 file pushed. 1.2 MB/s (3270 bytes in 0.003s)
/sdcard/t5-very-small-random-F32.gguf
the prebuild LLM model t5-very-small-random-F32.gguf already exist on Android phone
/sdcard/qwen1_5-1_8b-chat-q4_0.gguf
the prebuild LLM model qwen1_5-1_8b-chat-q4_0.gguf already exist on Android phone
/sdcard/gemma-3n-E2B-it-Q8_0.gguf
the prebuild LLM model gemma-3n-E2B-it-Q8_0.gguf already exist on Android phone
./out/ggmlhexagon-android/bin/libggml-base.so: 1 file pushed. 18.1 MB/s (6656456 bytes in 0.351s)
./out/ggmlhexagon-android/bin/libggml-cpu.so: 1 file pushed. 17.9 MB/s (3580440 bytes in 0.191s)
./out/ggmlhexagon-android/bin/libggmldsp-skel.so: 1 file pushed. 3.8 MB/s (34080 bytes in 0.009s)
./out/ggmlhexagon-android/bin/libggmldsp-skelv79.so: 1 file pushed. 4.5 MB/s (34080 bytes in 0.007s)
./out/ggmlhexagon-android/bin/libggml-hexagon.so: 1 file pushed. 18.1 MB/s (5847712 bytes in 0.309s)
./out/ggmlhexagon-android/bin/libggml.so: 1 file pushed. 28.5 MB/s (1761624 bytes in 0.059s)
./out/ggmlhexagon-android/bin/libllama.so: 1 file pushed. 18.5 MB/s (20835656 bytes in 1.076s)
./out/ggmlhexagon-android/bin/libmtmd.so: 1 file pushed. 18.0 MB/s (4859056 bytes in 0.258s)
8 files pushed. 18.4 MB/s (43609104 bytes in 2.263s)
./out/ggmlhexagon-android/bin/llama-cli: 1 file pushed. 18.4 MB/s (27625608 bytes in 1.433s)
-rwxrwxrwx 1 shell shell  6656456 2025-07-08 10:33 /data/local/tmp/libggml-base.so
-rwxrwxrwx 1 shell shell  3580440 2025-07-08 10:33 /data/local/tmp/libggml-cpu.so
-rwxrwxrwx 1 shell shell  5847712 2025-07-08 10:33 /data/local/tmp/libggml-hexagon.so
-rwxrwxrwx 1 shell shell  1476880 2025-07-08 10:30 /data/local/tmp/libggml-opencl.so
-rwxrwxrwx 1 shell shell 25159984 2025-07-08 10:32 /data/local/tmp/libggml-vulkan.so
./scripts/ggml-hexagon-for-binary-lib.cfg: 1 file pushed. 0.5 MB/s (3381 bytes in 0.007s)
adb push /home/zhouwg/kantvai/llama.cpp/prebuilts/ggml-dsp/20250627/libggmldsp-skelv79.so /data/local/tmp/libggmldsp-skel.so
/home/zhouwg/kantvai/llama.cpp/prebuilts/ggml-dsp/20250627/libggmldsp-skelv79.so: 1 file pushed. 17.9 MB/s (1099976 bytes in 0.059s)
/data/local/tmp/llama-cli  -ngl 99 -t 4 -n 256 --no-warmup  -mg 3 -no-cnv -m /sdcard/qwen1_5-1_8b-chat-q4_0.gguf -p "introduce the movie Once Upon a Time in America briefly.\n"
backend 3
[ggmlhexagon_init_dsp, 5653]: using Hexagon domain 3(Hexagon-cDSP)
[ggmlhexagon_init_dsp, 5654]: unsignedpd_enabled 1
[ggmlhexagon_init_dsp, 5696]: succeed to open domain 3(Hexagon-cDSP)
[ggmlhexagon_init_dsp, 5697]: only support offload GGML_OP_ADD and GGML_OP_MUL_MAT to cDSP currently
[ggmlhexagon_probe_dspinfo, 5478]: dsp arch version 0x79
[ggmlhexagon_probe_dspinfo, 5489]: device info: Qualcomm SnapDragon 8 Elite, QCOM_HTP_V79
[ggmlhexagon_probe_dspinfo, 5519]: vtcm_count 1
[ggmlhexagon_probe_dspinfo, 5520]: vtcm_page 8388608
[ggmlhexagon_probe_dspinfo, 5521]: hmx_depth 0
[ggmlhexagon_probe_dspinfo, 5522]: hmx_spatial 0
[ggmlhexagon_probe_dspinfo, 5523]: hvx_support_128b 6
[ggmlhexagon_probe_dspinfo, 5524]: unsigned pd supported 1
[ggmlhexagon_probe_dspinfo, 5525]: async fastrpc supported 0
[ggmlhexagon_set_rpc_latency, 5163]: set rpc qos 3, latency 100

[ggmlhexagon_init_rpcmempool, 5428]: capacity of rpc memory 2000 MiB
build: 6045 (cff201cf3) with Android (12896553, +pgo, +bolt, +lto, +mlgo, based on r530567c) clang version 19.0.0 (https://android.googlesource.com/toolchain/llvm-project 97a699bf4812a18fb657c2779f5296a4ab2694d2) for x86_64-unknown-linux-gnu
main: llama backend init
main: load the model and apply lora adapter, if any
llama_model_load_from_file_impl: using device Hexagon-cDSP (Qualcomm NPU(cDSP)) - 2000 MiB free
llama_model_loader: loaded meta data with 21 key-value pairs and 291 tensors from /sdcard/qwen1_5-1_8b-chat-q4_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen2
llama_model_loader: - kv   1:                               general.name str              = Qwen1.5-1.8B-Chat-AWQ-fp16
llama_model_loader: - kv   2:                          qwen2.block_count u32              = 24
llama_model_loader: - kv   3:                       qwen2.context_length u32              = 32768
llama_model_loader: - kv   4:                     qwen2.embedding_length u32              = 2048
llama_model_loader: - kv   5:                  qwen2.feed_forward_length u32              = 5504
llama_model_loader: - kv   6:                 qwen2.attention.head_count u32              = 16
llama_model_loader: - kv   7:              qwen2.attention.head_count_kv u32              = 16
llama_model_loader: - kv   8:     qwen2.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv   9:                       qwen2.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  10:                qwen2.use_parallel_residual bool             = true
llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,151936]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  13:                  tokenizer.ggml.token_type arr[i32,151936]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  14:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv  15:                tokenizer.ggml.eos_token_id u32              = 151645
llama_model_loader: - kv  16:            tokenizer.ggml.padding_token_id u32              = 151643
llama_model_loader: - kv  17:                tokenizer.ggml.bos_token_id u32              = 151643
llama_model_loader: - kv  18:                    tokenizer.chat_template str              = {% for message in messages %}{{'<|im_...
llama_model_loader: - kv  19:               general.quantization_version u32              = 2
llama_model_loader: - kv  20:                          general.file_type u32              = 2
llama_model_loader: - type  f32:  121 tensors
llama_model_loader: - type q4_0:  169 tensors
llama_model_loader: - type q6_K:    1 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_0
print_info: file size   = 1.04 GiB (4.85 BPW) 
load: missing pre-tokenizer type, using: 'default'
load:                                             
load: ************************************        
load: GENERATION QUALITY WILL BE DEGRADED!        
load: CONSIDER REGENERATING THE MODEL             
load: ************************************        
load:                                             
load: special tokens cache size = 293
load: token to piece cache size = 0.9338 MB
print_info: arch             = qwen2
print_info: vocab_only       = 0
print_info: n_ctx_train      = 32768
print_info: n_embd           = 2048
print_info: n_layer          = 24
print_info: n_head           = 16
print_info: n_head_kv        = 16
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 1
print_info: n_embd_k_gqa     = 2048
print_info: n_embd_v_gqa     = 2048
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-06
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 5504
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: causal attn      = 1
print_info: pooling type     = -1
print_info: rope type        = 2
print_info: rope scaling     = linear
print_info: freq_base_train  = 1000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 32768
print_info: rope_finetuned   = unknown
print_info: model type       = 1B
print_info: model params     = 1.84 B
print_info: general.name     = Qwen1.5-1.8B-Chat-AWQ-fp16
print_info: vocab type       = BPE
print_info: n_vocab          = 151936
print_info: n_merges         = 151387
print_info: BOS token        = 151643 '<|endoftext|>'
print_info: EOS token        = 151645 '<|im_end|>'
print_info: EOT token        = 151645 '<|im_end|>'
print_info: PAD token        = 151643 '<|endoftext|>'
print_info: LF token         = 198 'Ċ'
print_info: EOG token        = 151643 '<|endoftext|>'
print_info: EOG token        = 151645 '<|im_end|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: offloading 24 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 25/25 layers to GPU
load_tensors: hexagon-ion-buffer model buffer size =     0.56 MiB
load_tensors:   CPU_REPACK model buffer size =   651.38 MiB
load_tensors:   CPU_Mapped model buffer size =  1062.67 MiB
................................................................
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 4096
llama_context: n_ctx_per_seq = 4096
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = 0
llama_context: freq_base     = 1000000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_per_seq (4096) < n_ctx_train (32768) -- the full capacity of the model will not be utilized
llama_context:        CPU  output buffer size =     0.58 MiB
llama_kv_cache_unified: hexagon-ion-buffer KV buffer size =   768.00 MiB
llama_kv_cache_unified: size =  768.00 MiB (  4096 cells,  24 layers,  1 seqs), K (f16):  384.00 MiB, V (f16):  384.00 MiB
llama_kv_cache_unified: LLAMA_SET_ROWS=0, using old ggml_cpy() method for backwards compatibility
llama_context:        CPU compute buffer size =   300.75 MiB
llama_context: graph nodes  = 942
llama_context: graph splits = 1 (with bs=512), 145 (with bs=1)
common_init_from_params: setting dry_penalty_last_n to ctx_size = 4096
main: llama threadpool init, n_threads = 4

system_info: n_threads = 4 (n_threads_batch = 4) / 8 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | MATMUL_INT8 = 1 | DOTPROD = 1 | REPACK = 1 | 

introduce the movie Once Uponsampler seed: 1814123497
sampler params: 
	repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
	dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = 4096
	top_k = 40, top_p = 0.950, min_p = 0.050, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, top_n_sigma = -1.000, temp = 0.800
	mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
sampler chain: logits -> logit-bias -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist 
generate: n_ctx = 4096, n_batch = 2048, n_predict = 256, n_keep = 0

 a Time in America briefly.
Once Upon a Time in America is a 1981 American crime film directed by Martin Scorsese and starring Robert De Niro, Al Pacino, and James Caan. The movie is set in New York City during the 1950s and 1960s and follows the lives of three families: the Corleone, the Soprano, and the Jewish mob.
The Corleone family, led by patriarch Corleone (De Niro), is a powerful mafia family that operates in New York's seedy underbelly. They are involved in various criminal activities, including murder, extortion, and loan shark business. The Corleone family's patriarch, Michael Corleone (De Niro), is a man of honor who is dedicated to maintaining the family's reputation and protecting his loved ones.
The Soprano family, led by patriarch Silvestro Soprano (Al Pacino), is a more suburban family that operates in the criminal underworld of New York City. They are involved in drug trafficking, extortion, and money laundering. Silvestro Soprano is a cunning and calculating man who is always one step ahead of his enemies.
The Jewish mob, led by the head of the mob,

llama_perf_sampler_print:    sampling time =      32.56 ms /   269 runs   (    0.12 ms per token,  8260.40 tokens per second)
llama_perf_context_print:        load time =     852.19 ms
llama_perf_context_print: prompt eval time =      99.65 ms /    13 tokens (    7.67 ms per token,   130.46 tokens per second)
llama_perf_context_print:        eval time =    9815.76 ms /   255 runs   (   38.49 ms per token,    25.98 tokens per second)
llama_perf_context_print:       total time =   10733.28 ms /   268 tokens
[ggmlhexagon_print_running_timestamp, 2140]: ggml_hexagon_version:             1.13
[ggmlhexagon_print_running_timestamp, 2141]: ggml_dsp_version:                 0.98.8
[ggmlhexagon_print_running_timestamp, 2143]: hwaccel approach:                 2(HWACCEL_CDSP)
[ggmlhexagon_print_running_timestamp, 2145]: hexagon_backend:                  3(HEXAGON_BACKEND_CDSP)
[ggmlhexagon_print_running_timestamp, 2146]: enable pinned_memory:             NO
[ggmlhexagon_print_running_timestamp, 2149]: offload quantize GGML_OP_MUL_MAT: NO
[ggmlhexagon_print_running_timestamp, 2150]: using rpc ion memory pool:        YES
[ggmlhexagon_print_running_timestamp, 2151]: thread_counts with HWACCEL_CDSP:  8
[ggmlhexagon_print_running_timestamp, 2152]: mulmat algo type on cDSP:         0
[ggmlhexagon_probe_dspinfo, 5478]: dsp arch version 0x79
[ggmlhexagon_probe_dspinfo, 5489]: device info: Qualcomm SnapDragon 8 Elite, QCOM_HTP_V79
[ggmlhexagon_probe_dspinfo, 5519]: vtcm_count 1
[ggmlhexagon_probe_dspinfo, 5520]: vtcm_page 8388608
[ggmlhexagon_probe_dspinfo, 5521]: hmx_depth 0
[ggmlhexagon_probe_dspinfo, 5522]: hmx_spatial 0
[ggmlhexagon_probe_dspinfo, 5523]: hvx_support_128b 6
[ggmlhexagon_probe_dspinfo, 5524]: unsigned pd supported 1
[ggmlhexagon_probe_dspinfo, 5525]: async fastrpc supported 0
[ggmlhexagon_print_running_timestamp, 2158]: running timestamp:2025-07-08,10:41:08
[ggmlhexagon_deinit_cdsp, 5534]: enter ggmlhexagon_deinit_cdsp
[ggmlhexagon_deinit_cdsp, 5550]: leave ggmlhexagon_deinit_cdsp

llama-cli via the candidate ggml-hexagon on 8Elite phone (offload fp32 MUL_MAT, q4_0 MUL_MAT, fp32 ADD to cDSP directly)

forground running logs:

Details

zhouwg:$ ./scripts/build-run-android.sh run_llamacli 3
current working path:/home/zhouwg/kantvai/llama.cpp

/usr/bin/wget

/usr/bin/xzcat

/bin/ls
Android NDK already exist:         /home/zhouwg/kantvai/llama.cpp/prebuilts/android-ndk-r28 

Qualcomm QNN SDK already exist:    /home/zhouwg/kantvai/llama.cpp/prebuilts/QNN_SDK/qairt/2.36.0.250627/ 

Qualcomm Hexagon SDK already exist:/home/zhouwg/kantvai/llama.cpp/prebuilts/Hexagon_SDK/6.2.0.1 

/sdcard/t5-very-small-random-F32.gguf
the prebuild LLM model t5-very-small-random-F32.gguf already exist on Android phone
/sdcard/qwen1_5-1_8b-chat-q4_0.gguf
the prebuild LLM model qwen1_5-1_8b-chat-q4_0.gguf already exist on Android phone
/sdcard/gemma-3n-E2B-it-Q8_0.gguf
the prebuild LLM model gemma-3n-E2B-it-Q8_0.gguf already exist on Android phone
/data/local/tmp/libQnnCpu.so
/data/local/tmp/libQnnGpu.so
/data/local/tmp/libQnnHtp.so
QNN runtime libs already exist on Android phone
./scripts/ggml-hexagon.cfg: 1 file pushed. 0.5 MB/s (3270 bytes in 0.007s)
/sdcard/t5-very-small-random-F32.gguf
the prebuild LLM model t5-very-small-random-F32.gguf already exist on Android phone
/sdcard/qwen1_5-1_8b-chat-q4_0.gguf
the prebuild LLM model qwen1_5-1_8b-chat-q4_0.gguf already exist on Android phone
/sdcard/gemma-3n-E2B-it-Q8_0.gguf
the prebuild LLM model gemma-3n-E2B-it-Q8_0.gguf already exist on Android phone
./out/ggmlhexagon-android/bin/libggml-base.so: 1 file pushed. 16.9 MB/s (6656456 bytes in 0.375s)
./out/ggmlhexagon-android/bin/libggml-cpu.so: 1 file pushed. 16.9 MB/s (3580440 bytes in 0.202s)
./out/ggmlhexagon-android/bin/libggmldsp-skel.so: 1 file pushed. 3.7 MB/s (38496 bytes in 0.010s)
./out/ggmlhexagon-android/bin/libggmldsp-skelv79.so: 1 file pushed. 3.8 MB/s (38496 bytes in 0.010s)
./out/ggmlhexagon-android/bin/libggml-hexagon.so: 1 file pushed. 17.1 MB/s (5849016 bytes in 0.325s)
./out/ggmlhexagon-android/bin/libggml.so: 1 file pushed. 17.2 MB/s (1761624 bytes in 0.098s)
./out/ggmlhexagon-android/bin/libllama.so: 1 file pushed. 18.4 MB/s (20836456 bytes in 1.078s)
./out/ggmlhexagon-android/bin/libmtmd.so: 1 file pushed. 17.8 MB/s (4859056 bytes in 0.260s)
8 files pushed. 17.6 MB/s (43620040 bytes in 2.363s)
./out/ggmlhexagon-android/bin/llama-cli: 1 file pushed. 20.0 MB/s (27625608 bytes in 1.321s)
-rwxrwxrwx 1 shell shell  6656456 2025-07-07 09:56 /data/local/tmp/libggml-base.so
-rwxrwxrwx 1 shell shell  3580440 2025-07-07 09:56 /data/local/tmp/libggml-cpu.so
-rwxrwxrwx 1 shell shell  5849016 2025-07-07 09:56 /data/local/tmp/libggml-hexagon.so
-rwxrwxrwx 1 shell shell  1476880 2025-07-06 12:42 /data/local/tmp/libggml-opencl.so
-rwxrwxrwx 1 shell shell 25159984 2025-07-06 22:55 /data/local/tmp/libggml-vulkan.so
./scripts/ggml-hexagon-for-binary-lib.cfg: 1 file pushed. 0.7 MB/s (3381 bytes in 0.005s)
adb push /home/zhouwg/kantvai/llama.cpp/prebuilts/ggml-dsp/20250627/libggmldsp-skelv79.so /data/local/tmp/libggmldsp-skel.so
/home/zhouwg/kantvai/llama.cpp/prebuilts/ggml-dsp/20250627/libggmldsp-skelv79.so: 1 file pushed. 19.0 MB/s (1099976 bytes in 0.055s)
/data/local/tmp/llama-cli  -ngl 99 -t 4 -n 256 --no-warmup  -mg 3 -no-cnv -m /sdcard/qwen1_5-1_8b-chat-q4_0.gguf -p "introduce the movie Once Upon a Time in America briefly.\n"
backend 3
[ggmlhexagon_init_dsp, 5654]: using Hexagon domain 3(Hexagon-cDSP)
[ggmlhexagon_init_dsp, 5655]: unsignedpd_enabled 1
[ggmlhexagon_init_dsp, 5697]: succeed to open domain 3(Hexagon-cDSP)
[ggmlhexagon_init_dsp, 5698]: only support offload GGML_OP_ADD and GGML_OP_MUL_MAT to cDSP currently
[ggmlhexagon_probe_dspinfo, 5479]: dsp arch version 0x79
[ggmlhexagon_probe_dspinfo, 5490]: device info: Qualcomm SnapDragon 8 Elite, QCOM_HTP_V79
[ggmlhexagon_probe_dspinfo, 5520]: vtcm_count 1
[ggmlhexagon_probe_dspinfo, 5521]: vtcm_page 8388608
[ggmlhexagon_probe_dspinfo, 5522]: hmx_depth 0
[ggmlhexagon_probe_dspinfo, 5523]: hmx_spatial 0
[ggmlhexagon_probe_dspinfo, 5524]: hvx_support_128b 6
[ggmlhexagon_probe_dspinfo, 5525]: unsigned pd supported 1
[ggmlhexagon_probe_dspinfo, 5526]: async fastrpc supported 0
[ggmlhexagon_set_rpc_latency, 5164]: set rpc qos 3, latency 100

[ggmlhexagon_init_rpcmempool, 5429]: capacity of rpc memory 2000 MiB
build: 6033 (c41f7c11c) with Android (12896553, +pgo, +bolt, +lto, +mlgo, based on r530567c) clang version 19.0.0 (https://android.googlesource.com/toolchain/llvm-project 97a699bf4812a18fb657c2779f5296a4ab2694d2) for x86_64-unknown-linux-gnu
main: llama backend init
main: load the model and apply lora adapter, if any
llama_model_load_from_file_impl: using device Hexagon-cDSP (Qualcomm NPU(cDSP)) - 2000 MiB free
llama_model_loader: loaded meta data with 21 key-value pairs and 291 tensors from /sdcard/qwen1_5-1_8b-chat-q4_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen2
llama_model_loader: - kv   1:                               general.name str              = Qwen1.5-1.8B-Chat-AWQ-fp16
llama_model_loader: - kv   2:                          qwen2.block_count u32              = 24
llama_model_loader: - kv   3:                       qwen2.context_length u32              = 32768
llama_model_loader: - kv   4:                     qwen2.embedding_length u32              = 2048
llama_model_loader: - kv   5:                  qwen2.feed_forward_length u32              = 5504
llama_model_loader: - kv   6:                 qwen2.attention.head_count u32              = 16
llama_model_loader: - kv   7:              qwen2.attention.head_count_kv u32              = 16
llama_model_loader: - kv   8:     qwen2.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv   9:                       qwen2.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  10:                qwen2.use_parallel_residual bool             = true
llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,151936]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  13:                  tokenizer.ggml.token_type arr[i32,151936]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  14:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv  15:                tokenizer.ggml.eos_token_id u32              = 151645
llama_model_loader: - kv  16:            tokenizer.ggml.padding_token_id u32              = 151643
llama_model_loader: - kv  17:                tokenizer.ggml.bos_token_id u32              = 151643
llama_model_loader: - kv  18:                    tokenizer.chat_template str              = {% for message in messages %}{{'<|im_...
llama_model_loader: - kv  19:               general.quantization_version u32              = 2
llama_model_loader: - kv  20:                          general.file_type u32              = 2
llama_model_loader: - type  f32:  121 tensors
llama_model_loader: - type q4_0:  169 tensors
llama_model_loader: - type q6_K:    1 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_0
print_info: file size   = 1.04 GiB (4.85 BPW) 
load: missing pre-tokenizer type, using: 'default'
load:                                             
load: ************************************        
load: GENERATION QUALITY WILL BE DEGRADED!        
load: CONSIDER REGENERATING THE MODEL             
load: ************************************        
load:                                             
load: special tokens cache size = 293
load: token to piece cache size = 0.9338 MB
print_info: arch             = qwen2
print_info: vocab_only       = 0
print_info: n_ctx_train      = 32768
print_info: n_embd           = 2048
print_info: n_layer          = 24
print_info: n_head           = 16
print_info: n_head_kv        = 16
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 1
print_info: n_embd_k_gqa     = 2048
print_info: n_embd_v_gqa     = 2048
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-06
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 5504
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: causal attn      = 1
print_info: pooling type     = -1
print_info: rope type        = 2
print_info: rope scaling     = linear
print_info: freq_base_train  = 1000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 32768
print_info: rope_finetuned   = unknown
print_info: model type       = 1B
print_info: model params     = 1.84 B
print_info: general.name     = Qwen1.5-1.8B-Chat-AWQ-fp16
print_info: vocab type       = BPE
print_info: n_vocab          = 151936
print_info: n_merges         = 151387
print_info: BOS token        = 151643 '<|endoftext|>'
print_info: EOS token        = 151645 '<|im_end|>'
print_info: EOT token        = 151645 '<|im_end|>'
print_info: PAD token        = 151643 '<|endoftext|>'
print_info: LF token         = 198 'Ċ'
print_info: EOG token        = 151643 '<|endoftext|>'
print_info: EOG token        = 151645 '<|im_end|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: offloading 24 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 25/25 layers to GPU
load_tensors: hexagon-ion-buffer model buffer size =   895.37 MiB
load_tensors:   CPU_Mapped model buffer size =   819.24 MiB
.................................................................
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 4096
llama_context: n_ctx_per_seq = 4096
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = 0
llama_context: freq_base     = 1000000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_per_seq (4096) < n_ctx_train (32768) -- the full capacity of the model will not be utilized
llama_context:        CPU  output buffer size =     0.58 MiB
llama_kv_cache_unified: hexagon-ion-buffer KV buffer size =   768.00 MiB
llama_kv_cache_unified: size =  768.00 MiB (  4096 cells,  24 layers,  1 seqs), K (f16):  384.00 MiB, V (f16):  384.00 MiB
llama_kv_cache_unified: LLAMA_SET_ROWS=0, using old ggml_cpy() method for backwards compatibility
llama_context: hexagon-ion-buffer compute buffer size =   300.75 MiB
llama_context:        CPU compute buffer size =   148.01 MiB
llama_context: graph nodes  = 942
llama_context: graph splits = 292 (with bs=512), 145 (with bs=1)
common_init_from_params: setting dry_penalty_last_n to ctx_size = 4096
main: llama threadpool init, n_threads = 4

system_info: n_threads = 4 (n_threads_batch = 4) / 8 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | MATMUL_INT8 = 1 | DOTPROD = 1 | REPACK = 1 | 

sampler seed: 1534343090
sampler params: 
	repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
	dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = 4096
	top_k = 40, top_p = 0.950, min_p = 0.050, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, top_n_sigma = -1.000, temp = 0.800
	mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
introduce the movie Once Upon a Time in America brieflysampler chain: logits -> logit-bias -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist 
generate: n_ctx = 4096, n_batch = 2048, n_predict = 256, n_keep = 0

.
Once Upon a Time in America is a 1984 American film directed by Martin Scorsese, and produced by Scorsese, Robert Altman, and Frank Darabont. The film is based on the novel by William S. Burroughs and tells the story of the life of Aaron Burr, a New York City detective who is involved in the murder of his friend, Joe Clark, and the subsequent trial and execution of his brother, Jack.

The film opens with Burr, played by Robert De Niro, as a young man working as a police officer in the New York City's Lower East Side. He falls in love with a young woman named Emily, played by Meryl Streep, and they marry. However, their relationship is complicated by Burr's complicated past, including his involvement in a crime that resulted in the murder of Clark.

Burr becomes involved in the investigation of Clark's murder, which leads to a series of complex relationships and betrayals, including his own partner, Detective Harry Stone (played by Al Pacino), who is also investigating the case. Burr's relationships with other characters, including his wife, his son, and his friend, Jack, all play a role in the plot.

As the investigation progresses, Burr becomes increasingly

llama_perf_sampler_print:    sampling time =      41.74 ms /   269 runs   (    0.16 ms per token,  6444.50 tokens per second)
llama_perf_context_print:        load time =   18819.06 ms
llama_perf_context_print: prompt eval time =   17658.25 ms /    13 tokens ( 1358.33 ms per token,     0.74 tokens per second)
llama_perf_context_print:        eval time =   13614.33 ms /   255 runs   (   53.39 ms per token,    18.73 tokens per second)
llama_perf_context_print:       total time =   32513.51 ms /   268 tokens
[ggmlhexagon_print_running_timestamp, 2141]: ggml_hexagon_version:             1.13
[ggmlhexagon_print_running_timestamp, 2142]: ggml_dsp_version:                 0.98.8
[ggmlhexagon_print_running_timestamp, 2144]: hwaccel approach:                 2(HWACCEL_CDSP)
[ggmlhexagon_print_running_timestamp, 2146]: hexagon_backend:                  3(HEXAGON_BACKEND_CDSP)
[ggmlhexagon_print_running_timestamp, 2147]: enable pinned_memory:             NO
[ggmlhexagon_print_running_timestamp, 2150]: offload quantize GGML_OP_MUL_MAT: YES
[ggmlhexagon_print_running_timestamp, 2151]: using rpc ion memory pool:        YES
[ggmlhexagon_print_running_timestamp, 2152]: thread_counts with HWACCEL_CDSP:  8
[ggmlhexagon_print_running_timestamp, 2153]: mulmat algo type on cDSP:         0
[ggmlhexagon_probe_dspinfo, 5479]: dsp arch version 0x79
[ggmlhexagon_probe_dspinfo, 5490]: device info: Qualcomm SnapDragon 8 Elite, QCOM_HTP_V79
[ggmlhexagon_probe_dspinfo, 5520]: vtcm_count 1
[ggmlhexagon_probe_dspinfo, 5521]: vtcm_page 8388608
[ggmlhexagon_probe_dspinfo, 5522]: hmx_depth 0
[ggmlhexagon_probe_dspinfo, 5523]: hmx_spatial 0
[ggmlhexagon_probe_dspinfo, 5524]: hvx_support_128b 6
[ggmlhexagon_probe_dspinfo, 5525]: unsigned pd supported 1
[ggmlhexagon_probe_dspinfo, 5526]: async fastrpc supported 0
[ggmlhexagon_print_running_timestamp, 2159]: running timestamp:2025-07-07,10:33:16
[ggmlhexagon_deinit_cdsp, 5535]: enter ggmlhexagon_deinit_cdsp
[ggmlhexagon_deinit_cdsp, 5551]: leave ggmlhexagon_deinit_cdsp

background running logs:

Details

07-07 10:32:43.280 22633 22633 I ggml-hexagon: [ggmlhexagon_init_rpcmempool, 5429]: capacity of rpc memory 2000 MiB
07-07 10:32:44.931 22633 22633 I ggml-hexagon: [ggml_backend_hexagon_buffer_type_alloc_buffer, 6244]: size 938858496(895 MiB), rpc_mempool_usage 0(0 MiB), rpc_mempool_len 2088763392(1992 MiB)
07-07 10:32:45.072 22633 22633 I ggml-hexagon: [ggml_backend_hexagon_device_init_backend, 6488]: program specified param is not nullptr
07-07 10:32:45.072 22633 22633 I ggml-hexagon: [ggml_backend_hexagon_device_init_backend, 6500]: program specified dev_index 3
07-07 10:32:45.072 22633 22633 I ggml-hexagon: 
07-07 10:32:45.072 22633 22633 I ggml-hexagon: [ggml_backend_hexagon_buffer_type_alloc_buffer, 6244]: size 805306368(768 MiB), rpc_mempool_usage 938860544(895 MiB), rpc_mempool_len 2088763392(1992 MiB)
07-07 10:32:45.165 22633 22633 I ggml-hexagon: [ggml_backend_hexagon_buffer_type_alloc_buffer, 6244]: size 315359232(300 MiB), rpc_mempool_usage 1744166912(1663 MiB), rpc_mempool_len 2088763392(1992 MiB)
07-07 10:32:45.273 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 102675 microseconds
07-07 10:32:45.343 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 70247 microseconds
07-07 10:32:45.408 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 64532 microseconds
07-07 10:32:45.475 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 65493 microseconds
07-07 10:32:45.476 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 737 microseconds
07-07 10:32:45.639 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 163167 microseconds
07-07 10:32:45.801 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 161943 microseconds
07-07 10:32:45.961 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_5504x2048q4_0_5504x13f32 through CDSP: 159890 microseconds
07-07 10:32:45.962 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 330 microseconds
07-07 10:32:46.025 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63499 microseconds
07-07 10:32:46.089 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63173 microseconds
07-07 10:32:46.151 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62357 microseconds
07-07 10:32:46.214 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62774 microseconds
07-07 10:32:46.215 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 340 microseconds
07-07 10:32:46.376 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 161698 microseconds
07-07 10:32:46.540 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 163608 microseconds
07-07 10:32:46.705 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_5504x2048q4_0_5504x13f32 through CDSP: 164845 microseconds
07-07 10:32:46.706 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 364 microseconds
07-07 10:32:46.769 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63186 microseconds
07-07 10:32:46.832 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62878 microseconds
07-07 10:32:46.895 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62853 microseconds
07-07 10:32:46.960 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63285 microseconds
07-07 10:32:46.960 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 384 microseconds
07-07 10:32:47.128 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 167323 microseconds
07-07 10:32:47.294 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 165717 microseconds
07-07 10:32:47.451 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_5504x2048q4_0_5504x13f32 through CDSP: 155736 microseconds
07-07 10:32:47.452 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 340 microseconds
07-07 10:32:47.515 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63078 microseconds
07-07 10:32:47.578 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63416 microseconds
07-07 10:32:47.642 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62726 microseconds
07-07 10:32:47.705 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63134 microseconds
07-07 10:32:47.706 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 393 microseconds
07-07 10:32:47.868 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 161628 microseconds
07-07 10:32:48.033 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 164411 microseconds
07-07 10:32:48.200 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_5504x2048q4_0_5504x13f32 through CDSP: 166692 microseconds
07-07 10:32:48.201 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 783 microseconds
07-07 10:32:48.266 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63789 microseconds
07-07 10:32:48.329 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62859 microseconds
07-07 10:32:48.392 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62831 microseconds
07-07 10:32:48.456 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63001 microseconds
07-07 10:32:48.457 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 588 microseconds
07-07 10:32:48.622 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 164820 microseconds
07-07 10:32:48.788 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 165543 microseconds
07-07 10:32:48.952 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_5504x2048q4_0_5504x13f32 through CDSP: 163338 microseconds
07-07 10:32:48.953 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 429 microseconds
07-07 10:32:49.016 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63097 microseconds
07-07 10:32:49.080 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63218 microseconds
07-07 10:32:49.144 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63519 microseconds
07-07 10:32:49.207 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62849 microseconds
07-07 10:32:49.208 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 466 microseconds
07-07 10:32:49.374 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 165168 microseconds
07-07 10:32:49.540 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 166317 microseconds
07-07 10:32:49.708 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_5504x2048q4_0_5504x13f32 through CDSP: 166896 microseconds
07-07 10:32:49.709 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 343 microseconds
07-07 10:32:49.773 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63073 microseconds
07-07 10:32:49.837 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63680 microseconds
07-07 10:32:49.900 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62714 microseconds
07-07 10:32:49.963 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62638 microseconds
07-07 10:32:49.964 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 299 microseconds
07-07 10:32:50.132 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 168373 microseconds
07-07 10:32:50.298 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 165830 microseconds
07-07 10:32:50.467 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_5504x2048q4_0_5504x13f32 through CDSP: 167299 microseconds
07-07 10:32:50.467 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 544 microseconds
07-07 10:32:50.531 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63307 microseconds
07-07 10:32:50.595 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63611 microseconds
07-07 10:32:50.658 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62807 microseconds
07-07 10:32:50.734 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63202 microseconds
07-07 10:32:50.735 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 408 microseconds
07-07 10:32:50.898 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 162205 microseconds
07-07 10:32:51.064 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 165975 microseconds
07-07 10:32:51.229 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_5504x2048q4_0_5504x13f32 through CDSP: 164740 microseconds
07-07 10:32:51.230 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 858 microseconds
07-07 10:32:51.294 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63579 microseconds
07-07 10:32:51.357 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 61822 microseconds
07-07 10:32:51.420 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62867 microseconds
07-07 10:32:51.483 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62784 microseconds
07-07 10:32:51.484 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 379 microseconds
07-07 10:32:51.647 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 163023 microseconds
07-07 10:32:51.813 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 165876 microseconds
07-07 10:32:51.976 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_5504x2048q4_0_5504x13f32 through CDSP: 162867 microseconds
07-07 10:32:51.977 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 482 microseconds
07-07 10:32:52.040 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63172 microseconds
07-07 10:32:52.104 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63288 microseconds
07-07 10:32:52.167 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62580 microseconds
07-07 10:32:52.231 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63138 microseconds
07-07 10:32:52.231 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 465 microseconds
07-07 10:32:52.394 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 162221 microseconds
07-07 10:32:52.560 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 165824 microseconds
07-07 10:32:52.728 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_5504x2048q4_0_5504x13f32 through CDSP: 167355 microseconds
07-07 10:32:52.728 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 461 microseconds
07-07 10:32:52.792 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63234 microseconds
07-07 10:32:52.855 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62881 microseconds
07-07 10:32:52.918 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62743 microseconds
07-07 10:32:52.981 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62109 microseconds
07-07 10:32:52.982 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 426 microseconds
07-07 10:32:53.151 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 168829 microseconds
07-07 10:32:53.317 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 165717 microseconds
07-07 10:32:53.483 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_5504x2048q4_0_5504x13f32 through CDSP: 164403 microseconds
07-07 10:32:53.484 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 569 microseconds
07-07 10:32:53.547 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63324 microseconds
07-07 10:32:53.611 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63151 microseconds
07-07 10:32:53.673 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62614 microseconds
07-07 10:32:53.737 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62935 microseconds
07-07 10:32:53.737 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 320 microseconds
07-07 10:32:53.899 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 161571 microseconds
07-07 10:32:54.067 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 167873 microseconds
07-07 10:32:54.233 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_5504x2048q4_0_5504x13f32 through CDSP: 164739 microseconds
07-07 10:32:54.233 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 476 microseconds
07-07 10:32:54.297 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 64051 microseconds
07-07 10:32:54.360 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 61821 microseconds
07-07 10:32:54.423 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62685 microseconds
07-07 10:32:54.487 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63123 microseconds
07-07 10:32:54.487 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 451 microseconds
07-07 10:32:54.655 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 166645 microseconds
07-07 10:32:54.821 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 166138 microseconds
07-07 10:32:54.985 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_5504x2048q4_0_5504x13f32 through CDSP: 163319 microseconds
07-07 10:32:54.985 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 451 microseconds
07-07 10:32:55.049 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62948 microseconds
07-07 10:32:55.112 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62923 microseconds
07-07 10:32:55.175 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62721 microseconds
07-07 10:32:55.239 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62996 microseconds
07-07 10:32:55.240 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 382 microseconds
07-07 10:32:55.402 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 161535 microseconds
07-07 10:32:55.568 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 165969 microseconds
07-07 10:32:55.733 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_5504x2048q4_0_5504x13f32 through CDSP: 163694 microseconds
07-07 10:32:55.734 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 530 microseconds
07-07 10:32:55.797 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63044 microseconds
07-07 10:32:55.859 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 61586 microseconds
07-07 10:32:55.922 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62869 microseconds
07-07 10:32:55.986 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62972 microseconds
07-07 10:32:55.988 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 777 microseconds
07-07 10:32:56.154 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 165586 microseconds
07-07 10:32:56.323 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 169534 microseconds
07-07 10:32:56.490 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_5504x2048q4_0_5504x13f32 through CDSP: 166122 microseconds
07-07 10:32:56.491 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 701 microseconds
07-07 10:32:56.554 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62859 microseconds
07-07 10:32:56.618 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62895 microseconds
07-07 10:32:56.681 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62721 microseconds
07-07 10:32:56.745 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63430 microseconds
07-07 10:32:56.746 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 616 microseconds
07-07 10:32:56.909 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 163059 microseconds
07-07 10:32:57.075 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 166108 microseconds
07-07 10:32:57.242 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_5504x2048q4_0_5504x13f32 through CDSP: 165892 microseconds
07-07 10:32:57.243 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 850 microseconds
07-07 10:32:57.307 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63122 microseconds
07-07 10:32:57.368 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 60971 microseconds
07-07 10:32:57.431 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62598 microseconds
07-07 10:32:57.495 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63155 microseconds
07-07 10:32:57.496 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 340 microseconds
07-07 10:32:57.661 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 165116 microseconds
07-07 10:32:57.827 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 165644 microseconds
07-07 10:32:57.992 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_5504x2048q4_0_5504x13f32 through CDSP: 164261 microseconds
07-07 10:32:57.993 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 656 microseconds
07-07 10:32:58.057 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63133 microseconds
07-07 10:32:58.120 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63189 microseconds
07-07 10:32:58.183 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62741 microseconds
07-07 10:32:58.247 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63415 microseconds
07-07 10:32:58.248 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 434 microseconds
07-07 10:32:58.414 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 165628 microseconds
07-07 10:32:58.581 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 166254 microseconds
07-07 10:32:58.749 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_5504x2048q4_0_5504x13f32 through CDSP: 167509 microseconds
07-07 10:32:58.750 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 464 microseconds
07-07 10:32:58.813 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62751 microseconds
07-07 10:32:58.874 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 60855 microseconds
07-07 10:32:58.937 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62610 microseconds
07-07 10:32:59.000 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63193 microseconds
07-07 10:32:59.001 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 350 microseconds
07-07 10:32:59.166 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 164997 microseconds
07-07 10:32:59.330 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 164144 microseconds
07-07 10:32:59.498 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_5504x2048q4_0_5504x13f32 through CDSP: 165988 microseconds
07-07 10:32:59.499 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 667 microseconds
07-07 10:32:59.563 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62839 microseconds
07-07 10:32:59.627 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63215 microseconds
07-07 10:32:59.691 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62897 microseconds
07-07 10:32:59.755 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63560 microseconds
07-07 10:32:59.756 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 456 microseconds
07-07 10:32:59.917 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 160839 microseconds
07-07 10:33:00.083 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 166246 microseconds
07-07 10:33:00.255 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_5504x2048q4_0_5504x13f32 through CDSP: 170329 microseconds
07-07 10:33:00.256 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 823 microseconds
07-07 10:33:00.320 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62541 microseconds
07-07 10:33:00.384 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63328 microseconds
07-07 10:33:00.447 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62766 microseconds
07-07 10:33:00.511 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63418 microseconds
07-07 10:33:00.512 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 503 microseconds
07-07 10:33:00.677 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 164779 microseconds
07-07 10:33:00.844 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 166389 microseconds
07-07 10:33:01.011 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_5504x2048q4_0_5504x13f32 through CDSP: 165619 microseconds
07-07 10:33:01.012 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 480 microseconds
07-07 10:33:01.076 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62843 microseconds
07-07 10:33:01.140 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63243 microseconds
07-07 10:33:01.204 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62795 microseconds
07-07 10:33:01.268 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63215 microseconds
07-07 10:33:01.269 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 444 microseconds
07-07 10:33:01.432 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 162424 microseconds
07-07 10:33:01.598 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 165901 microseconds
07-07 10:33:01.760 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_5504x2048q4_0_5504x13f32 through CDSP: 161241 microseconds
07-07 10:33:01.761 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 1029 microseconds
07-07 10:33:01.825 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63047 microseconds
07-07 10:33:01.888 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63154 microseconds
07-07 10:33:01.952 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62938 microseconds
07-07 10:33:02.016 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63613 microseconds
07-07 10:33:02.017 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 447 microseconds
07-07 10:33:02.183 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 165398 microseconds
07-07 10:33:02.348 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x5504q4_0_2048x13f32 through CDSP: 164852 microseconds
07-07 10:33:02.516 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_5504x2048q4_0_5504x13f32 through CDSP: 167764 microseconds
07-07 10:33:02.517 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x13f32_2048x13f32 through CDSP: 586 microseconds
07-07 10:33:02.581 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62718 microseconds
07-07 10:33:02.644 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62968 microseconds
07-07 10:33:02.708 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 62806 microseconds
07-07 10:33:02.772 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of MUL_MATf32_2048x2048q4_0_2048x13f32 through CDSP: 63649 microseconds
07-07 10:33:02.773 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 428 microseconds
07-07 10:33:02.832 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 131 microseconds
07-07 10:33:02.833 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 100 microseconds
07-07 10:33:02.833 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 101 microseconds
07-07 10:33:02.836 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 98 microseconds
07-07 10:33:02.837 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 100 microseconds
07-07 10:33:02.837 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 91 microseconds
07-07 10:33:02.840 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 104 microseconds
07-07 10:33:02.841 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 91 microseconds
07-07 10:33:02.842 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 97 microseconds
07-07 10:33:02.844 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 98 microseconds
07-07 10:33:02.845 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 102 microseconds
07-07 10:33:02.845 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 93 microseconds
07-07 10:33:02.848 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 96 microseconds
07-07 10:33:02.848 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 100 microseconds
07-07 10:33:02.848 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 96 microseconds
07-07 10:33:02.850 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 90 microseconds
07-07 10:33:02.851 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 93 microseconds
07-07 10:33:02.851 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 104 microseconds
07-07 10:33:02.853 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 119 microseconds
07-07 10:33:02.854 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 90 microseconds
07-07 10:33:02.854 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 91 microseconds
07-07 10:33:02.856 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 88 microseconds
07-07 10:33:02.857 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 95 microseconds
07-07 10:33:02.857 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 86 microseconds
07-07 10:33:02.859 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 96 microseconds
07-07 10:33:02.859 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 88 microseconds
07-07 10:33:02.860 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 94 microseconds
07-07 10:33:02.862 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 90 microseconds
07-07 10:33:02.862 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 93 microseconds
07-07 10:33:02.863 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 83 microseconds
07-07 10:33:02.864 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 93 microseconds
07-07 10:33:02.865 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 85 microseconds
07-07 10:33:02.865 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 86 microseconds
07-07 10:33:02.867 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 85 microseconds
07-07 10:33:02.867 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 88 microseconds
07-07 10:33:02.868 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 81 microseconds
07-07 10:33:02.869 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 91 microseconds
07-07 10:33:02.870 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 81 microseconds
07-07 10:33:02.870 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 87 microseconds
07-07 10:33:02.872 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 86 microseconds
07-07 10:33:02.872 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 90 microseconds
07-07 10:33:02.872 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 81 microseconds
07-07 10:33:02.874 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 93 microseconds
07-07 10:33:02.874 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 80 microseconds
07-07 10:33:02.875 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 87 microseconds
07-07 10:33:02.877 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 85 microseconds
07-07 10:33:02.877 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 88 microseconds
07-07 10:33:02.877 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 81 microseconds
07-07 10:33:02.879 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 91 microseconds
07-07 10:33:02.879 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 80 microseconds
07-07 10:33:02.880 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 101 microseconds
07-07 10:33:02.881 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 84 microseconds
07-07 10:33:02.882 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 84 microseconds
07-07 10:33:02.882 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 77 microseconds
07-07 10:33:02.883 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 90 microseconds
07-07 10:33:02.884 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 79 microseconds
07-07 10:33:02.884 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 84 microseconds
07-07 10:33:02.886 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 84 microseconds
07-07 10:33:02.886 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 89 microseconds
07-07 10:33:02.886 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 80 microseconds
07-07 10:33:02.888 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 87 microseconds
07-07 10:33:02.888 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 81 microseconds
07-07 10:33:02.888 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 85 microseconds
07-07 10:33:02.890 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 82 microseconds
07-07 10:33:02.890 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 87 microseconds
07-07 10:33:02.891 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 83 microseconds
07-07 10:33:02.892 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 90 microseconds
07-07 10:33:02.893 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 79 microseconds
07-07 10:33:02.893 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 84 microseconds
07-07 10:33:02.894 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 82 microseconds
07-07 10:33:02.895 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 93 microseconds
07-07 10:33:02.895 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 78 microseconds
07-07 10:33:02.911 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 96 microseconds
07-07 10:33:02.912 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 76 microseconds
07-07 10:33:02.912 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 85 microseconds
07-07 10:33:02.913 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 79 microseconds
07-07 10:33:02.914 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 83 microseconds
07-07 10:33:02.914 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 76 microseconds
07-07 10:33:02.915 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 88 microseconds
07-07 10:33:02.915 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 76 microseconds
07-07 10:33:02.916 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 84 microseconds
07-07 10:33:02.917 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 79 microseconds
07-07 10:33:02.917 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 83 microseconds
07-07 10:33:02.918 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 78 microseconds
07-07 10:33:02.919 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 84 microseconds
07-07 10:33:02.919 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 75 microseconds
07-07 10:33:02.920 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 85 microseconds
07-07 10:33:02.921 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 79 microseconds
07-07 10:33:02.921 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 83 microseconds
07-07 10:33:02.922 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 76 microseconds
07-07 10:33:02.923 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 88 microseconds
07-07 10:33:02.923 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 78 microseconds
07-07 10:33:02.923 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 82 microseconds
07-07 10:33:02.925 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 83 microseconds
07-07 10:33:02.925 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 83 microseconds
07-07 10:33:02.925 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 76 microseconds
07-07 10:33:02.927 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 84 microseconds
07-07 10:33:02.927 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 77 microseconds
07-07 10:33:02.927 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 84 microseconds
07-07 10:33:02.929 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 80 microseconds
07-07 10:33:02.929 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 84 microseconds
07-07 10:33:02.929 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 74 microseconds
07-07 10:33:02.931 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 110 microseconds
07-07 10:33:02.931 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 76 microseconds
07-07 10:33:02.931 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 80 microseconds
07-07 10:33:02.933 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 78 microseconds
07-07 10:33:02.933 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 83 microseconds
07-07 10:33:02.933 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 76 microseconds
07-07 10:33:02.935 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 84 microseconds
07-07 10:33:02.935 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 76 microseconds
07-07 10:33:02.935 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 82 microseconds
07-07 10:33:02.937 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 78 microseconds
07-07 10:33:02.937 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 81 microseconds
07-07 10:33:02.937 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 75 microseconds
07-07 10:33:02.939 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 86 microseconds
07-07 10:33:02.939 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 98 microseconds
07-07 10:33:02.939 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 81 microseconds
07-07 10:33:02.940 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 79 microseconds
07-07 10:33:02.941 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 82 microseconds
07-07 10:33:02.941 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 77 microseconds
07-07 10:33:02.942 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 86 microseconds
07-07 10:33:02.943 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 75 microseconds
07-07 10:33:02.943 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 83 microseconds
07-07 10:33:02.944 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 79 microseconds
07-07 10:33:02.945 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 79 microseconds
07-07 10:33:02.945 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 76 microseconds
07-07 10:33:02.946 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 84 microseconds
07-07 10:33:02.946 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 76 microseconds
07-07 10:33:02.947 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 83 microseconds
07-07 10:33:02.948 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 79 microseconds
07-07 10:33:02.948 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 81 microseconds
07-07 10:33:02.949 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 77 microseconds
07-07 10:33:02.950 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 85 microseconds
07-07 10:33:02.950 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 74 microseconds
07-07 10:33:02.951 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 81 microseconds
07-07 10:33:02.952 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 79 microseconds
07-07 10:33:02.952 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 82 microseconds
07-07 10:33:02.953 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 76 microseconds
07-07 10:33:02.954 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 84 microseconds
07-07 10:33:02.954 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 75 microseconds
07-07 10:33:02.954 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 81 microseconds
07-07 10:33:02.956 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 76 microseconds
07-07 10:33:02.956 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 81 microseconds
07-07 10:33:02.956 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 73 microseconds
07-07 10:33:02.973 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 93 microseconds
07-07 10:33:02.973 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 77 microseconds
07-07 10:33:02.973 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 81 microseconds
07-07 10:33:02.975 22633 22633 I ggml-hexagon: [info, 1310]: inference duration of ADDf32_2048x1f32_2048x1f32 through CDSP: 80 microseconds

Conclusion

After compare ggml-opencl, ggml-vulkan, ggml-hexagon, the default ggml backend on Snapdragon 8Elite based Android phone, I personally think:

the maturity of these 4 backends on Snapdraon based Android phone as following:

the default ggml backend(ARM NEON backend) >= ggml-opencl >>> ggml-hexagon(aka Hexagon NPU backend) > ggml-vulkan

the maturity of Qualcomm's GPU and NPU driver as following:

OpenCL-based GPU driver > FastRPC-based NPU driver > Vulkan-based GPU driver

the maturity of Qualcomm's GPU and NPU drivers will be as following and ggml-hexagon will be an ideal backend if Qualcomm can improve their FastRPC-based NPU driver.

NPU driver > OpenCL-based GPU driver > Vulkan-based GPU driver

Qualcomm's Hexagon SDK team can remove the qidl mechanism in the new Hexagon SDK and provide a simple and effective approach/mechanism to exchange arbitrary data between ARM AP side and cDSP side, let developers do what they want to do on the world-class and state-of-the-art Hexagon NPU(aka Hexagon-cDSP) rather than study the confusing codes which generated by qidl mechanisim, this move will be helpful for their high-level QNN SDK/SNPE SDK and helpful for llama.cpp community. Obviously, it would be a great move if Qualcomm could open their HMX instructions and the corresponding tech docs.

At the same time, compare to ggml-vulkan on Snapdragon based Android phone,

We can clearly see that ggml-hexagon works good although the inference performance of quantized mulmat is really bad at the moment.
AI experts can clearly see that ggml-hexagon support UMA feature because Vulkan-based GPU driver and FastRPC-based NPU driver are both comes from Qualcomm.

Analysis

The default ggml backend is maintained by tech experts around the world whom have no KPI(do it for interest and passion) and experts from ARM,Qualcomm,Google,Nvidia,AMD,Intel,Loongson...
Qualcomm privides OpenCL-based GPU driver for ggml-opencl, Vulkan-based GPU driver for ggml-vulkan, NPU-driver for ggml-hexagon. Accordingly, Qualcomm provides Hexagon SDK for ggml-hexagon(or other name), QNN SDK for QNN-based End-to-End solution(e.g.: Qualcomm's official AI-Hub), SNPE SDK for other scenarios.
The ggml-vulkan backend supports many many AI operators and works fine/well on x86-Linux or arm-Linux or Windows but didn't works fine as expected on Snapdragon based Android phone. As the original author(I'm not sure whether is from Nvidia or AMD) of ggml-vulkan said on 10/25/2024: "Sadly, in my experience Qualcomm is not handling Vulkan in a proper way, like AMD, Nvidia and Intel are, so there are a lot of quirks to deal with."
The candidate ggml-hexagon backend only supports GGML_OP_ADD(fp32) and GGML_OP_MUL_MAT(fp32 + quantized) but the performance and stability of these two operators are positive(especially fp32 ADD and fp32 MUL_MAT) and good. Compare to ggml-vulkan on Android phone, it seems it's a feasible and correct path:
- This backend also supports UMA feature although it would be better that the existing FastRPC mechanism can be improved.
- AI experts have great flexibility on cDSP side, especially compared to the QNN API on ARM AP side.
- AI experts can help to improve the performance and add other AI operators on cDSP side accordingly(which is exactly similar to CUDA kernels or OpenCL kernels).

0 replies

zhouwg · 2025-07-12T09:34:57Z

zhouwg
Jul 12, 2025
Maintainer Author

[Time:Jul 12 2025] From the original author of ggml:

@zhouwg I am really sorry, but I am once again considering blocking you from the projects. You are posting an enormous amount of comments, repeating the same stuff over and over again, tagging maintainers and accusing/insinuating - I have stopped reading your comments a long time ago. But this is just generating a lot of noise and it becomes annoying.

I don't know really what is the problem that you have - probably some language or communication barrier. I can't really tell. I think we try to accept all points of views and believe we have done a good job overall. Blocking accounts is the last thing I want to do, but IMO this is borderline spamming.

I will give this a bit more thought in the next few days and go through internal discussion with the members before making the decision to block you. In the meantime: - please stop posting here anything other than specific technical comments like everybody else. No need to praise every single sentence by every single contributor each time, or discuss their educational or professional background, or any of that stuff above. If you don't then I'll just block you again immediately.

0 replies

zhouwg · 2025-07-12T09:35:14Z

zhouwg
Jul 12, 2025
Maintainer Author

[Time:Jul 12 2025, updated on Jul 13 2025]
Thanks for your reminder. My points are:

PR-12063 is a hard-forked PR of my initial PR and PR-12063 was opened on 02/25/2025 without any breakthrough progress and I think this behaviour breaks the rule based order, their refactoring work of my initial PR has been going on for about a year since 07/2024 and still a draft PR. At the same time, PR-12326 is ready for review as an initial version because PR-12326 is already a functional and practical PR and I personally think it's more mature than PR-8273 or the initial ggml-vulkan, I'm not sure whether this is fair to other developers in this tech community?
The beautiful and grand picture which described in the PR-12063 on 02/25/2025 has been verified a fake news: the so-called "standout" feature(mapping ggml cgraph to a single QNN graph) in PR-12063 is fake. PR-12063's new approach which claimed on 05/27/2025 is exactly equivalent to PR-12326 which already implemented before 04/24/2025.
After 05/27/2025, domain tech experts all can see that PR-12063 and PR-12326 are exactly equivalent and the implementation is incompatible. in other words, there is a long-term conflict between PR-12063 and PR-12326. I understand why you can't really tell because there is culture barrier.
PR-12326's benchmark data can be reproduced by any third-party, PR-12063 didn't provide the detailed steps to help other developers to check/reproduce their beautiful benchmark data which they published in Performance of llama.cpp on Snapdragon X Elite/Plus ggml-org/llama.cpp#8273. I'd like to check and reproduce PR-12063's beautiful benchmark data which they published in this tech community on my self-purchased 8Gen3 and 8Elite phone. I personally think as a pure tech community we should have zero tolerance for fake news and fake data because this place is not mainland China.
Finally, I really don't understand: as one of the core maintainers and a modern C++ master and the original author of the ggml backend subsystem, slaren can clearly clarified that he is not the original author of ggml-vulkan, why it's so difficult for the author of that competing PR([WIP]backend: Integrating QNN (Qualcomm AI Engine Direct) as a dedicated backend for Qualcomm NPUs ggml-org/llama.cpp#12063)? As I said many times: I'll close the PR-12326 immediately and waiting for the official PR of Hexagon-NPU from Qualcomm if the author of PR-12063 is really comes from Qualcomm China Shenzhen branch.
I'll so sorry that I was blocked again because I like this tech community which is outside of mainland China.

0 replies

Navigation Menu

semi-fully version of discussions-14356(https://github.com/ggml-org/llama.cpp/discussions/14356)(backuped on Jul 12 2025) #64

Uh oh!

{{title}}

Uh oh!

Replies: 4 comments

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Select a reply

Uh oh!

semi-fully version of discussions-14356(https://github.com/ggml-org/llama.cpp/discussions/14356)(backuped on Jul 12 2025) #64

Uh oh!

zhouwg Jul 12, 2025 Maintainer

Introduction

Table of Contents

Part 1: Research Papers

on-device inference

core technologies

fundamental technologies

Part 2: On-Device Inference framework

Part 3: Hardware acceleration

OpenCL

Vulkan

Qualcomm Hexagon NPU

QNN solution

llama.cpp solution

Part 4: LLamacpp based Android APPs

Part 5: State-of-the-art on-device LLM models

Part 6: Mobile SoC References

Part 7: Acknowledgement

Disclaimer

Replies: 4 comments

Uh oh!

Uh oh!

zhouwg Jul 12, 2025 Maintainer Author

Benchmark data of project ggml-hexagon

How to verify on Android phone

Overview

Steps

What does 0/1/2/3/4 mean in the script?

HVX and HMX

I'm confused about QNN-NPU and Hexagon-NPU, Are they different implementations of the same hardware?

Technical merits in PR-12326:

Updated benchmark data of fp32 mulmat on Hexagon cDSP(actually updated on 06/27/2025, merged in this comment on Jun 8 2025)

Uh oh!

Uh oh!

zhouwg Jul 12, 2025 Maintainer Author

Overall evaluation of ggml-opencl,ggml-vulkan,ggml-hexagon and the default ggml backend on Android phone

Introduction

How to build ggml-opencl backend for Android device on Linux

How to build ggml-vulkan backend for Android device on Linux

How to build ggml-hexagon backend for Android device on Linux

Benchmark data

Conclusion

Analysis

Uh oh!

Uh oh!

zhouwg Jul 12, 2025 Maintainer Author

Uh oh!

Uh oh!

zhouwg Jul 12, 2025 Maintainer Author

zhouwg
Jul 12, 2025
Maintainer

zhouwg
Jul 12, 2025
Maintainer Author

zhouwg
Jul 12, 2025
Maintainer Author

zhouwg
Jul 12, 2025
Maintainer Author

zhouwg
Jul 12, 2025
Maintainer Author