From 074642188435831fbf6da60207008d8f9d98482d Mon Sep 17 00:00:00 2001 From: Shuheng Deng Date: Fri, 10 Oct 2025 15:48:23 +0800 Subject: [PATCH] Update the vision llm page and add content of UI demo --- .../2-generate-apk.md | 53 +++++++++++++++++++ .../{2-benchmark.md => 3-benchmark.md} | 26 +++++---- 2 files changed, 68 insertions(+), 11 deletions(-) create mode 100644 content/learning-paths/mobile-graphics-and-gaming/vision-llm-inference-on-android-with-kleidiai-and-mnn/2-generate-apk.md rename content/learning-paths/mobile-graphics-and-gaming/vision-llm-inference-on-android-with-kleidiai-and-mnn/{2-benchmark.md => 3-benchmark.md} (85%) diff --git a/content/learning-paths/mobile-graphics-and-gaming/vision-llm-inference-on-android-with-kleidiai-and-mnn/2-generate-apk.md b/content/learning-paths/mobile-graphics-and-gaming/vision-llm-inference-on-android-with-kleidiai-and-mnn/2-generate-apk.md new file mode 100644 index 0000000000..6e4de6e791 --- /dev/null +++ b/content/learning-paths/mobile-graphics-and-gaming/vision-llm-inference-on-android-with-kleidiai-and-mnn/2-generate-apk.md @@ -0,0 +1,53 @@ +--- +title: Benchmark the Vision Transformer performance with KleidiAI +weight: 4 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Clone Vision Language Models repo + +In this section, you will run the Qwen model in action using a demo application using a Android Package Kit (APK). + +This repository is set up to enable building the app as an Android Studio project. + +Run the following commands to clone the repository and checkout the source tree: + +```bash +git clone https://gitlab.arm.com/kleidi/kleidi-examples/vision-language-models +``` + +## Build the App Using Android Studio + +You can use Android Studio to build the app and create an APK. + +### Open project and build + +Open Android Studio. + +Go to **File > Open**. + +Navigate to the vision-language-models directories, and click `Open`. + +This triggers a build of the project, and you should see output similar to the following on completion: + +```output +BUILD SUCCESSFUL in 1m 42s +``` + +### Generate and Run the APK + +Navigate to **Build > Generate App Bundles or APKs**. Select **Generate APKs**. + +The build will be executed, and then the app will be copied and installed on the Android device. + +After opening the app, you will see the splash screen: + +![Loading screenshot](Loading_page.png) + +Finally, you can use the UI to chat with the app. Try uploading an image and ask a question on it. + +![Loading screenshot](chat2.png) + +The final step is to examine how KleidiAI can improve the performance of the model. Continue to the next section to find out. \ No newline at end of file diff --git a/content/learning-paths/mobile-graphics-and-gaming/vision-llm-inference-on-android-with-kleidiai-and-mnn/2-benchmark.md b/content/learning-paths/mobile-graphics-and-gaming/vision-llm-inference-on-android-with-kleidiai-and-mnn/3-benchmark.md similarity index 85% rename from content/learning-paths/mobile-graphics-and-gaming/vision-llm-inference-on-android-with-kleidiai-and-mnn/2-benchmark.md rename to content/learning-paths/mobile-graphics-and-gaming/vision-llm-inference-on-android-with-kleidiai-and-mnn/3-benchmark.md index 863eb1a49c..3849c44ed1 100644 --- a/content/learning-paths/mobile-graphics-and-gaming/vision-llm-inference-on-android-with-kleidiai-and-mnn/2-benchmark.md +++ b/content/learning-paths/mobile-graphics-and-gaming/vision-llm-inference-on-android-with-kleidiai-and-mnn/3-benchmark.md @@ -1,6 +1,6 @@ --- title: Build the MNN Command-line ViT Demo -weight: 4 +weight: 5 ### FIXED, DO NOT MODIFY layout: learningpathall @@ -29,7 +29,7 @@ Run the following commands to clone the MNN repository and checkout the source t cd $HOME git clone https://github.com/alibaba/MNN.git cd MNN -git checkout a739ea5870a4a45680f0e36ba9662ca39f2f4eec +git checkout fa3b2161a9b38ac1e7dc46bb20259bd5eb240031 ``` Create a build directory and run the build script. @@ -40,10 +40,9 @@ The first time that you do this, build the binaries with the `-DMNN_KLEIDIAI` fl cd $HOME/MNN/project/android mkdir build_64 && cd build_64 -../build_64.sh "-DMNN_LOW_MEMORY=true -DLLM_SUPPORT_VISION=true -DMNN_KLEIDIAI=FALSE \ - -DMNN_CPU_WEIGHT_DEQUANT_GEMM=true -DMNN_BUILD_LLM=true \ - -DMNN_SUPPORT_TRANSFORMER_FUSE=true -DMNN_ARM82=true -DMNN_OPENCL=true \ - -DMNN_USE_LOGCAT=true -DMNN_IMGCODECS=true -DMNN_BUILD_OPENCV=true" +../build_64.sh "-DMNN_BUILD_LLM=true -DMNN_BUILD_LLM_OMNI=ON -DLLM_SUPPORT_VISION=true \ +-DMNN_BUILD_OPENCV=true -DMNN_IMGCODECS=true -DMNN_LOW_MEMORY=true \ +-DMNN_CPU_WEIGHT_DEQUANT_GEMM=true -DMNN_BUILD_LLM=true -DMNN_SUPPORT_TRANSFORMER_FUSE=true" ``` {{% notice Note %}} If your NDK toolchain isn't set up correctly, you might run into issues with the above script. Make a note of where the NDK was installed - this will be a directory named after the version you downloaded earlier. Try exporting the following environment variables before re-running `build_64.sh`: @@ -102,14 +101,19 @@ prefill speed = 192.28 tok/s ## Enable KleidiAI and Re-run Inference -The next step is to re-generate the binaries with KleidiAI activated. This is done by updating the flag `-DMNN_KLEIDIAI` to `TRUE`. +The next step is to re-generate the binaries with KleidiAI activated. This is done by inserting a hint into the code. + +From the `MNN` directory, run: +```bash +sed -i '/void Llm::setRuntimeHint(std::shared_ptr &rtg) {/a\ + rtg->setHint(MNN::Interpreter::CPU_ENABLE_KLEIDIAI, 1);' transformers/llm/engine/src/llm.cpp +``` From the `build_64` directory, run: ```bash -../build_64.sh "-DMNN_LOW_MEMORY=true -DLLM_SUPPORT_VISION=true -DMNN_KLEIDIAI=TRUE \ --DMNN_CPU_WEIGHT_DEQUANT_GEMM=true -DMNN_BUILD_LLM=true \ --DMNN_SUPPORT_TRANSFORMER_FUSE=true -DMNN_ARM82=true -DMNN_OPENCL=true \ --DMNN_USE_LOGCAT=true -DMNN_IMGCODECS=true -DMNN_BUILD_OPENCV=true" +../build_64.sh "-DMNN_BUILD_LLM=true -DMNN_BUILD_LLM_OMNI=ON -DLLM_SUPPORT_VISION=true \ +-DMNN_BUILD_OPENCV=true -DMNN_IMGCODECS=true -DMNN_LOW_MEMORY=true \ +-DMNN_CPU_WEIGHT_DEQUANT_GEMM=true -DMNN_BUILD_LLM=true -DMNN_SUPPORT_TRANSFORMER_FUSE=true" ``` ## Update Files on the Device