Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -8,31 +8,31 @@ layout: learningpathall

## Set up your development environment

In this Learning Path, you will learn how to build and deploy a simple LLM-based chat app to an Android device using ONNX Runtime. You will learn how to build the ONNX runtime and ONNX Runtime generate() API and how to run the Phi-3 model for the Android application.
In this learning path, you will learn how to build and deploy a simple LLM-based chat app to an Android device using ONNX Runtime. You will learn how to build the ONNX Runtime and ONNX Runtime generate() API and how to run the Phi-3 model for the Android application.

The first step is to prepare a development environment with the required software:
Your first task is to prepare a development environment with the required software:

- Android Studio (latest version recommended)
- Android NDK (tested with version 27.0.12077973)
- Python 3.11
- CMake (tested with version 3.28.1)
- Ninja (tested with version 1.11.1)

The instructions were tested on an x86 Windows machine with at least 16GB of RAM.
The following instructions were tested on an x86 Windows machine with at least 16GB of RAM.

## Install Android Studio and Android NDK

Follow these steps to install and configure Android Studio:

1. Download and install the latest version of [Android Studio](https://developer.android.com/studio/).

2. Navigate to `Tools -> SDK Manager`.
2. Navigate to **Tools > SDK Manager**.

3. In the `SDK Platforms` tab, check `Android 14.0 ("UpsideDownCake")`.
3. In the **SDK Platforms** tab, check **Android 14.0 ("UpsideDownCake")**.

4. In the `SDK Tools` tab, check `NDK (Side by side)`.
4. In the **SDK Tools** tab, check **NDK (Side by side)**.

5. Click Ok and Apply.
5. Click **OK** and **Apply**.

## Install Python 3.11

Expand All @@ -50,9 +50,7 @@ The instructions were tested with version 3.28.1

## Install Ninja

Ninja is a minimalistic build system designed to efficiently handle incremental builds, particularly in large-scale software projects, by focusing on speed and simplicity.

The Ninja generator needs to be used to build on Windows for Android.
Ninja is a minimalistic build system designed to efficiently handle incremental builds, particularly in large-scale software projects, by focusing on speed and simplicity. The Ninja generator is used to build on Windows for Android.

[Download and install Ninja]( https://github.com/ninja-build/ninja/releases)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ You might be able to use a later commit. These steps have been tested with the c

### Build for Android CPU

The Ninja generator needs to be used to build on Windows. First, set JAVA_HOME to the path to your JDK install. You can point to the JDK from Android Studio, or a standalone JDK install.
You use the Ninja generator to build on Windows for Android. First, set JAVA_HOME to the path to your JDK install. You can point to the JDK from Android Studio, or a standalone JDK install.

```bash
$env:JAVA_HOME="C:\Program Files\Android\Android Studio\jbr"
Expand All @@ -44,7 +44,7 @@ Now run the following command:

```

Android Archive (AAR) files, which can be imported directly in Android Studio, will be generated by using the above command with `--build_java`
An Android Archive (AAR) file, which can be imported directly in Android Studio, will be generated by using the above command with `--build_java`

When the build is complete, confirm the shared library and the AAR file have been created:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,15 @@ weight: 4
layout: learningpathall
---

## Cross-compile the ONNX Runtime generate() API for Android CPU
## Cross-compile the ONNX Runtime Generate() API for Android CPU

The Generate() API in ONNX Runtime is designed for text generation tasks using models like Phi-3. It implements the generative AI loop for ONNX models, including pre and post processing, inference with ONNX Runtime, logits processing, search and sampling, and KV cache management. You can learn more by reading the [ONNX Runtime generate() API page](https://onnxruntime.ai/docs/genai/).
The Generate() API in ONNX Runtime is designed for text generation tasks using models like Phi-3. It implements the generative AI loop for ONNX models, including:
- pre- and post-processing
- inference with ONNX Runtime
- logits processing
- search and sampling
- KV cache management.
You can learn more by reading the [ONNX Runtime generate() API page](https://onnxruntime.ai/docs/genai/).


### Clone onnxruntime-genai repo
Expand All @@ -27,7 +33,7 @@ You might be able to use later commits. These steps have been tested with the co

### Build for Android CPU

The Ninja generator needs to be used to build on Windows for Android. Make sure JAVA_HOME is set before running the following command:
Ninja generator is used to build on Windows for Android. Make sure you have set JAVA_HOME before running the following command:

```bash
python -m pip install requests
Expand Down
Original file line number Diff line number Diff line change
@@ -1,17 +1,18 @@
---
title: Run Benchmark on Android phone
title: Run a benchmark on an Android phone
weight: 5

### FIXED, DO NOT MODIFY
layout: learningpathall
---

## Run example code for running Phi-3-mini
## Run a Phi-3 model on your Android phone

You can now prepare and run a Phi-3-mini model on your Android smartphone, and view performance metrics:

### Build model runner

You will now cross-compile the model runner to run on Android using the commands below:
First, cross-compile the model runner to run on Android using the commands below:

``` bash
cd onnxruntime-genai
Expand All @@ -21,7 +22,7 @@ cd examples\c
mkdir build
cd build
```
Run the cmake command as shown:
Run the `cmake` command as shown:

```bash
cmake -DCMAKE_TOOLCHAIN_FILE=C:\Users\$env:USERNAME\AppData\Local\Android\Sdk\ndk\27.0.12077973\build\cmake\android.toolchain.cmake -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-27 -DCMAKE_BUILD_TYPE=Release -G "Ninja" ..
Expand All @@ -30,25 +31,25 @@ ninja

After successful build, a binary program called `phi3` will be created.

### Prepare phi-3-mini model
### Prepare Phi-3-mini model

Phi-3 ONNX models are hosted on HuggingFace. You can download the Phi-3-mini model with huggingface-cli command:
Phi-3 ONNX models are hosted on HuggingFace. You can download the Phi-3-mini model by using the `huggingface-cli` command:

``` bash
pip install huggingface-hub[cli]
huggingface-cli download microsoft/Phi-3-mini-4k-instruct-onnx --include cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4/* --local-dir .
```
This command downloads the model into a folder called cpu_and_mobile.
This command downloads the model into a folder called `cpu_and_mobile`.

The phi-3-mini (3B) model has a short (4k) context version and a long (128k) context version. The long context version can accept much longer prompts and produce longer output text, but it does consume more memory. In this learning path, you will use the short context version, which is quantized to 4-bits.
The Phi-3-mini (3B) model has a short (4k) context version and a long (128k) context version. The long context version can accept much longer prompts and produce longer output text, but it does consume more memory. In this learning path, you will use the short context version, which is quantized to 4-bits.


### Run on Android via adb shell

#### Connect your android phone
#### Connect your Android phone
Connect your phone to your computer using a USB cable.

You need to enable USB debugging on your Android device. You can follow [Configure on-device developer options](https://developer.android.com/studio/debug/dev-options) to enable USB debugging.
You need to enable USB debugging on your Android device. You can follow [Configure on-device developer options](https://developer.android.com/studio/debug/dev-options) to do this.

Once you have enabled USB debugging and connected via USB, run:

Expand Down Expand Up @@ -79,7 +80,7 @@ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/data/local/tmp
./phi3 cpu-int4-rtn-block-32-acc-level-4
```

This will allow the runner program to load the model, and then it will prompt you to input the text prompt to the model. After you enter yout input prompt, the text output by the model will be displayed. On completion, the performance metrics similar to what is shown below should be displayed:
This will allow the runner program to load the model. It will then prompt you to input the text prompt to the model. After you enter your input prompt, the text output by the model will be displayed. On completion, performance metrics similar to those shown below should be displayed:

```
Prompt length: 64, New tokens: 931, Time to first: 1.79s, Prompt tokens per second: 35.74 tps, New tokens per second: 6.34 tps
Expand Down
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
---
title: Build and Run Android chat app
title: Build and run an Android chat app
weight: 6

### FIXED, DO NOT MODIFY
layout: learningpathall
---

## Build Android chat app
## Build an Android chat app

Another way to run the model is to use an Android GUI app.
You can use the Android demo application included in the [onnxruntime-inference-examples repository](https://github.com/microsoft/onnxruntime-inference-examples) to demonstrate local inference.
Expand All @@ -27,9 +27,9 @@ You could probably use a later commit but these steps have been tested with the

Open the `mobile\examples\phi-3\android` directory with Android Studio.

#### (Optional) In case you want to use ONNX Runtime AAR you built
#### (Optional) In case you want to use the ONNX Runtime AAR you built

Copy ONNX Runtime AAR you built before if needed:
Copy ONNX Runtime AAR you built earlier in this learning path:

```bash
Copy onnxruntime\build\Windows\Release\java\build\android\outputs\aar\onnxruntime-release.aar mobile\examples\phi-3\android\app\libs
Expand All @@ -43,12 +43,12 @@ Update `build.gradle.kts (:app)` as below:
implementation(files("libs/onnxruntime-release.aar"))
```

After that, click `File`->`Sync Project with Gradle`
Finally, click **File > Sync Project with Gradle**

#### Build and run the app

When you press Run, the build will be executed, and then the app will be copied and installed on the Android device. This app will automatically download the Phi-3-mini model during the first run. After the download, you can input the prompt in the text box and execute it to run the model.
When you select **Run**, the build will be executed, and then the app will be copied and installed on the Android device. This app will automatically download the Phi-3-mini model during the first run. After the download, you can input the prompt in the text box and execute it to run the model.

You should now see a running app on your phone that looks like this:
You should now see a running app on your phone, which looks like this:

![App screenshot](screenshot.png)
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,10 @@ who_is_this_for: This is an advanced topic for software developers interested in

learning_objectives:
- Build ONNX Runtime and ONNX Runtime generate() API for Android.
- Run the Phi-3 model using ONNX Runtime on an Arm-based smartphone.
- Run a Phi-3 model using ONNX Runtime on an Arm-based smartphone.

prerequisites:
- A Windows x86_64 development machine with at least 16GB of RAM. You should also be able to use Linux or MacOS for the build, but the instructions for it have not been included in this learning path.
- A Windows x86_64 development machine with at least 16GB of RAM.
- An Android phone with at least 8GB of RAM. This learning path was tested on Samsung Galaxy S24.

author_primary: Koki Mitsunami
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ review:
- A cloud-based data storage service for deep learning models.
correct_answer: 1
explanation: >
ONNX Runtime is a cross-platform inference engine designed to to run machine-learning models in the ONNX format. It optimizes model performance across various hardware environments, including CPUs, GPUs, and specialized accelerators.
ONNX Runtime is a cross-platform inference engine designed to to run machine learning models in the ONNX format. It optimizes model performance across various hardware environments, including CPUs, GPUs and specialized accelerators.

- questions:
question: >
Expand All @@ -20,7 +20,7 @@ review:
- A toolkit for converting machine learning models to ONNX format.
correct_answer: 2
explanation: >
Phi models are a series of large language models developed to perform natural language processing tasks such as text generation, completion, and comprehension.
Phi models are a series of Large Language Models developed to perform natural language processing tasks such as text generation, completion and comprehension.

- questions:
question: >
Expand All @@ -31,7 +31,7 @@ review:
- It allows models to be exchanged between different frameworks, such as PyTorch and TensorFlow.
correct_answer: 3
explanation: >
The ONNX (Open Neural Network Exchange) format is an open-source standard designed to enable the sharing and use of machine learning models across different frameworks such as PyTorch, TensorFlow, and others. It allows models to be exported in a unified format, making them interoperable and ensuring they can run on various platforms or hardware.
The ONNX (Open Neural Network Exchange) format is an open-source standard designed to enable the sharing and use of machine learning models across different frameworks such as PyTorch and TensorFlow. It allows models to be exported in a unified format, making them interoperable and ensuring they can run on various platforms or hardware.



Expand Down