Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ See the [CMake install guide](/install-guides/cmake/) for troubleshooting instru

### Install Bazel

Bazel is an open-source build tool which we will use to build LiteRT libraries.
Bazel is an open-source build tool which you will use to build LiteRT libraries.

{{< tabpane code=true >}}
{{< tab header="Linux">}}
Expand All @@ -98,22 +98,24 @@ wget https://dl.google.com/android/repository/android-ndk-r25b-linux.zip
unzip android-ndk-r25b-linux.zip
{{< /tab >}}
{{< tab header="MacOS">}}
brew install --cask android-studio temurin
wget https://dl.google.com/android/repository/android-ndk-r25b-darwin.zip
unzip android-ndk-r25b-darwin
mv android-ndk-r25b-darwin ~/Library/Android/android-ndk-r25b
{{< /tab >}}
{{< /tabpane >}}

For easier access and execution of Android NDK tools, add these to the `PATH` and set the `ANDROID_NDK` variable:
For easier access and execution of Android NDK tools, add these to the `PATH` and set the `NDK_PATH` variable:

{{< tabpane code=true >}}
{{< tab header="Linux">}}
export ANDROID_NDK=$WORKSPACE/android-ndk-r25b/
export PATH=$ANDROID_NDK/toolchains/llvm/prebuilt/linux-x86_64/bin/:$PATH
export NDK_PATH=$WORKSPACE/android-ndk-r25b/
export PATH=$NDK_PATH/toolchains/llvm/prebuilt/linux-x86_64/bin/:$PATH
{{< /tab >}}
{{< tab header="MacOS">}}
export ANDROID_NDK=~/Library/Android/sdk/ndk/27.0.12077973/
export PATH=$PATH:$ANDROID_NDK/toolchains/llvm/prebuilt/darwin-x86_64/bin
export NDK_PATH=~/Library/Android/android-ndk-r25b
export PATH=$PATH:$NDK_PATH/toolchains/llvm/prebuilt/darwin-x86_64/bin
export PATH=$PATH:~/Library/Android/sdk/cmdline-tools/latest/bin
{{< /tab >}}
{{< /tabpane >}}

Now that your development environment is ready and all pre-requisites installed, you can test the Audio Stable Open model.
Now that your development environment is ready and all pre-requisites installed, you can test the Audio Stable Open Small model.
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,12 @@ layout: learningpathall

## Download the model

Stable Audio Open is an open-source model optimized for generating short audio samples, sound effects, and production elements using text prompts.
Stable Audio Open Small is an open-source model optimized for generating short audio samples, sound effects, and production elements using text prompts.

[Log in](https://huggingface.co/login) to HuggingFace and navigate to the model landing page:

```bash
https://huggingface.co/stabilityai/stable-audio-open-small
https://huggingface.co/stabilityai/stable-audio-open-small/tree/main
```

You may need to fill out a form with your contact information to use the model:
Expand All @@ -26,15 +26,11 @@ Download and copy the configuration file `model_config.json` and the model itsel
ls $WORKSPACE/model_config.json $WORKSPACE/model.ckpt
```

## Test the model
You can learn more about this model [here](https://huggingface.co/stabilityai/stable-audio-open-small).

To test the model, use the Stable Audio demo site, which lets you experiment directly through a web-based interface:
### Good prompting practices

```bash
https://stableaudio.com/
```

Use the UI to enter a prompt. A good prompt can include:
A good prompt for this audio generation model can include:

* Music genre and subgenre.
* Musical elements (texture, rhythm and articulation).
Expand All @@ -45,5 +41,5 @@ The order of prompt parameters matters. For more information, see the [Prompt st

You can explore training and inference code for audio generation models in the [Stable Audio Tools repository](https://github.com/Stability-AI/stable-audio-tools).

Now that you've downloaded and tested the model, continue to the next section to convert the model to LiteRT.
Now that you've downloaded the model, continue to the next section to convert the model to LiteRT.

Original file line number Diff line number Diff line change
Expand Up @@ -5,27 +5,33 @@ weight: 4
### FIXED, DO NOT MODIFY
layout: learningpathall
---
In this section, you will learn about the audio generation model. You will then clone a repository to run conversion steps, which is needed to generate the inference application.

## Stable Audio Open Small Model
## Stable Audio Open Small

The open-sourced model includes three main parts. They are described in the table below, and come together through the pipeline shown in the image.

|Submodule|Description|
|------|------|
|Conditioners| Includes a T5-based text encoder for the input prompt and a numerical duration encoder. These components convert the inputs into embeddings passed to the DiT model. |
|Diffusion Transformer (DiT)| Denoises random noise over multiple steps to produce structured latent audio, guided by conditioner embeddings. |
|AutoEncoder| Compresses audio waveforms into a latent representation for processing by the DiT model, and decompresses the output back into audio. |

The submodules work together to provide the pipeline as shown below:

![Model structure#center](./model.png)

As part of this section, you will covert each of the three submodules into [LiteRT](https://ai.google.dev/edge/litert) format, using two separate conversion routes:
1. Conditioners submodule - ONNX to LiteRT using [onnx2tf](https://github.com/PINTO0309/onnx2tf) tool.
2. DiT and AutoEncoder submodules - PyTorch to LiteRT using Google AI Edge Torch tool.
In this section, you will explore two different conversion routes, to convert the submodules to [LiteRT](https://ai.google.dev/edge/litert) format. Both methods will be run using Python wrapper scripts from the examples repository.

1. **ONNX to LiteRT**: using the `onnx2tf` tool. This is the traditional two-step approach (PyTorch -> ONNX -> LiteRT). You will use it to convert the Conditioners submodule.

2. **PyTorch to LiteRT**: using the Google AI Edge Torch tool. You will use this tool to convert the DiT and AutoEncoder submodules.


### Create virtual environment and install dependencies
## Download the sample code

The Conditioners submodule is made of the T5Encoder model. You will use the ONNX to TFLite conversion for this submodule.

To avoid dependency issues, create a virtual environment. In this guide, we will use `virtualenv`:
To avoid dependency issues, create a virtual environment. For example, you can use the following command:

```bash
cd $WORKSPACE
Expand All @@ -37,11 +43,11 @@ Clone the examples repository:

```bash
cd $WORKSPACE
git clone https://github.com/ARM-software/ML-examples/tree/main/kleidiai-examples/audiogen
cd audio-stale-open-litert
git clone https://github.com/ARM-software/ML-examples.git
cd ML-examples/kleidiai-examples/audiogen/
```

We now install the needed python packages for this, including *onnx2tf* and *ai_edge_litert*
Install the needed Python packages for this, including *onnx2tf* and *ai_edge_litert*

```bash
bash install_requirements.sh
Expand All @@ -61,20 +67,19 @@ ImportError: cannot import name 'AttrsDescriptor' from 'triton.compiler.compiler
($WORKSPACE/env/lib/python3.10/site-packages/triton/compiler/compiler.py)
```

Install the following dependency and rerun the script:
Reinstall the following dependency:
```bash
pip install triton==3.2.0
bash install_requirements.sh
```

{{% /notice %}}

### Convert Conditioners Submodule

The Conditioners submodule is based on the T5Encoder model. We convert it first to ONNX, then to LiteRT.
The Conditioners submodule is based on the T5Encoder model. First, convert it to ONNX, then to LiteRT.

For this conversion we include the following steps:
1. Load the Conditioners submodule from the Stable Audio Open model configuration and checkpoint.
For this conversion, the following steps are needed:
1. Load the Conditioners submodule from the Stable Audio Open Small model configuration and checkpoint.
2. Export the Conditioners submodule to ONNX via *torch.onnx.export()*.
3. Convert the resulting ONNX file to LiteRT using *onnx2tf*.

Expand All @@ -84,28 +89,29 @@ You can use the provided script to convert the Conditioners submodule:
python3 ./scripts/export_conditioners.py --model_config "$WORKSPACE/model_config.json" --ckpt_path "$WORKSPACE/model.ckpt"
```

After successful conversion, you now have a `conditioners.onnx` model in your current directory.
After successful conversion, you now have a `tflite_conditioners` directory containing models with different precisions (e.g., float16, float32).

You will be using the float32.tflite model for on-device inference.

### Convert DiT and AutoEncoder

To convert the DiT and AutoEncoder submodules, use the [Generative API](https://github.com/google-ai-edge/ai-edge-torch/tree/main/ai_edge_torch/generative/) provided by the ai-edge-torch tools. This enables you to export a generative PyTorch model directly to tflite using three main steps:
To convert the DiT and AutoEncoder submodules, use the [Generative API](https://github.com/google-ai-edge/ai-edge-torch/tree/main/ai_edge_torch/generative/) provided by the ai-edge-torch tools. This enables you to export a generative PyTorch model directly to `.tflite` using three main steps:

1. Model re-authoring.
2. Quantization.
3. Conversion.

Convert the DiT and AutoEncoder submodules using the provided python script:
Convert the DiT and AutoEncoder submodules using the provided Python script:

```bash
CUDA_VISIBLE_DEVICES="" python3 ./scripts/export_dit_autoencoder.py --model_config "$WORKSPACE/model_config.json" --ckpt_path "$WORKSPACE/model.ckpt"
python3 ./scripts/export_dit_autoencoder.py --model_config "$WORKSPACE/model_config.json" --ckpt_path "$WORKSPACE/model.ckpt"
```

After successful conversion, you now have `dit_model.tflite` and `autoencoder_model.tflite` models in your current directory and can deactivate the virtual environment:
After successful conversion, you now have `dit_model.tflite` and `autoencoder_model.tflite` models in your current directory.

```bash
deactivate
```
A more detailed explanation of the above scripts is available [here](https://github.com/ARM-software/ML-examples/blob/main/kleidiai-examples/audiogen/scripts/README.md)

For easier access, we add all needed models to one directory:
For easy access, add all needed models to one directory:

```bash
export LITERT_MODELS_PATH=$WORKSPACE/litert-models
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ layout: learningpathall

## LiteRT

LiteRT (short for Lite Runtime), formerly known as TensorFlow Lite, is Google's high-performance runtime for on-device AI.
LiteRT (short for Lite Runtime), formerly known as TensorFlow Lite, is Google's high-performance runtime for on-device AI. Designed for low-latency, resource-efficient execution, LiteRT is optimized for mobile and embedded environments — making it a natural fit for Arm CPUs running models lite Stable Audio Open Small. You will build the runtime using the framework using the Bazel build tool.

## Build LiteRT libraries

Expand All @@ -20,55 +20,51 @@ git clone https://github.com/tensorflow/tensorflow.git tensorflow_src
cd tensorflow_src
```

We will use a specific commit of tensorflow for build so you can checkout and set the `TF_SRC_PATH`:
Check out the specified commit of TensorFlow, and set the `TF_SRC_PATH`:
```bash
git checkout 84dd28bbc29d75e6a6d917eb2998e4e8ea90ec56
export TF_SRC_PATH=$(pwd)
```

We can use `bazel` to build LiteRT libraries, first we use configure script to create a custom configuration for this:
A script is available to configure the `bazel` build environment. Run it to create a custom TFLite build for Android:

You can now create a custom TFLite build for android:

Ensure the `ANDROID_NDK` variable is set to your previously installed Android NDK:
{{% notice Reminder %}}
Ensure the `NDK_PATH` variable is set to your previously installed Android NDK:
{{< tabpane code=true >}}
{{< tab header="Linux">}}
export ANDROID_NDK=$WORKSPACE/android-ndk-r25b/
export PATH=$ANDROID_NDK/toolchains/llvm/prebuilt/linux-x86_64/bin/:$PATH
export NDK_PATH=$WORKSPACE/android-ndk-r25b/
export PATH=$NDK_PATH/toolchains/llvm/prebuilt/linux-x86_64/bin/:$PATH
{{< /tab >}}
{{< tab header="MacOS">}}
export TF_CXX_FLAGS="-DTF_MAJOR_VERSION=0 -DTF_MINOR_VERSION=0 -DTF_PATCH_VERSION=0 -DTF_VERSION_SUFFIX=''"
export ANDROID_NDK=~/Library/Android/sdk/ndk/27.0.12077973/
export PATH=$PATH:$ANDROID_NDK/toolchains/llvm/prebuilt/darwin-x86_64/bin
export PATH=$PATH:~/Library/Android/sdk/cmdline-tools/latest/bin
export NDK_PATH=~/Library/Android/android-ndk-r25b
export PATH=$PATH:$NDK_PATH/toolchains/llvm/prebuilt/darwin-x86_64/bin
{{< /tab >}}
{{< /tabpane >}}
{{% /notice %}}

Set the TensorFlow version
The configuration script is interactive. Run it using the command below, and use the table to set the parameters for this Learning Path use-case.

```bash
export TF_CXX_FLAGS="-DTF_MAJOR_VERSION=0 -DTF_MINOR_VERSION=0 -DTF_PATCH_VERSION=0 -DTF_VERSION_SUFFIX=''"
```


Now you can configure TensorFlow. Here you can set the custom build parameters needed as follows:

```bash { output_lines = "2-14" }
python3 ./configure.py
Please specify the location of python. [Default is $WORKSPACE/bin/python3]:
Please input the desired Python library path to use. Default is [$WORKSPACE/lib/python3.10/site-packages]
Do you wish to build TensorFlow with ROCm support? [y/N]: n
Do you wish to build TensorFlow with CUDA support? [y/N]: n
Do you want to use Clang to build TensorFlow? [Y/n]: n
Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]: y
Please specify the home path of the Android NDK to use. [Default is /home/user/Android/Sdk/ndk-bundle]: /home/user/Workspace/tools/ndk/android-ndk-r25b
Please specify the (min) Android NDK API level to use. [Available levels: [16, 17, 18, 19, 21, 22, 23, 24, 26, 27, 28, 29, 30, 31, 32, 33]] [Default is 21]: 30
Please specify the home path of the Android SDK to use. [Default is /home/user/Android/Sdk]:
Please specify the Android SDK API level to use. [Available levels: ['31', '33', '34', '35']] [Default is 35]:
Please specify an Android build tools version to use. [Available versions: ['30.0.3', '34.0.0', '35.0.0']] [Default is 35.0.0]:
```

Once the bazel configuration is complete, you can build TFLite as follows:
|Question|Input|
|---|---|
|Please specify the location of python. [Default is $WORKSPACE/bin/python3]:| Enter (default) |
|Please input the desired Python library path to use[$WORKSPACE/lib/python3.10/site-packages] | Enter |
|Do you wish to build TensorFlow with ROCm support? [y/N]|N (No)|
|Do you wish to build TensorFlow with CUDA support?|N|
|Do you want to use Clang to build TensorFlow? [Y/n]|N|
|Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]|y (Yes) |
|Please specify the home path of the Android NDK to use. [Default is /home/user/Android/Sdk/ndk-bundle]| Enter |
|Please specify the (min) Android NDK API level to use. [Default is 21] | 27 |
|Please specify the home path of the Android SDK to use. [Default is /home/user/Android/Sdk]| Enter |
|Please specify the Android SDK API level to use. [Default is 35]| Enter |
|Please specify an Android build tools version to use. [Default is 35.0.0]| Enter |
|Do you wish to build TensorFlow with iOS support? [y/N]:| n |

Once the Bazel configuration is complete, you can build TFLite as follows:

```console
bazel build -c opt --config android_arm64 //tensorflow/lite:libtensorflowlite.so \
--define tflite_with_xnnpack=true \
Expand All @@ -77,7 +73,15 @@ bazel build -c opt --config android_arm64 //tensorflow/lite:libtensorflowlite.so
--define tflite_with_xnnpack_qu8=true
```

This will produce a `libtensorflowlite.so` shared library for android with XNNPack enabled, which we will use to build the example next.
The final step is to build flatbuffers used by the application:
```
cd $WORKSPACE/tensorflow_src
mkdir flatc-native-build && cd flatc-native-build
cmake ../tensorflow/lite/tools/cmake/native_tools/flatbuffers
cmake --build .
```

With flatbuffers and LiteRT built, you can now build the application for Android devices.



Expand Down
Loading