From c60ae1d829eada34b2205c2e6895fab53cfc39c2 Mon Sep 17 00:00:00 2001 From: Emma Ning <43255631+EmmaNingMS@users.noreply.github.com> Date: Wed, 15 May 2024 18:33:33 -0700 Subject: [PATCH] Update README.md and add model optimization sample --- mobile/examples/phi-3/android/README.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/mobile/examples/phi-3/android/README.md b/mobile/examples/phi-3/android/README.md index 50614f193..129a6d961 100644 --- a/mobile/examples/phi-3/android/README.md +++ b/mobile/examples/phi-3/android/README.md @@ -1,12 +1,13 @@ -# ONNX Runtime Mobile Phi-3 Android sample application with ORT GenAI library support +# Local Chatbot on Android with Phi-3, ONNX Runtime Mobile and ONNX Runtime Generate() API ## Overview -This is a basic [Phi-3](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-onnx) example application for [ONNX Runtime](https://github.com/microsoft/onnxruntime) on Android with [Ort-GenAI](https://github.com/microsoft/onnxruntime-genai) support for running generative AI models. This app demonstrates the usage of phi-3 model in a simple question answering chatbot mode. +This is a basic [Phi-3](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct) Android example application with [ONNX Runtime mobile](https://onnxruntime.ai/docs/tutorials/mobile/) and [ONNX Runtime Generate() API](https://github.com/microsoft/onnxruntime-genai) with support for efficiently running generative AI models. This app demonstrates the usage of phi-3 model in a simple question answering chatbot mode. ### Model -The model used here is from the source: [HuggingFace](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-onnx#phi-3-mini-4k-instruct-onnx-models) -and optimized for int4 CPU and Mobile usage. +The model used here is [ONNX Phi-3 model on HuggingFace](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-onnx/tree/main/cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4) with INT4 quantization and optimizations for mobile usage. + +You can also optimize your fine-tuned PyTorch Phi-3 model for mobile usage following this example [Phi3 optimization with Olive](https://github.com/microsoft/Olive/tree/main/examples/phi3). ### Requirements - Android Studio Giraffe | 2022.3.1 or later (installed on Mac/Windows/Linux)