LFX-Mentorship-Pre-test-3170

This repository is a test integration of Whisper.cpp as a new WASI-NN backend for the LFX Mentorship project#3170

Table of Contents

• System Information
• Preparation
• Running llama
  • Preparation
  • Build
  • Selecting the Model
  • Talking with Llama
• Running whisper
  • Whisper preparation
  • Transcribe audio file
  • Real time transcription
  • Multilingual Transcription

System Information

OS: MacOs Sonoma 14.3 23D56 arm64
Kernel: 23.3.0
Homebrew: 4.2.8
cmake version: 3.28.3

Preparation

Create a clone using git.

git clone https://github.com/WasmEdge/WasmEdge.git && cd WasmEdge
git checkout hydai/0.13.5_ggml_lts

Running llama

Preparation

Installing dependencies

Cmake:brew install cmake

Ninja:brew install ninja

LLVM:brew install llvm

Follow the guide to run llama.cpp.

Build

Please run in the WasmEdge directory.

cmake -GNinja -Bbuild -DCMAKE_BUILD_TYPE=Release \
-DWASMEDGE_PLUGIN_WASI_NN_BACKEND="GGML" \
-DWASMEDGE_PLUGIN_WASI_NN_GGML_LLAMA_METAL=ON \
-DWASMEDGE_PLUGIN_WASI_NN_GGML_LLAMA_BLAS=OFF \
.

cmake --build build
cmake -install build

Download a cross-platform compatible portable Wasm file for the chat app. This application allows you to chat with the model via the command line.

curl -LO https://github.com/second-state/llama-utils/raw/main/chat/llama-chat.wasm

Selecting the Model

Choose and download a model in ggml format. This time, llama-2-7b-chat.Q5_K_M.gguf was selected.Please download your preferred model from here.

curl -LO https://huggingface.co/TheBloke/Llama-2-7b-Chat-GGUF/resolve/main/llama-2-7b-chat.Q5_K_M.gguf

Talking with Llama

wasmedge --dir .:. \
  --nn-preload default:GGML:AUTO:llama-2-7b-chat.Q5_K_M.gguf \
    wasmedge-ggml-llama.wasm default

If it doesn't work as expected, please check here.

Running whisper

Let's run whisper.cpp for transcription and real-time transcription!

Whisper preparation

1. Clone the repository.

git clone https://github.com/ggerganov/whisper.cpp.git
cd whisper.cpp

2. Select a model Download the whisper model converted to ggml format. This time, base.en was used.

bash ./models/download-ggml-model.sh base.en

You can choose your favorite model from here. Models are tiny, base, small, medium, large, with accuracy increasing from left to right, but also file size and memory usage.

Models are multilingual unless the model name includes .en.

Transcribe audio file.

Transcribe using a sample audio.

./main -f samples/jfk.wav

Success is achieved if transcription is completed as shown below.

And so my fellow Americans, ask not what your country can do for you, ask what you can do for your country.

Real time transcription

1. Download dependencies. We will download the dependencies and proceed with the execution.

brew install sdl2

2. Transcription in real time When executed, it can transcribe from microphone input.

make stream
./stream -m ./models/ggml-base.en.bin -t 8 --step 500 --length 5000

The demo audio was created with ElevenLabs.

whisper-sample.mp4

Multilingual Transcription

To use languages other than English, please specify the language using the -l option. Without this option, the system may default to English.

The example provided uses the 'base' model.The following command allows for real-time transcription in Japanese.

bash ./models/download-ggml-model.sh base

make stream
./stream -m ./models/ggml-base.bin -l ja -t 8 --step 500 --length 5000