Milo: a Three.js voice companion with Whisper and Kokoro on ORT Web

[Dew Tech _ Mike (michaelegbo)]

Sharing Milo, our experimental Three.js voice companion. Its browser audio worker uses ONNX Runtime Web through Transformers.js and kokoro-js for Whisper English transcription and the optional local Kokoro voice.

The local audio path uses quantized ONNX models on the WASM CPU backend in a Web Worker. Runtime assets are served from the same origin under cross-origin isolation, and CPU threads are capped at four. The audio graph drives the avatar's mouth and hand/body motion. The lip movement is an approximation, not phoneme alignment.

The application work that may interest other ORT Web users is around the inference lifecycle: explicit model preparation, cached-file detection after reload, partial-download recovery, cancellation, and separate controls for releasing model memory versus deleting saved files. A preset clip lets visitors see the character speak before loading any model.

Scope and privacy: this is not a claim that the whole hosted application runs on ONNX or stays offline. Local Qwen replies use wllama separately. The current website also offers hosted speech, which sends output text to a voice provider, and optional personal ChatGPT replies, which send messages/context through Milo to OpenAI. Microphone recordings and Whisper transcription stay in the browser; local Kokoro is the voice fallback.

• Try Milo
• Audio worker implementation
• Source and setup

Real browser audio and cancellation paths have been exercised in desktop Chrome on Windows. This does not establish every browser or phone, and large models can exceed device memory. Milo is source available under PolyForm Noncommercial 1.0.0; it is free for personal/noncommercial use, with separate model and dependency licenses.

Disclosure: this is our project, built with AI coding assistance; this showcase was prepared and posted with an AI assistant at the owner's request. Sharing it as an integration example, not an upstream bug report or request for support. Feedback on the audio worker and model lifecycle is welcome.