https://assetstore.unity.com/packages/slug/351516
OpenAI Whisper is an automatic speech recognition (ASR) model trained on diverse audio; it transcribes speech to text for English. This package bundles the tiny English variant for lightweight, offline transcription on Windows.
ONNX Runtime is a high-performance inference engine for ONNX models; this package uses it to run the Whisper model efficiently on Windows.
DirectML is a Windows machine-learning acceleration API that runs on DirectX 12-capable GPUs and can fall back to CPU when needed.
This is a Windows x64 package that embeds the OpenAI Whisper tiny English model inside a single native DLL, accelerated by ONNX Runtime with DirectML. Supports the Unity Editor and Windows Standalone 64-bit builds. Setup is turnkey and fully offline: no downloads, no updates—unpack the DLL, open the example scenes directly from the package, and choose your audio source (clip or microphone) for transcription or dictation.
- Supported: Windows 64-bit — Unity Editor and Windows Standalone player builds.
- Not supported: macOS, Linux, mobile, consoles.
- Native plugin built on ONNX Runtime + DirectML; the full Whisper tiny EN model is contained in one DLL (no external model downloads, works fully offline).
- Editor utilities to unpack binaries and example content.
- Example scenes showing:
- Audio clip transcription (from StreamingAssets).
- Microphone capture for dictation with push-to-talk or continuous capture.
- Plugins/Windows/x64: native DirectML/ONNXRuntime DLL with the embedded Whisper tiny EN model (unpacked via menu).
- Tests/Runtime/Scenes: example scenes that can be opened directly from the package or copied into your project.
- StreamingAssets/Audio: sample
WhisperTest.wavfor clip-based transcription (unpacked via menu). - Editor scripts: menu items to unpack DLLs, test audio, and examples.
- C# samples: microphone and clip transcription workflows.
- Unpack the DLL:
Assets → WhisperTinyEn → Unpack WhisperTinyEn DLLs(combines the split archive into a single native DLL inPlugins/Windows/x64). The native DLL is shipped split so each file stays under common version-control size limits (often ~100 MB). If your repo enforces that limit, you can add the unpacked DLL to.gitignoreand rerun Unpack WhisperTinyEn DLLs after pulling or on machines that need to rebuild the extracted binary. - Unpack test audio:
Assets → WhisperTinyEn → Unpack WhisperTinyEn Example Audioalso drops the sampleWhisperTest.wavintoAssets/StreamingAssets/Audiofor the clip-based example. - Open an example scene (no import needed): open directly from the package (or copy to your project if you prefer) and press Play in the Editor. Everything runs offline; no extra downloads are required.
- Audio Clip: Use the included
WhisperTest.wavin StreamingAssets, or point to your own clip. Clips are downmixed and resampled to 16 kHz mono before inference. - Microphone (dictation): Uses Unity
MicrophoneAPI (input). Modes:- Continuous capture: start/stop buttons, segments sent on silence break or max duration.
- Push-to-talk: hold a button to capture, release to send.
- English transcription from PCM (16 kHz mono floats).
- Rolling transcript buffer with max length guard.
- UI hooks via TextMeshPro; dropdowns to pick audio source/device (input only; loopback not implemented in this sample).
- Live progress percent + partial transcript updates during decoding.
- Prefers a DX12-capable GPU/driver for DirectML acceleration; will fall back to CPU if GPU isn’t available.
- Keep Plugins and StreamingAssets in place; the model DLL must remain alongside the plugin binaries.
- Works offline; no model downloads or updates required.
- Works in Editor and Windows Standalone 64-bit builds (x64 only).
- On unsupported platforms,
GetVersion,DetectLanguage, andDecodereturn an empty string. - Long clips (> ~30s) are processed in multiple overlapping windows and stitched into a single transcript.
- Model: OpenAI Whisper (tiny.en) converted to ONNX. See available models.
- Runtime: ONNX Runtime with DirectML execution provider (Windows x64).
- Unity: TextMeshPro + UGUI for UI bindings; Unity
MicrophoneAPI for input.
- This package is licensed under the Unity Asset Store Terms of Service and EULA (see https://unity.com/legal/as-terms and the license overview at https://support.unity.com/hc/en-us/articles/208601846-Asset-Store-licenses-Extension-Assets-Single-and-Multi-Entity-assets).
- This asset is intended to be treated as an Extension Asset; per Unity’s terms, you must purchase one seat per user who has access to the raw asset files.
- Third-party components:
- DirectML (MIT, https://github.com/microsoft/DirectML/blob/master/LICENSE)
- ONNX Runtime (MIT, https://github.com/microsoft/onnxruntime/blob/main/LICENSE)
- OpenAI Whisper (MIT, https://github.com/openai/whisper/blob/main/LICENSE)
- Missing TMPro types: ensure your asmdef references
Unity.TextMeshPro(andUnityEngine.UIfor UGUI). - DLL load issues: confirm x64 Editor, DirectML-capable GPU/driver, and that DLLs were unpacked to
Plugins/Windows/x64. - No audio input devices: check OS privacy settings and that a microphone exists.
var whisperTinyEn = new DllWhisperTinyEn(); // create the client (loads the native DLL/model)
var version = whisperTinyEn.GetVersion(); // optional: native DLL version string
// pcm: float[] mono, 16 kHz samples
var transcript = whisperTinyEn.Decode(pcm); // decode transcriptThat’s it: create the client, pass PCM (mono, 16 kHz floats), and decode.
DllWhisperTinyEn()- Input: none; creates the client and loads the native DLL/model (may throw if binaries are missing).
- Output: client instance; inference methods will return empty strings on unsupported platforms.
GetVersion()- Output: native DLL version string (for example, "1.2.3"); empty string on unsupported platforms.
DetectLanguage(float[] pcmMono16k)- Input: mono 16 kHz PCM samples (
float[]), values typically in [-1, 1]. - Output: language code string (for example,
"en"); empty string on unsupported platforms or load failure. - Notes:
- This package is Whisper tiny.en (English-only).
DetectLanguage(...)is provided for API parity and typically returns"en".
- This package is Whisper tiny.en (English-only).
- Input: mono 16 kHz PCM samples (
Decode(float[] pcmMono16k)- Input: mono 16 kHz PCM samples (
float[]), values typically in [-1, 1]. - Output: transcript string; empty string on unsupported platforms or load failure.
- Notes:
- Long clips are decoded in multiple overlapping windows and stitched into a single transcript.
- If an in-flight decode is cancelled via
Abort(),Decodereturns an empty string.
- Input: mono 16 kHz PCM samples (
DllWhisperTinyEn.SetProgress(DllWhisperTinyEn.ProgressCallback callback)- Registers a native progress callback.
- Callback signature:
void (int stage, int current, int total, string partialText)stage: 0..3- 0: preprocess / segment-level progress (for long clips)
- 1: encoder
- 2: decoder
- 3: done
current/total: stage-local counterspartialText: partial transcript (may be empty). Most useful during stage 2.
- Threading: the callback may be invoked from a native/background thread. Do not touch Unity objects from the callback; copy values into fields and update UI on the main thread.
DllWhisperTinyEn.ClearProgress()- Clears the registered progress callback.
To compute a simple overall “total percent” across 3 running stages (0..2):
static int TotalPercent(int stage, int current, int total)
{
const int stages = 3;
double stageNormalized = (total > 0) ? Math.Clamp((double)current / total, 0.0, 1.0) : 0.0;
double totalNormalized = (Math.Clamp(stage, 0, stages - 1) + stageNormalized) / stages;
return (int)Math.Round(totalNormalized * 100.0);
}If you want smoother progress for long clips (multi-window decode), treat stage 0 current/total as (segmentIndex/segmentCount) and compute:
// total = (segmentIndex*100 + withinSegment) / segmentCountDllWhisperTinyEn.Abort()- Requests cooperative cancellation of the currently-running native operation (for example, an in-flight
Decode). - This is a global/native cancellation flag; it is not tied to a specific
DllWhisperTinyEninstance.
- Requests cooperative cancellation of the currently-running native operation (for example, an in-flight
var clip = /* your AudioClip */; // source clip
var buffer = new float[clip.samples * clip.channels]; // allocate interleaved buffer
clip.GetData(buffer, 0); // read samples from clip
// Downmix to mono and resample to 16 kHz if needed
float[] pcmMono16k = DownmixAndResample(buffer, clip.channels, clip.frequency, targetRate: 16000);string device = Microphone.devices.FirstOrDefault(); // pick a mic
int sampleRate = 16000; // request 16 kHz
var micClip = Microphone.Start(device, loop: true, lengthSec: 20, frequency: sampleRate);
// Periodically read new samples, append to a List<float>, then send segments to Whisper
micClip.GetData(tempBuffer, offsetSamples); // grab the newest chunk
float[] pcmMono16k = tempBuffer; // already mono 16 kHz when requestedGuidelines:
- Aim for mono 16 kHz floats. Downmix if the clip is stereo, and resample if its rate differs from 16 kHz.
- For mic input, request 16000 Hz in
Microphone.Startto avoid resampling; otherwise resample to 16 kHz before calling the API.
- WhisperExampleAudioClip: Transcribe an audio clip to text.
- WhisperExampleMicrophone: Transcribe a microphone/audio source to text.

