Skip to content

About

Unity Package for the Whisper Tiny English model

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

16 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 

Repository files navigation

Whisper Tiny EN (OnnxRuntime/DirectML) for Unity

https://assetstore.unity.com/packages/slug/351516

OpenAI Whisper is an automatic speech recognition (ASR) model trained on diverse audio; it transcribes speech to text for English. This package bundles the tiny English variant for lightweight, offline transcription on Windows.

ONNX Runtime is a high-performance inference engine for ONNX models; this package uses it to run the Whisper model efficiently on Windows.

DirectML is a Windows machine-learning acceleration API that runs on DirectX 12-capable GPUs and can fall back to CPU when needed.

This is a Windows x64 package that embeds the OpenAI Whisper tiny English model inside a single native DLL, accelerated by ONNX Runtime with DirectML. Supports the Unity Editor and Windows Standalone 64-bit builds. Setup is turnkey and fully offline: no downloads, no updates—unpack the DLL, open the example scenes directly from the package, and choose your audio source (clip or microphone) for transcription or dictation.

Platforms

  • Supported: Windows 64-bit — Unity Editor and Windows Standalone player builds.
  • Not supported: macOS, Linux, mobile, consoles.

What’s inside

  • Native plugin built on ONNX Runtime + DirectML; the full Whisper tiny EN model is contained in one DLL (no external model downloads, works fully offline).
  • Editor utilities to unpack binaries and example content.
  • Example scenes showing:
    • Audio clip transcription (from StreamingAssets).
    • Microphone capture for dictation with push-to-talk or continuous capture.

Package contents

  • Plugins/Windows/x64: native DirectML/ONNXRuntime DLL with the embedded Whisper tiny EN model (unpacked via menu).
  • Tests/Runtime/Scenes: example scenes that can be opened directly from the package or copied into your project.
  • StreamingAssets/Audio: sample WhisperTest.wav for clip-based transcription (unpacked via menu).
  • Editor scripts: menu items to unpack DLLs, test audio, and examples.
  • C# samples: microphone and clip transcription workflows.

Quick setup

  1. Unpack the DLL: Assets → WhisperTinyEn → Unpack WhisperTinyEn DLLs (combines the split archive into a single native DLL in Plugins/Windows/x64). The native DLL is shipped split so each file stays under common version-control size limits (often ~100 MB). If your repo enforces that limit, you can add the unpacked DLL to .gitignore and rerun Unpack WhisperTinyEn DLLs after pulling or on machines that need to rebuild the extracted binary.
  2. Unpack test audio: Assets → WhisperTinyEn → Unpack WhisperTinyEn Example Audio also drops the sample WhisperTest.wav into Assets/StreamingAssets/Audio for the clip-based example.
  3. Open an example scene (no import needed): open directly from the package (or copy to your project if you prefer) and press Play in the Editor. Everything runs offline; no extra downloads are required.

Audio sources

  • Audio Clip: Use the included WhisperTest.wav in StreamingAssets, or point to your own clip. Clips are downmixed and resampled to 16 kHz mono before inference.
  • Microphone (dictation): Uses Unity Microphone API (input). Modes:
    • Continuous capture: start/stop buttons, segments sent on silence break or max duration.
    • Push-to-talk: hold a button to capture, release to send.

Example features

  • English transcription from PCM (16 kHz mono floats).
  • Rolling transcript buffer with max length guard.
  • UI hooks via TextMeshPro; dropdowns to pick audio source/device (input only; loopback not implemented in this sample).
  • Live progress percent + partial transcript updates during decoding.

Notes

  • Prefers a DX12-capable GPU/driver for DirectML acceleration; will fall back to CPU if GPU isn’t available.
  • Keep Plugins and StreamingAssets in place; the model DLL must remain alongside the plugin binaries.
  • Works offline; no model downloads or updates required.
  • Works in Editor and Windows Standalone 64-bit builds (x64 only).
  • On unsupported platforms, GetVersion, DetectLanguage, and Decode return an empty string.
  • Long clips (> ~30s) are processed in multiple overlapping windows and stitched into a single transcript.

Tech stack

  • Model: OpenAI Whisper (tiny.en) converted to ONNX. See available models.
  • Runtime: ONNX Runtime with DirectML execution provider (Windows x64).
  • Unity: TextMeshPro + UGUI for UI bindings; Unity Microphone API for input.

Licensing

Troubleshooting

  • Missing TMPro types: ensure your asmdef references Unity.TextMeshPro (and UnityEngine.UI for UGUI).
  • DLL load issues: confirm x64 Editor, DirectML-capable GPU/driver, and that DLLs were unpacked to Plugins/Windows/x64.
  • No audio input devices: check OS privacy settings and that a microphone exists.

Minimal API

var whisperTinyEn = new DllWhisperTinyEn();          // create the client (loads the native DLL/model)

var version = whisperTinyEn.GetVersion();            // optional: native DLL version string

// pcm: float[] mono, 16 kHz samples
var transcript = whisperTinyEn.Decode(pcm);          // decode transcript

That’s it: create the client, pass PCM (mono, 16 kHz floats), and decode.

API

  • DllWhisperTinyEn()
    • Input: none; creates the client and loads the native DLL/model (may throw if binaries are missing).
    • Output: client instance; inference methods will return empty strings on unsupported platforms.
  • GetVersion()
    • Output: native DLL version string (for example, "1.2.3"); empty string on unsupported platforms.
  • DetectLanguage(float[] pcmMono16k)
    • Input: mono 16 kHz PCM samples (float[]), values typically in [-1, 1].
    • Output: language code string (for example, "en"); empty string on unsupported platforms or load failure.
    • Notes:
      • This package is Whisper tiny.en (English-only). DetectLanguage(...) is provided for API parity and typically returns "en".
  • Decode(float[] pcmMono16k)
    • Input: mono 16 kHz PCM samples (float[]), values typically in [-1, 1].
    • Output: transcript string; empty string on unsupported platforms or load failure.
    • Notes:
      • Long clips are decoded in multiple overlapping windows and stitched into a single transcript.
      • If an in-flight decode is cancelled via Abort(), Decode returns an empty string.

Progress

  • DllWhisperTinyEn.SetProgress(DllWhisperTinyEn.ProgressCallback callback)
    • Registers a native progress callback.
    • Callback signature: void (int stage, int current, int total, string partialText)
      • stage: 0..3
        • 0: preprocess / segment-level progress (for long clips)
        • 1: encoder
        • 2: decoder
        • 3: done
      • current / total: stage-local counters
      • partialText: partial transcript (may be empty). Most useful during stage 2.
    • Threading: the callback may be invoked from a native/background thread. Do not touch Unity objects from the callback; copy values into fields and update UI on the main thread.
  • DllWhisperTinyEn.ClearProgress()
    • Clears the registered progress callback.

To compute a simple overall “total percent” across 3 running stages (0..2):

static int TotalPercent(int stage, int current, int total)
{
    const int stages = 3;
	double stageNormalized = (total > 0) ? Math.Clamp((double)current / total, 0.0, 1.0) : 0.0;
	double totalNormalized = (Math.Clamp(stage, 0, stages - 1) + stageNormalized) / stages;
	return (int)Math.Round(totalNormalized * 100.0);
}

If you want smoother progress for long clips (multi-window decode), treat stage 0 current/total as (segmentIndex/segmentCount) and compute:

// total = (segmentIndex*100 + withinSegment) / segmentCount

Cancellation

  • DllWhisperTinyEn.Abort()
    • Requests cooperative cancellation of the currently-running native operation (for example, an in-flight Decode).
    • This is a global/native cancellation flag; it is not tied to a specific DllWhisperTinyEn instance.

Getting PCM data

From an AudioClip

var clip = /* your AudioClip */;                              // source clip
var buffer = new float[clip.samples * clip.channels];         // allocate interleaved buffer
clip.GetData(buffer, 0);                                      // read samples from clip

// Downmix to mono and resample to 16 kHz if needed
float[] pcmMono16k = DownmixAndResample(buffer, clip.channels, clip.frequency, targetRate: 16000);

From Microphone (Unity Microphone API)

string device = Microphone.devices.FirstOrDefault();          // pick a mic
int sampleRate = 16000;                                       // request 16 kHz
var micClip = Microphone.Start(device, loop: true, lengthSec: 20, frequency: sampleRate);

// Periodically read new samples, append to a List<float>, then send segments to Whisper
micClip.GetData(tempBuffer, offsetSamples);                   // grab the newest chunk
float[] pcmMono16k = tempBuffer;                              // already mono 16 kHz when requested

Guidelines:

  • Aim for mono 16 kHz floats. Downmix if the clip is stereo, and resample if its rate differs from 16 kHz.
  • For mic input, request 16000 Hz in Microphone.Start to avoid resampling; otherwise resample to 16 kHz before calling the API.

Examples

  • WhisperExampleAudioClip: Transcribe an audio clip to text.

Example 1

  • WhisperExampleMicrophone: Transcribe a microphone/audio source to text.

Example 2

About

Unity Package for the Whisper Tiny English model

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors