Wyoming Faster Whisper

Wyoming protocol server for the faster-whisper speech to text system.

Source

Clone the repository and set up Python virtual environment:

git clone https://github.com/OHF-Voice/wyoming-faster-whisper.git
cd wyoming-faster-whisper
script/setup

Run a server anyone can connect to:

script/run --model tiny-int8 --language en --uri 'tcp://0.0.0.0:10300' --data-dir /data --download-dir /data

The --model can also be a HuggingFace model like Systran/faster-distil-whisper-small.en

NOTE: Models are downloaded to the first --data-dir directory.

Whisper has never heard of your thermostat. "What's the temperature of the Ecobee?" comes back as "What's the temperature of the incubi?" — the acoustics were fine, the model just has no reason to think that word exists.

Given a long-lived access token, the server reads the names in your home over the Home Assistant websocket API and feeds them to the model as a prompt, which fixes exactly that class of error:

script/run --uri 'tcp://0.0.0.0:10300' --data-dir /data \
    --hass-token "$TOKEN" --hass-api 'http://homeassistant.local:8123/api'

Requires the hass extra:

pip install 'wyoming-faster-whisper[hass]'

It collects the names of conversation-exposed entities and their aliases, plus your area and floor names — the names a speaker can actually say. It also reads which area each exposed entity is in, so the areas you can actually command rank above the ones you can't. Nothing else is read, and no service is ever called.

The fetch is free in latency terms: it starts when the audio starts, while the speaker is still talking, and the names are ready by the time the audio stops. Home Assistant being slow or unreachable only costs freshness — the previous names are used, or none at all, and the transcript still comes back.

A large home has more names than the budget holds, so they are added in priority order and cut off when it runs out:

1. Areas and floors that hold an exposed entity — said in nearly every command ("turn on the office lamp"), and known to be real targets because something in them can actually be commanded.
2. Entity names in the domains people say out loud — light, switch, fan, media_player, climate, scene, todo. These are the proper nouns that get misheard.
3. The remaining areas and floors — sayable, but with nothing exposed in them there is no command they can complete.
4. The remaining entity names — in a big home mostly sensors, which are hundreds in number and usually asked about by area ("the temperature in the office") rather than by their own name.

Aliases are not a tier of their own. An alias is what you say instead of the name the integration gave the thing, so each one sits directly behind the name it belongs to and shares its tier — a light's "standing light" is kept or dropped along with the light, never left out while lower-priority names get in.

Edit PRIORITY_DOMAINS in wyoming_faster_whisper/hass_api.py to change what lands in tier 2. Run with --debug to see how many names were dropped and the exact prompt used. --initial-prompt still works and is kept at the front of the prompt, ahead of anything discovered from Home Assistant.

This biases faster-whisper and qwen3-asr, the backends that take a prompt. Others ignore it.

For qwen3-asr the prompt is not free: the model has to read it before it starts decoding, at roughly 2.8ms per token, so a 50-name list can double the time for a short command.

The default model avoids this. The prompt sits ahead of the audio in the chat template, so its state depends only on the prompt and is computed once, then reused for every later utterance. The layout is chosen from the files present, so older model directories with decoder_init/decoder_step keep working as before — pass --model rhasspy/qwen3-asr-0.6b-onnx-int4 to use one.

Measured on a Pi 5 (4 threads, 3.2s command, 50 names):

The latency win is for short commands. Long-form audio gains little (~1.04x on a 30s clip), because the cached prompt is a small share of that work — though the memory saving grows with length.

Accuracy is unchanged: on LibriSpeech test-other (n=200) the two produce byte-identical transcripts with no prompt (5.35% WER for both), and 5.33% vs 5.43% with a 50-name prompt.

docker run -it -p 10300:10300 -v /path/to/local/data:/data rhasspy/wyoming-whisper \
    --model tiny-int8 --language en

NOTE: Models are downloaded to /data, so make sure this points to a Docker volume.

Source

Dockerfile.gpu runs the speech-to-text backends on an NVIDIA GPU. It is not published to Docker Hub — you build it yourself, because it comes out around 10.7 GB (mostly the CUDA torch wheel) against ~1.6 GB for the CPU image, and Home Assistant OS has no GPU passthrough, so everyone who can use it is already running Docker directly.

git clone https://github.com/OHF-Voice/wyoming-faster-whisper.git
cd wyoming-faster-whisper
docker build -f Dockerfile.gpu -t wyoming-whisper:gpu .

Running it needs the NVIDIA Container Toolkit on the host, and --gpus:

docker run -it --gpus all -p 10300:10300 -v /path/to/local/data:/data \
    wyoming-whisper:gpu --language en

--device cuda is the default in this image; pass --device cuda:1 to pick a GPU or --device cpu to fall back. The default faster-whisper model is Systran/faster-whisper-small (float16) rather than the int8 model used on the CPU, and a GPU can comfortably run much larger ones:

docker run -it --gpus all -p 10300:10300 -v /path/to/local/data:/data \
    wyoming-whisper:gpu --model Systran/faster-whisper-large-v3 --language en

Notes and limits:

• NVIDIA and amd64 only. CTranslate2, which faster-whisper is built on, has no ROCm or Intel XPU backend and publishes no arm64 CUDA wheel. The torch-based backends (--stt-library transformers, --stt-library funasr) would work on ROCm or XPU, but that would be a different image.
• Budget the disk. ~10.7 GB for the image, plus build cache. Don't build it on a Pi by accident.
• --stt-library sherpa runs on the CPU even in this image. A CUDA sherpa-onnx build exists, but sherpa-onnx bundles its own onnxruntime, and two CUDA-enabled onnxruntime builds in one process segfault. Its default Parakeet models are int8, which the CUDA provider gains little on, so the CPU wheel is the better half of that trade.

If --device cuda produces no speedup, check the log: the server warns when the installed onnxruntime or sherpa-onnx build has no usable CUDA support.

For a local install rather than a container, the two onnxruntime-based backends have GPU counterparts of their extras — onnx-asr-gpu and qwen3-asr-gpu, which pull onnxruntime-gpu in place of onnxruntime:

pip install 'wyoming-faster-whisper[transformers,onnx-asr-gpu,qwen3-asr-gpu]'

faster-whisper additionally needs a CUDA-enabled CTranslate2 and torch built for your CUDA version, and it depends on onnxruntime (for its bundled Silero VAD), which will pull the CPU package back in and clobber onnxruntime-gpu — both install the same onnxruntime module and whichever lands second wins, silently, since the CPU provider still loads every model. Reinstall onnxruntime-gpu last. Dockerfile.gpu does exactly this and is the working reference.

Every command-line option can also be set from the environment, which is what Docker Compose gives you to configure a container without rewriting its command:. The variable is the option's name, uppercased, with dashes as underscores and a WYO_WHISPER_ prefix:

Precedence: a command-line argument always wins over the environment, so a variable left over in a container can never silently override an explicit argument. A variable that starts with WYO_WHISPER_ but matches no option is reported at startup rather than ignored, so a typo doesn't leave the server running with a default you thought you had changed.

A few options don't take a plain string on the command line, so they read one specially:

• Flags (--debug, --vad-filter, --local-files-only, ...) take 1/true/yes/on or 0/false/no/off/empty. Unlike a plain "is it set?" test, WYO_WHISPER_DEBUG=false really does mean off.
• --data-dir, which can be repeated, splits on : — WYO_WHISPER_DATA_DIR=/data:/media/models. Passing any --data-dir on the command line replaces the list rather than adding to it.
• --vad-clip, which takes any number of values, splits on commas or spaces. Empty means the flag with no values, i.e. every library.
• --zeroconf, whose value is optional, takes the name to announce, or empty for the default name.

Any other variable left empty means the same as not setting it at all, which is what a compose file or a .env produces for a value whose author left it blank. An empty WYO_WHISPER_URI gets you argparse's "the following arguments are required", not a server that starts up and fails on an empty string.

Each variable also has a _FILE form naming a file to read the value out of: WYO_WHISPER_HASS_TOKEN_FILE=/run/secrets/hass_token. That is the convention Docker Compose and Swarm secrets use — a secret is mounted as a file rather than handed over as a variable — and it keeps a long-lived Home Assistant token out of both the process's command line and its environment. The file is preferred over the plain variable when both are set, and its trailing newline is stripped.

services:
  whisper:
    image: rhasspy/wyoming-whisper
    ports:
      - "10300:10300"
    volumes:
      - ./data:/data
    environment:
      WYO_WHISPER_MODEL: tiny-int8
      WYO_WHISPER_LANGUAGE: en
      WYO_WHISPER_HASS_API: http://homeassistant.local:8123/api
      WYO_WHISPER_HASS_TOKEN_FILE: /run/secrets/hass_token
    secrets:
      - hass_token

secrets:
  hass_token:
    file: ./secrets/hass_token.txt

In the Docker image, --uri, --data-dir, and --device have defaults baked into the entrypoint; those are dropped when the matching variable is set, so WYO_WHISPER_URI and friends work there too.