Skip to content

STT

Jump to bottom
pschatzmann edited this page Oct 28, 2026 · 1 revision

Draft: this page is work in progress.

Speech-to-Text (STT) is the counterpart of Text-to-Speech: a microphone records your voice and you get back the recognized text. AudioTools provides the glue: it reads the microphone (I2S, AudioBoardStream, analog ...), converts the audio to the format that the recognizer expects (usually 16 kHz, mono, 16 bit), detects when somebody is speaking (VolumeMeter) and sends the data where it needs to go.

As with TTS there is no "one size fits all" solution. There are three approaches:

  • STT service on the internet: the microcontroller sends the audio to a cloud service and gets back the text. Best accuracy, any language, but you need internet access and usually an API key, and you pay per minute.
  • STT on a local server: the same, but the recognizer runs in your own network, e.g. on a PC or a Raspberry Pi (Vosk, Whisper ...). No cloud, no costs, and the audio stays at home.
  • STT on the microcontroller: the recognition runs on the device itself, without any network (TinySTT, TensorFlow Lite). Works offline and with the smallest latency, but the accuracy is limited.

Recording the Audio

Whatever approach you use, the first steps are the same:

  • Format: almost all recognizers expect 16000 Hz, mono, 16 bit PCM. Configure the microphone like this, or put a FormatConverterStream in front.
  • Voice activity: you don't want to send silence. A VolumeMeter can report when the audio gets active or silent: setActivityCallback(callback, threshold, silence_ms). Use it to start and stop an utterance.
  • Bandwidth: 16 kHz mono 16 bit is 32000 bytes per second, which is no problem for WiFi (see Communication).
AudioBoardStream mic(AudioKitEs8388V1);
VolumeMeter meter(mic);

void audioActive(bool active) {
  // active == false: the speaker stopped: end of the utterance
}

void setup() {
  auto cfg = mic.defaultConfig(RX_MODE);
  cfg.sample_rate = 16000;
  cfg.channels = 1;
  cfg.bits_per_sample = 16;
  cfg.input_device = ADC_INPUT_LINE2;  // microphone of the AudioKit
  mic.begin(cfg);

  meter.setActivityCallback(audioActive, 0.007, 2000);  // threshold, ms of silence
  meter.begin(cfg);
}

STT Service on the Internet

Cloud services like OpenAI, Google Cloud Speech-to-Text, Azure Speech or Deepgram provide the best accuracy and support many languages. They offer two kinds of APIs:

  • Upload a recording (REST): record the utterance (e.g. as WAV with a WAVEncoder into a memory buffer, a file or PSRAM) and send it with an HTTPS POST request. The answer is a JSON document with the text. This is simple, but you get the result only after the upload, and the recording must fit into memory.
  • Streaming (WebSocket or gRPC): send the audio while it is being recorded and get partial results back. This gives the shortest delay, and the memory requirements are small. The pattern is the same as for the local Vosk server below.

Things to consider:

  • You need an API key: don't put it into a sketch that you share.
  • HTTPS needs a lot of RAM on the microcontroller (TLS).
  • The costs are per minute of audio: only send audio when somebody is speaking.

There is no ready-to-run example for a cloud service yet.

STT on a Local Server

If you don't want to depend on the cloud, you can run the recognizer in your own network: e.g. on a PC, a NAS or a Raspberry Pi 4/5. The microcontroller streams the audio to the server and gets back the text.

Vosk

Vosk is an open source recognizer for 20+ languages that runs well on a Raspberry Pi. The vosk-server provides a WebSocket API, and there are Docker images with the models included, e.g. for English:

docker run -d -p 2700:2700 alphacep/kaldi-en:latest

The protocol is simple:

  1. Send a text message with the configuration: {"config":{"sample_rate":16000}}
  2. Send the PCM audio as binary messages.
  3. Send {"eof" : 1} at the end of the utterance.
  4. You receive the results as text messages: {"partial" : "..."} while you speak and {"text" : "..."} for the final result.

With the arduinoWebSockets library this looks as follows:

#include <WiFi.h>
#include <WebSocketsClient.h>  // https://github.com/Links2004/arduinoWebSockets
#include "AudioTools.h"
#include "AudioTools/AudioLibs/AudioBoardStream.h"

WebSocketsClient webSocket;
AudioBoardStream mic(AudioKitEs8388V1);
VolumeMeter meter(mic);

// end of the utterance: request the final result
void audioActive(bool active) {
  if (!active) webSocket.sendTXT("{\"eof\" : 1}");
}

void webSocketEvent(WStype_t type, uint8_t* payload, size_t length) {
  if (type == WStype_CONNECTED) {
    webSocket.sendTXT("{\"config\":{\"sample_rate\":16000}}");
  } else if (type == WStype_TEXT) {
    Serial.write(payload, length);  // JSON with the recognized text
    Serial.println();
  }
}

void setup() {
  Serial.begin(115200);
  WiFi.begin("ssid", "password");
  while (WiFi.status() != WL_CONNECTED) delay(500);

  webSocket.begin("192.168.1.44", 2700, "/");  // address of the Vosk server
  webSocket.onEvent(webSocketEvent);
  webSocket.setReconnectInterval(5000);

  auto cfg = mic.defaultConfig(RX_MODE);
  cfg.sample_rate = 16000;
  cfg.channels = 1;
  cfg.bits_per_sample = 16;
  cfg.input_device = ADC_INPUT_LINE2;
  mic.begin(cfg);

  meter.setActivityCallback(audioActive, 0.007, 2000);
  meter.begin(cfg);
}

void loop() {
  webSocket.loop();
  uint8_t buffer[512];
  size_t len = meter.readBytes(buffer, sizeof(buffer));
  // only send audio while somebody is speaking
  if (len > 0 && meter.isActive() && webSocket.isConnected()) {
    webSocket.sendBIN(buffer, len);
  }
}

The complete sketch is streams-audiotools-vosk (currently in the sandbox).

Whisper and others

  • whisper.cpp comes with a server that accepts a recorded WAV file via HTTP POST and returns the text. Whisper is more accurate than Vosk, but it needs more processing power: a PC is recommended, a Raspberry Pi 5 works with the small models.
  • Rhasspy and Home Assistant (Wyoming protocol) provide complete voice assistants with STT, intent recognition and TTS.

STT on the Microcontroller

TinySTT

TinySTT is a header-only C++ speech-to-text library for microcontrollers, the counterpart of TinyTTS. It is based on AudioTools and works in two stages:

  • Speech → Phonemes: log-mel features (computed with an AudioTools FFT of your choice) and a small streaming CTC phoneme model (0.9 M parameters, int8).
  • Phonemes → Text: a beam search constrained by a lexicon, with a language model or a command grammar such as turn (on | off) the (light | fan).

The two stages can run in one loop, on two cores or even on two different devices (connected by a serial line, UDP or ESP-NOW). No TensorFlow Lite or other inference runtime is needed.

TinySTT is an AudioOutput: you copy the microphone data into it, and it prints each recognized utterance as one line:

#include "AudioTools.h"
#include "AudioTools/FFT/AudioRealFFT.h"
#include "LittleFS.h"
#include "TinySTT.h"

I2SStream mic;
AudioRealFFT fft;
ModelContainer model;
TinySTT stt(fft, model, Serial);  // prints one line per utterance
StreamCopy copier(stt, mic);

void setup() {
  Serial.begin(115200);
  LittleFS.begin();
  File file = LittleFS.open("/tinystt.tstt", "r");
  model.begin(file, file.size());

  auto cfg = mic.defaultConfig(RX_MODE);
  cfg.sample_rate = 16000;
  cfg.channels = 1;
  cfg.bits_per_sample = 16;
  mic.begin(cfg);

  stt.begin();  // starts the decoder task
}

void loop() { copier.copy(); }

Things to consider:

  • Hardware: you need a board with PSRAM: the English models need 3.1 MB or 9.5 MB. Speech → phonemes runs faster than real time on the ESP32-S3 (0.43 RTF), the ESP32-P4 (0.16 RTF) and the RP2350 with PSRAM.
  • Accuracy: the word error rate on LibriSpeech is 26 - 30 %. This is good for voice commands with a grammar, but not yet for reliable dictation.
  • Language: English only.

See the TinySTT documentation for the details and the examples.

Keyword spotting with TensorFlow Lite

If you only need to recognize a few words, keyword spotting is the most efficient solution: the TensorFlow Lite MicroSpeech example recognizes "yes" and "no" with the TfLiteAudioStream.

Choosing an Approach

Internet service Local server Microcontroller
Examples OpenAI, Google, Azure, Deepgram Vosk, Whisper, Rhasspy TinySTT, TensorFlow Lite
Accuracy very good good (Vosk) to very good (Whisper) commands, keywords
Languages many many English (TinySTT)
Network internet local WiFi none
Costs per minute own server none
Privacy audio leaves your home audio stays at home audio stays on the device
Hardware any WiFi board any WiFi board + PC / Raspberry Pi ESP32-S3 / P4 or RP2350 with PSRAM
  • Want to control your device with a few spoken commands, without any network? Use TinySTT with a grammar, or TensorFlow Lite for a handful of keywords.
  • Want to transcribe free speech and have a PC or Raspberry Pi running anyway? Use a local server like Vosk or Whisper.
  • Need the best accuracy or many languages and don't mind the cloud? Use an internet service.

Clone this wiki locally