Skip to content

Oído v2: real-time speech recognition on one ESP32-S3 core

Latest

Choose a tag to compare

@danivs10 danivs10 released this 03 Nov 21:38
· 23 commits to main since this release

Oído v2: faster than speech on one ESP32-S3 core

Version 1 ran at 1.8-2.0x slower than speech on the chip. Version 2 runs at 0.54 (variant B) and 0.72 (variant A) of the audio duration with the on-chip language model, and the final text appears 259 ms (B) / 431 ms (A) after the end of speech (audio fed at real-time speed; 178-634 ms on 15-89 s of continuous speech). Measured on a physical board (ESP32-S3 N16R8, one core, 240 MHz); the board's transcripts equal the host build's on 18 of 18 clips.

LibriSpeech test-clean / test-other, streaming 0.64 s chunks + language model WER %
A (16 blocks, 7.7 MB int4) 7.1 / 16.4
B (12 blocks, 5.9 MB int4) 9.8 / 21.0

Streaming costs accuracy: test-other stays above 10 % in every streaming mode (1.28 s chunks: A 6.0 / 14.0). English only. The firmware's end-of-speech wait (0.8 s by default) comes on top of the delays above.

Files

Try it

git clone https://github.com/lokutor-ai/oido && cd oido/esp32/host && make
python live_demo.py --model ../../models/oido_v2a.tnm        # laptop microphone, same engine as the chip
../tools/flash.sh /dev/ttyUSB0 ../../models/oido_v2a.tnm      # ESP32-S3-DevKitC-1 N16R8 + INMP441 microphone