A minimal, efficient encoder-decoder transformer model for speech-to-text (ASR) tasks. Inspired by OpenAI's Whisper, designed for research and educational purposes.
It is a lightweight automatic speech recognition (ASR) system. It follows the encoder-decoder transformer paradigm, processing audio features and generating transcriptions. The project aims to provide a simple, readable codebase for understanding and experimenting with modern ASR techniques.
Fig: Encoder-Decoder ASR Model Architecture
- Encoder: Processes input audio features (e.g., log-mel spectrograms) and produces hidden, contextual representations.
- Decoder: Autoregressively generates text tokens from the encoder's output.
- Positional Encoding: Used in both encoder and decoder to provide sequence order information.
- Downsampler: Reduces the temporal resolution of input features for efficiency.
The tokenizer is based on Byte Pair Encoding (BPE), similar to Whisper. It converts text to token IDs and vice versa, supporting multilingual and special tokens as needed.
Fig: Tokenization process
Audio or Sound is bascially air pressure that varies over time. It is the change in atmospheric presure caused by the vibration of air molecules. These fluctuations create regions of high and low pressure, which we perceive as sound waves. The frequency of these fluctuations determines the pitch of the sound, while the amplitude determines its loudness.
Fig: Waveform of a sound signal
For ease of processing, these audio signals are converted into a spectrogram, more precisely a log-mel spectrogram. It captures the frequence-time-intensity representation of the audio signal, making it suitable for input to the model.
Fig: Log-Mel Spectrogram of a sound signal
This helps in filtering out noise and irrelevant sounds from audio sources. It ensures words spoken by different people, man or woman, creates a similar spectrogram, making it easier for the model to learn and generalize.
The corresonsing audio transcript is tokenized into a sequence of tokens. For tokenization, we use a Byte Pair Encoding (BPE) tokenizer, which is efficient for handling large vocabularies and multilingual text.
For example, the Start-of-Sequence (SOS) token is used to indicate the beginning of a transcription, and the End-of-Sequence (EOS) token indicates its end. The tokenizer also handles special tokens like padding and unknown words.
labels: [50257, 32, 1862, 2576, 12049, 477, 287, 11398, 318, 5055, 319, 257, 13990, 290, 2045, 379, 257, 8223, 50258]
text: <SOS>A young girl dressed all in pink is standing on a fence and looking at a horse<EOS>
Training scripts and utilities are provided in the tinywhisper/train/ directory:
train.py: Main training loop, data loading, and optimization- Supports custom datasets and data augmentation
- Configurable via
tinywhisper/config/config.py
- Prepare your dataset (audio files and transcripts)
- Configure training parameters in
config.py - Run the training script:
python -m tinywhisper.train.train
Evaluation scripts are in tinywhisper/eval/:
evaluation.py: Computes WER/CER and other metrics on test data
You can use the model for inference after training:
- Load a trained checkpoint
- Use the inference utilities in
tinywhisper/inference/
This project is licensed under the MIT License. See LICENSE for details.