Speech Commands Classification with Transformers

Authors: Jakub Kępka, Damian Kąkol Course: Deep Learning

Overview

This project investigates keyword spotting on the Google Speech Commands dataset in a 12-class setting:

Class	Type
silence	special
unknown	special
yes, no, up, down, left, right, on, off, stop, go	target commands

Three research questions are addressed:

1. Does a lightweight Transformer (KWT-style) outperform a simple CNN?
2. Does a pretrained audio Transformer (AST) further improve accuracy?
3. Does explicit handling of silence/unknown (class rebalancing or hierarchical pipeline) improve robustness?

---

Project Structure

speech-commands/
├── configs/            # YAML experiment configs (data, model, train, experiments)
├── data/raw/           # Dataset downloaded here (auto-populated)
├── outputs/            # Checkpoints, logs, figures, result tables
├── scripts/            # Entry-point scripts
│   ├── prepare_data.py  – download & inspect dataset
│   ├── train.py         – train a single model
│   ├── evaluate.py      – evaluate on test split
│   ├── run_experiment.py – run an experiment across multiple seeds
│   └── export_results.py – aggregate metrics into summary tables
├── src/
│   ├── data/            – dataset, transforms, sampler, labels
│   ├── models/          – CNN baseline, KWT, AST wrapper, hierarchical
│   ├── training/        – trainer, losses, callbacks, seed
│   ├── evaluation/      – metrics, confusion matrix, reports
│   ├── inference/       – single-file predictor
│   └── utils/           – config loading, I/O, logging
└── tests/               – pytest unit & integration tests

---

Quick Start

0. (macOS only) Install Audio Backend

On macOS, you need to install libsndfile for audio file loading:

# Option A: Use the setup script
bash scripts/setup_audio_backend.sh

# Option B: Manual installation
brew install libsndfile

If this step is skipped and the backend is missing, silence samples will be generated as zeros (the training will still work, but less realistic).

1. Install Dependencies

pip install -r requirements.txt
# or
pip install -e ".[dev]"

2. Download Dataset

python scripts/prepare_data.py --root ./data/raw

3. Train a Model

# CNN baseline
python scripts/train.py --config configs/experiments/A1_cnn_baseline.yaml

# KWT-small
python scripts/train.py --config configs/experiments/A2_kwt_small.yaml

# KWT-medium
python scripts/train.py --config configs/experiments/A3_kwt_medium.yaml

# AST fine-tuning
python scripts/train.py --config configs/experiments/A4_ast.yaml

Note: A4_ast is intended for CUDA environments. In the current project setup, local macOS execution of AST is not supported and may fail at native-library initialisation time. Run A4 on Colab/Linux with GPU.

4. Evaluate on Test Split

python scripts/evaluate.py \
    --config configs/experiments/A1_cnn_baseline.yaml \
    --checkpoint outputs/checkpoints/A1/A1_CNN_Baseline_ch32-64-128-256_do10_bs128_seed42_best.pt

5. Run Full Experiment (multiple seeds)

python scripts/run_experiment.py --config configs/experiments/C1_flat.yaml
# Seeds are taken from the config file (default: [42, 123, 456])

6. Grid Search (multiple models, seeds, batch sizes)

python scripts/run_grid_experiments.py \
  --models A1 A2 A3 \
  --seeds 42 \
  --batch-sizes 128 256

This runs 6 training jobs: A1+128, A1+256, A2+128, A2+256, A3+128, A3+256 all on seed 42.

7. Export Summary Table

python scripts/export_results.py --tables-dir outputs/tables

8. Generate Per-Model Training/Test Report Figures

python scripts/report_model_results.py \
  --tables-dir outputs/tables \
  --figures-dir outputs/figures

For each model directory in outputs/tables/{model}, this creates files in outputs/figures/{model}/summary:

• training_curves.png — train/val loss + accuracy (all runs + mean/std)
• confusion_matrix_mean.png — mean normalized confusion matrix on test set
• test_metrics_by_batch.png — test accuracy and macro F1 grouped by batch size (mean over seeds)
• test_results.csv and test_summary_by_batch.csv
• analysis.txt — short text summary with best batch size by macro F1

---

Models

Model	File	Configuration
CNN Baseline	src/models/cnn_baseline.py	4 ConvBlocks → GAP → Linear
KWT-small	src/models/kwt.py	d=128, 4 layers, 4 heads
KWT-medium	src/models/kwt.py	d=192, 6 layers, 4 heads
AST	src/models/ast_wrapper.py	Pretrained AudioSet, fine-tuned head
Hierarchical	src/models/hierarchical.py	Stage1 (3-class) + Stage2 (10-class)

---

Experiments

ID	Model	Strategy	Seeds
A1	CNN Baseline	flat 12-class	1
A2	KWT-small	flat 12-class	1
A3	KWT-medium	flat 12-class	1
A4	AST	fine-tuning	1
C1	KWT-small	flat, no rebalancing	3
C2	KWT-small	flat + weighted sampler	3
C3	KWT-small	hierarchical pipeline	3

---

Running Tests

pytest tests/ -v

---

Reported Metrics

Each run produces in experiment-specific subdirectories, for example outputs/tables/A1/:

• <run>_metrics.json — accuracy, macro F1, per-class recall, n_params
• <run>_metrics.csv — same as CSV
• <run>_history.json — per-epoch train/val loss and accuracy

Run names now include the seed, batch size, and key architecture parameters, for example:

• A1_CNN_Baseline_ch32-64-128-256_do10_bs128_seed42
• A2_KWT_small_d128_h4_L4_do10_bs128_seed42

Each run produces figures in experiment-specific subdirectories, for example outputs/figures/A1/:

• <run>_confusion.png — normalised confusion matrix