MEXA — Mexican-language ASR Benchmark & Leaderboard

A contamination-resistant ASR benchmark for Mexican Spanish + 23 Mexican Indigenous languages (Nahuatl, Mixtec, Zapotec, Mazatec, Zoque, Amuzgo, Chinantec, Pame, Tlapanec, Triqui, Totonac, Purépecha-area — 6+ families), with a public HF Space leaderboard scored on a private, held-out test set.

The point it proves: every current SOTA — Whisper family, NVIDIA Parakeet/Canary, Meta MMS, IBM Granite — is good on Mexican Spanish (~14% WER) and useless on Mexican Indigenous languages (≥99% WER). That gap is the lab's mission.

Layout

Path	What
src/mexa/	The benchmark library: normalize (tone/glottal-aware WER/CER), fingerprint (decontamination), splits, registry (sources), build_benchmark, evaluate, eval_mms, eval_whisper_fw
scripts/	Baseline runners (run_baselines, baseline_nemo, baseline_mmasr, baseline_vllm_whisper), HF push (push_hf), Space deploy (deploy_space), run_gpu.sh/run_sotas.sh
space/	The Gradio leaderboard Space (ZeroGPU, private benchmark, admin-only writes)
docs/	benchmark-design.md, decisions.md (ADRs), infra.md

Design (see docs/benchmark-design.md)

• Private held-out test so models can't overfit; a public fingerprint registry (audio + text hashes) lets anyone decontaminate training data without seeing the test.
• Official corpus splits (speaker- and sentence-disjoint, verified).
• WER and CER, light normalization that keeps tone marks & glottal stops.

HF artifacts (under Thermostatic/)

• mexa-asr-benchmark (private test) · mexa-asr-fingerprints (public) · mexa-asr-leaderboard (public) · Space mexa-asr-leaderboard-space

Usage

NEBLINIA_DATA=/path/to/shared/data
PYTHONPATH=src uv run python -m mexa.build_benchmark        # build benchmark + fingerprints
PYTHONPATH=src uv run python scripts/run_baselines.py openai/whisper-large-v3
uv run python scripts/push_hf.py leaderboard               # push results