A Spanish Speech Benchmark for Heterogeneous Acoustic Conditions
As large audio language models (LALMs) advance, robust evaluation frameworks have become essential. In this context, Spanish speech understanding under realistic acoustic conditions has received particularly little attention. We introduce ESCUCHA, the first Spanish speech understanding benchmark designed to evaluate LALMs across heterogeneous acoustic conditions and reasoning abilities. ESCUCHA comprises 1,000 human-curated questions paired with audio, totaling 162.9 hours sourced directly “from the wild” rather than drawn from existing datasets, with durations ranging from a few seconds to over 80 minutes. The benchmark emphasizes reasoning, spanning 9 perceptual and 10 reasoning categories, and it captures linguistic diversity through multiple Spanish accents and non-normative speech. ESCUCHA further includes multi-audio questions, spo- ken questions, and audio instructions, and it flags which questions support open-ended evaluation. Benchmarking several state-of-the-art multimodal and speech models reveals substantial performance gaps relative to trained humans.
| Property | Value |
|---|---|
| Total questions | 1,000 |
| MCQA questions | 900 |
| AIF questions | 100 |
| MCQA 4-choice / 3-choice / 2-choice | 848 / 45 / 7 |
| Comparative (2-audio) questions | 254 |
| Distinct perception categories | 9 |
| Distinct reasoning categories | 10 |
| Total audio | ~163 h |
| Median duration per question | 86 s |
| Pathological-speech subset | 158 questions |
The benchmark is released as data/tsv/ESCUCHA.tsv (tab-separated, 32 columns) and data/tsv/ESCUCHA.json. Each JSON record:
{
"id": "2bea5f95-fede-4d08-9c25-df2f67b607c7",
"question": "¿En qué audio el habla es más inteligible?",
"choices": ["Primero", "Segundo", "Tercero", "Son iguales"],
"answer": "Primero",
"categories": {
"perception": "Audio Quality, Artifacts & Channel Characteristics",
"reasoning": "Comparative and Preference-Based Judgments"
},
"total_duration": 213.76,
"length_type": "Medium (2–10min)",
"eval_type": "mcqa",
"aux_metadata": {
"url_1": "https://www.youtube.com/watch?v=...",
"url_2": "https://www.youtube.com/watch?v=...",
"need_choices_to_answer": "Yes",
"constraint_id": "",
"constraint_params": "",
"verifier": "",
...
}
}For single-audio questions url_2 (and all _2 fields) are empty strings.
Every question is tagged with at least one perception category and one reasoning category.
| Category | Description |
|---|---|
| Speaker Identification | Speaker identity, voice characteristics, conversational roles |
| Speaker Demographics | Age, sex, sociolinguistic traits, speech disorders |
| Language Identification | Language, dialect, accent, code-switching |
| Lexical and Phrase-Level Recognition | Words, idioms, colloquialisms, specialized vocabulary |
| Syntactic and Sentence-Structure Processing | Subordination, coordination, ellipsis, clitics |
| Prosody Detection | Rhythm, accent, intonation, pauses |
| Paralinguistic/Emotion Recognition | Emotions, irony, sarcasm, laughter, sighs |
| Speech Activity, Turn-Taking & Overlap | Turn-taking, interruptions, backchannels |
| Audio Quality, Artifacts & Channel Characteristics | Noise, distortion, channel failures, intelligibility |
| Category | Description |
|---|---|
| Social Role and Relationship Inference | Family/professional relationships, status, politeness |
| Speaker Intent, Pragmatics and Causal Reasoning | Speech acts, implicature, irony, causality |
| Quantitative Reasoning | Counting, comparing quantities, arithmetic |
| Temporal and Ordering Reasoning | Event sequences, timelines, intervals |
| Logical/Consistency Reasoning | Contradiction detection, argument tracking |
| Cross-frontier Entity Linking | Linking mentions to real-world entities |
| Ground Truth and World Knowledge Integration | Encyclopedic knowledge, cultural expectations |
| Semantic Abstraction and Summarization | Main ideas, reformulation, synthesis |
| Comparative and Preference-Based Judgments | Comparatives, preferences, choosing between options |
| Coherence and Discourse Structure | Cohesion, narrative and argumentative structure |
| Bucket | Range | Questions |
|---|---|---|
| Brief | ≤ 30 s | ~130 |
| Short | 30 s – 2 min | ~432 |
| Medium | 2 – 10 min | ~186 |
| Long | 10 – 30 min | ~127 |
| Extended | > 30 min | ~125 |
Audio files are not distributed directly. Use the provided script to download them from YouTube:
python src/download_audio.py
# or with explicit paths:
python src/download_audio.py --tsv data/tsv/ESCUCHA.tsv --output data/audioFiles are saved as data/audio/{id}_1.wav and, for comparative questions, data/audio/{id}_2.wav. All files are 16 kHz mono PCM WAV. Already-downloaded files are skipped automatically.
Dependencies:
pip install yt-dlp
sudo apt install ffmpeg # or brew install ffmpeg on macOS
# Deno — needed by yt-dlp for JavaScript-challenge videos
curl -fsSL https://deno.land/install.sh | sh
export PATH="$HOME/.deno/bin:$PATH"python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txtsrc/evaluate.py is self-contained. It reads a JSON file where each record is a benchmark item with an added output field (your model's response):
python src/evaluate.py \
--input results/your_model_responses.json \
--output results/your_model_scored.jsonInput format — the ESCUCHA JSON with one extra field per item:
[
{
"id": "2bea5f95-...",
"question": "¿En qué audio el habla es más inteligible?",
"choices": ["Primero", "Segundo", "Tercero", "Son iguales"],
"answer": "Primero",
"eval_type": "mcqa",
"output": "A"
}
]The script prints overall accuracy, MCQA accuracy, AIF accuracy, and per-category / per-length breakdowns.
A response is correct if it matches either:
- The full answer text (case-insensitive):
"primero"matches"Primero" - The letter label (A–D) for the correct position in
choices:"A"ifchoices[0]is the gold answer
The choices array is pre-shuffled; the gold answer is uniformly distributed across positions.
AIF questions require a free-form Spanish response under a deterministic format constraint (e.g., "respond in exactly 3 bullet points"). Each item in aux_metadata carries constraint_id, constraint_category, and constraint_params. The verifier requires no external API call.
Verifier families: length (word / sentence counts), keyword inclusion / exclusion, format (lists, paragraphs, sections), structure (postscript, starts / ends with), style (case, question-only, keyword frequency).
| Split | Expected accuracy |
|---|---|
| MCQA 4-choice (848 qs) | 25.00 % |
| MCQA 3-choice (45 qs) | 33.33 % |
| MCQA 2-choice (7 qs) | 50.00 % |
| MCQA overall (900 qs) | 25.61 % |
| AIF (100 qs) | ~0 % |
| Whole benchmark (1,000 qs) | 23.05 % |
notebooks/dataset_overview.ipynb reproduces all paper figures and a summary statistics table. Figures are written to data/figures/ as PDF and PNG (300 dpi).
pip install pandas matplotlib seaborn jupyter ipykernel
jupyter notebook notebooks/dataset_overview.ipynbESCUCHA/
├── data/tsv/
│ ├── ESCUCHA.tsv # 1,000-question benchmark (tab-separated)
│ └── ESCUCHA.json # Same data in JSON format
├── notebooks/
│ └── dataset_overview.ipynb # Dataset statistics and paper figures
├── src/
│ ├── evaluate.py # Self-contained scorer (MCQA + AIF verifiers)
│ └── download_audio.py # Download audio from YouTube via yt-dlp
├── requirements.txt
└── README.md