MUGEN: Evaluating and Improving Multi-audio Understanding of Large Audio-Language Models

The official GitHub page of the paper "MUGEN: Evaluating and Improving Multi-audio Understanding of Large Audio-Language Models"

• Authors: Chih-Kai Yang*, Yun-Shao Tsai*, Yu-Kai Guo†, Ping-Le Tsai†, Yen-Ting Piao‡, Hung-Wei Chen‡, Ting-Lin Hsiao‡, Yun-Man Hsu‡, Ke-Han Lu, Hung-yi Lee (*†‡Equal Contribution)
• Affiliation: National Taiwan University · NTU AI Center of Research Excellence (NTU AI-CoRE)
• Paper link: https://arxiv.org/abs/2603.09714

Overview

Abstract

TL;DR: We propose MUGEN, a benchmark for multi-audio understanding in LALMs, and show that current models, including proprietary ones, degrade sharply as the number of concurrent audio inputs grows.

While multi-audio understanding is critical for large audio-language models (LALMs), it remains underexplored. We introduce MUGEN, a comprehensive benchmark evaluating this capability across speech, general audio, and music. Our experiments reveal consistent weaknesses in multi-audio settings, and performance degrades sharply as the number of concurrent audio inputs increases, identifying input scaling as a fundamental bottleneck. We further investigate training-free strategies and observe that Audio-Permutational Self-Consistency (APSC), which diversifies the order of audio candidates, helps models form more robust aggregated predictions, yielding up to 6.28% accuracy gains. Combining this permutation strategy with Chain-of-Thought further improves performance to 6.74%. These results expose blind spots in current LALMs and provide a foundation for evaluating complex auditory comprehension.

Key findings

• LALMs perform substantially better on semantic tasks than on non-semantic ones, exposing an uneven capability distribution.
• Accuracy declines steadily as the number of concurrent audio candidates increases — input scaling is a systematic bottleneck.
• Chain-of-Thought alone fails to resolve auditory perception bottlenecks; Audio-Permutational Self-Consistency (APSC) yields the largest gains, especially when combined with CoT.

News

• Our paper is now available on arXiv.
• Our dataset is released on HuggingFace! All 35 tasks are available at MUGEN-Benchmark.

Benchmark

MUGEN contains 9,250 audio clips across 35 tasks organized into 7 evaluation dimensions: Semantics & Pragmatics, Speaker & Demographics, Affective & Paralinguistic State, Temporal Awareness, Acoustic Scene & Event Analysis, Music Analysis, and Compositional Acoustic Reasoning. Each question is formulated as 5-way multiple choice over candidate audio clips; ten tasks additionally include a reference audio for reference-conditioned comparison.

See data/README.md for the data schema and evaluation/README.md for the LLM-judge protocol.

Baselines

• DeSTA2.5-Audio
• Qwen2.5-Omni-7B
• Audio Flamingo 3
• Voxtral-Mini-3B, Voxtral-Small-24B
• Phi-4-Multimodal-Instruct
• Gemini-3-pro (Low / High thinking levels)
• Cascade baseline: Whisper-large-v3 + Gemini-3-pro (ASR + LLM)

Open-source models are run with vLLM under greedy decoding (Voxtral uses the authors' recommended temperature=0.2, top_p=0.95; Gemini-3-pro uses temperature=1).

Leaderboard

Accuracy (%) on MUGEN across seven dimensions: Semantics & Pragmatics (S&P), Speaker & Demographics (S&D), Affective & Paralinguistic (A&P), Temporal Awareness (TA), Acoustic Scene & Event Analysis (AS&E), Music Analysis (MA), and Compositional Acoustic Reasoning (CA). Overall is the micro-average over all tasks. "Low" / "High" denote the thinking levels of Gemini. Best within each group in bold.

Model	S&P	S&D	A&P	TA	AS&E	MA	CA	Overall
Open-source LALMs
DeSTA2.5-Audio	46.67	22.00	21.75	17.20	30.40	18.00	28.40	24.91
Qwen2.5-Omni	70.00	12.00	32.50	15.20	10.80	53.20	18.00	28.69
Audio Flamingo 3	25.33	22.50	15.00	21.20	23.20	1.20	19.20	17.43
Voxtral-Mini-3B	59.33	25.50	30.00	20.00	22.40	24.80	23.60	27.83
Voxtral-Small-24B	64.67	27.50	32.50	22.80	22.40	25.60	16.80	28.63
Phi-4-Multimodal-Instruct	52.00	21.00	21.00	28.00	24.80	24.80	24.80	26.29
Proprietary LALM
Gemini-3-pro (Low)	89.33	68.00	77.00	48.00	55.60	69.60	69.20	67.66
Gemini-3-pro (High)	90.67	71.50	77.50	53.60	56.80	68.40	72.80	69.60
Cascaded systems
ASR + LLM (Low)	75.33	20.00	27.00	28.80	26.40	26.00	18.00	29.09
ASR + LLM (High)	82.67	16.50	27.75	28.80	26.00	26.00	22.40	30.06

Citation

If you find MUGEN helpful, please consider citing our paper:

@article{yang2026mugen,
    title={MUGEN: Evaluating and Improving Multi-audio Understanding of Large Audio-Language Models},
    author={Yang, Chih-Kai and Tsai, Yun-Shao and Guo, Yu-Kai and Tsai, Ping-Le and Piao, Yen-Ting and Chen, Hung-Wei and Hsiao, Ting-Lin and Hsu, Yun-Man and Lu, Ke-Han and Lee, Hung-yi},
    journal={arXiv preprint arXiv:2603.09714},
    year={2026}
}