English | 简体中文

Today's Large Audio Language Models (LALMs) are stuck in an offline paradigm: you hand them a complete audio clip, wait, and get a reply. Streaming audio models exist, but each one only handles a single, isolated task. There has never been a general streaming audio language model. We formalize that missing capability as a new concept the Audio Interaction Model and build the first one. AudioInteraction is a unified Audio Interaction Model that:

✅ Runs conventional offline audio tasks (ASR, S2TT, AQA...)

✅ Runs streaming audio tasks in real time (Voice chatting...)

✅ Achieves general streaming audio instruction following on a live stream

✅ Does all of the above inside a single, all-in-one model, and be always-on and proactive

Technical Report 📖 / StreamAudio-2M 🤗 / AudioInteraction Model 🤗 / Streaming-Audio-Bench 🏆

▶ Click to watch AudioInteraction listen, decide, and speak — live (YouTube)

• [Coming]: We will release the full dataset and data curation pipeline.

• [Coming]: The full training configs and pipeline.

• May 20, 2026: 🔥 We release StreamAudio-2M.

• May 20, 2026: 🔥 We release the AudioInteraction Inference and Training Codebase.

• May 19, 2026: 🔥 AudioInteraction model weights are now available on Hugging Face.

• May 19, 2026: 🔥 We release the AudioInteraction Technical Report.

• Quick Start
• Demos
• SoundFlow: Train your own Audio Interaction Model
• StreamAudio-2M dataset
• Evaluation results
• License, Citation & Stars

AudioInteraction is an always-on model: it keeps listening to incoming audio frames and decides for itself when to speak. By default it stays in a ⟨Silent⟩ state and only emits output when the task or the acoustic context warrants it — so you can open a single session, stream audio into it continuously, and watch every capability take turns on its own.

Installation

git clone https://github.com/AudioInteraction/AudioInteraction.git
cd AudioInteraction

conda create -n AudioInteraction python=3.12 -y
conda activate AudioInteraction
# please check if you are using torch-cuda
pip install -r requirements.txt
# install ffmpeg
conda install -c conda-forge ffmpeg

Download Weights

# download model weights from huggingface
export PYTHONPATH=./
python download.py

Run inference first, then start the WebUI demo.

# Add project root to PYTHONPATH
export PYTHONPATH=./

# 1. Offline inference
python infer_offline.py

# To test bundled samples, set input_path in infer_offline.py to one of:
# sample/01_count_bark/sequence.json
# sample/02_translate/sequence.json
# sample/03_cough_music/sequence.json

# 2. Real-time inference
python infer_online.py

# Download model weights from Hugging Face first
python web/server.py

# Open in browser:
# http://localhost:5001

Most audio models do one job and wait to be asked. AudioInteraction's defining trait is that all of its abilities live in the same continuous stream, and the model itself decides which one is needed at each moment. The demo below is one unbroken session, one model, no mode switches, no prompts — transcription, understanding, conversation, and proactive intervention simply happen as the soundscape changes.

Offline audio models answer a finished clip, but real audio needs a model that listens continuously and decides, moment to moment, whether to speak. SoundFlow trains a single model that at every chunk chooses between ⟨Speak⟩ and ⟨Silent⟩, so recognition, translation, and dialogue become instructions inside one always-on perceive–decide–respond loop — a Large Audio Interaction Model (LAIM) — instead of separate per-task models. The framework covers the whole pipeline: stitching short clips into long interactions for data, chunk-level decision training with history review and comprehension-aware silence, and asynchronous FIFO inference that cuts first-frame latency by 4.5×.

You can fine-tune AudioInteraction on your own streaming data, and you can also use this repository to train standard offline audio language models. There are two steps: build the training data, then train.

Edit the path constants at the top of each script first:

Online (streaming, multi-turn audio). One JSON object per line:

{"conversation": [
    {"audio_path": "/path/to/turn1.wav", "assistant": "reply 1", "emotion": "normal"},
    {"audio_path": "/path/to/turn2.wav", "assistant": "reply 2", "emotion": "happy"}
]}

• audio_path and assistant are required on every turn.
• emotion is optional and defaults to "normal". Allowed values: happy, sad, angry, surprise, normal, urgent.
• To make the model stay silent on a turn, set assistant to "<no need to response>".

A single-turn shorthand is also accepted:

{"merge_path": "/path/to/audio.wav", "assistant": "reply", "emotion": "normal"}

Offline (single-turn). One JSON object per line, either the flat form:

{"user": "user text", "assistant": "reply", "audio_path": "/path/to/audio.wav"}

or the online-style multi-turn shape, in which case only the first turn is used:

{"conversation": [{"user": "...", "assistant": "...", "audio_path": "..."}, ...]}

assistant is always required. The task variant is decided by which other fields are present:

# Online: <input.jsonl> <output.jsonl> <error.log> <feature_dir>
python src/audiointeraction/dataset/get_dataset_online.py \
    <input.jsonl> <output.jsonl> <error.log> <feature_dir>
# Example:
# python src/audiointeraction/dataset/get_dataset_online.py \
#     data/online_raw.jsonl data/online.jsonl logs/online.err features/online

# Offline: <input.jsonl> <output.jsonl> <error.log> <feature_dir>
python src/audiointeraction/dataset/get_dataset_offline.py \
    <input.jsonl> <output.jsonl> <error.log> <feature_dir>
# Example:
python src/audiointeraction/dataset/get_dataset_offline.py \
#     data/offline_raw.jsonl data/offline.jsonl logs/offline.err features/offline

Both scripts are resumable: re-running picks up where the previous run stopped, skipping any idx that was already written. For a parallel multi-GPU template, see src/audiointeraction/dataset/process_get_feature.sh.

# 1. Set the two data roots referenced by config.yaml
export DATA_ROOT=/path/to/your/jsonl/data
export CHECKPOINT_ROOT=/path/to/your/checkpoints
# Example:
# export DATA_ROOT=/data/audiointeraction/jsonl
# export CHECKPOINT_ROOT=/data/audiointeraction/ckpts

# 2. Edit hyperparameters / data sources in src/audiointeraction/finetune/config.yaml

# 3. Launch
python src/audiointeraction/finetune/full.py --config src/audiointeraction/finetune/config.yaml
# Example:
# python src/audiointeraction/finetune/full.py --config src/audiointeraction/finetune/config.yaml

StreamAudio-2M is a ~2.6M-item streaming instruction-following corpus (7.4M rounds, 66.7K hours) covering seven capabilities — audio understanding, real-time ASR, speech translation, voice chatting, proactive response, and environment-aware agent — built by collecting clips from real-world datasets (AudioSet, CommonVoice, CoVoST2, MOSS, …), synthesizing text into speech with CosyVoice, then concatenating them into streaming sequences with environmental noise and token-level annotation.

Each line is one streaming sequence made of multiple turns:

{
  "id": "voice_chatting_000123",
  "stream_scene_type": "Home Smart",
  "num_turns": 2,
  "turns": [
    {
      "user": "Turn the living room lights down a bit.",
      "assistant": "Sure, dimming them to 40%.",
      "emotion": "normal",
      "scene_type": "Home Smart",
      "audio_path": "voice_chatting/000123/turn_0.wav"
    },
    {
      "user": "Thanks. What's the temperature in here?",
      "assistant": "It's 22.5 degrees in the living room.",
      "emotion": "normal",
      "scene_type": "Home Smart",
      "audio_path": "voice_chatting/000123/turn_1.wav"
    }
  ]
}

Set assistant to "<no need to response>" for a turn where the model should stay silent.

We sincerely thank the creators, maintainers, and contributors of the public datasets and resources used in this work. We also thank the broader large audio language model community for laying the groundwork that made streaming audio modeling possible.

In particular, this project builds on the following open-source repositories:

• Qwen2.5-Omni — the audio encoder and language model backbone behind AudioInteraction.
• LitGPT — the training framework our finetuning code is built on.
• CosyVoice — the text-to-speech model used to synthesize speech during data construction.

This project will be released under the Apache-2.0 License. You can do everything with AudioInteraction 🎉

Citation: You can cite AudioInteraction using the following BibTeX entry. Thank you for your kindness 🙂

@misc{xie2026audiointeractionmodel,
      title={Audio Interaction Model}, 
      author={Zhifei Xie and Zihang Liu and Ze An and Xiaobin Hu and Yue Liao and Ziyang Ma and Dongchao Yang and Mingbao Lin and Deheng Ye and Shuicheng Yan and Chunyan Miao},
      year={2026},
      eprint={2606.05121},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2606.05121}, 
}