X2Streaming-ASR

English | 简体中文

Speech flows in, final text flows out: captions appear as you speak, ambiguous words wait for context, and committed text is safe for downstream agents to act on.

News · Overview · Demo · Why X2 · Results · Method · Quick Start · Citation

• [2026-09] The paper is on arXiv: 2609.08672.
• [2026-10] Model weights on Hugging Face: x-square-robot/X2Streaming-ASR-4B-1009.

X2Streaming-ASR is a low-latency, append-only streaming ASR model for Chinese, English, and code-switching speech. On the evaluated benchmarks, it emits each character or word with a mean latency of only 32–109 ms in Chinese and 12–85 ms in English after the word ends, while achieving the best CER/WER on several test sets.

Built on Voxtral Mini Realtime, it splits streaming recognition into when to commit and what to commit. Every 80 ms the model chooses between LISTEN and DECODE:

• Wait when uncertain. If the latest audio is ambiguous, the model keeps listening for more context.
• Emit when ready. Once the context is sufficient, it decodes the new audio immediately.
• Never roll back. Output is append-only, so text is never revised after it appears.
• Native causal streaming. Only newly arrived audio is encoded, and history is reused through the KV cache instead of being recomputed or cut off by a sliding window.

This repository is inference-only: adaptive streaming and full-context offline inference, Transformers and vLLM backends, JSONL batch transcription, and a web demo.

Unmute for audio.

• Colored tokens in the reference transcript identify the model that first produced a reliably aligned committed result: X2 in cyan, Qwen in blue, R2T2 in orange, and X-ASR in purple. Once assigned, a color never changes.
• The live error rate compares each model's committed output with subtitle cues that ended at least 1.5 seconds before the current playback position.

Fixed chunks or lookaheads make every word wait the same amount of time. Committing exactly at the acoustic boundary gets homophones wrong. Revising partial results makes text flicker and is unsafe for voice agents. Re-encoding the whole history or sliding a short window either slows down on long audio or forgets earlier speech. X2Streaming-ASR learns a per-word commit policy that avoids all four.

• Lowest latency on all 12 test sets. Mean emission latency is 12–109 ms, compared with 194–735 ms for the streaming baselines.
• Best accuracy on 6 test sets. AISHELL-1, AISHELL-3, WenetSpeech Net, GigaSpeech clean, VoxPopuli, and TED-LIUM.
• Lowest overall error rate in both languages. Pooled over all test utterances, X2Streaming-ASR reaches 5.65% CER on Chinese (1.15M characters) and 8.35% WER on English (577K words), ahead of the next best system, Confucius4-R2T2 (quality), at 5.79% and 8.43% with 8–12× higher latency. Pooled rates weight each test set by its size, so the large, harder sets dominate: GigaSpeech is about 70% of the English words (10.01% WER), and the two WenetSpeech sets are about 55% of the Chinese characters. The per-set averages are 4.65% CER and 5.64% WER.
• Adaptive waiting beats fixed lookahead. On AISHELL-1, a fixed lookahead needs 396 ms of mean latency to reach 1.58% CER; X2Streaming-ASR reaches 1.53% at 32 ms.

All systems are evaluated with a single hard commit and no rollback. Latency is measured from the forced-aligned end of each character or word to its emission.

Chinese scores are CER (%). English scores are WER (%). Latency is in milliseconds.

See the paper for complete benchmark settings, baselines, and ablations.

Every 80 ms audio token, the model predicts Wait (keep listening) or Emit (start decoding). After Emit, it writes the text for the new audio and ends with Eos.

X2Streaming-ASR keeps the causal audio encoder, adapter, and decoder-only language model of Voxtral Realtime, but replaces its fixed delay with a learned decision:

1. LISTEN. For each new 80 ms audio token, predict Wait if the context is insufficient or ambiguous, or Emit if it is enough to commit.
2. DECODE. After Emit, generate the text for the newly heard audio until Eos, then return to LISTEN.
3. Reuse history. The encoder runs incrementally and the language model keeps its KV cache, so no chunk triggers recomputation of earlier audio. Within the configured context length, nothing is discarded by a sliding window.

The commit policy is trained in three stages: streaming recognition, a supervised warm start from probed commit times, and GRPO with character-level credit assignment that rewards accuracy first and latency second. See the paper for details.

Linux, Python 3.11, and a CUDA-compatible NVIDIA GPU are recommended.

git clone https://github.com/ZwlinZw/X2StreamingASR.git
cd X2StreamingASR

conda create -n x2streamingasr python=3.11 -y
conda activate x2streamingasr
pip install -r requirements.txt

Install optional vLLM and web-demo dependencies when needed:

pip install -r requirements-vllm.txt -r demo/requirements.txt

Model weights are on Hugging Face: x-square-robot/X2Streaming-ASR-4B-1009.

Download and set MODEL_PATH:

pip install -U huggingface_hub

huggingface-cli download x-square-robot/X2Streaming-ASR-4B-1009 --local-dir ./X2Streaming-ASR-4B-1009
export MODEL_PATH="$(pwd)/X2Streaming-ASR-4B-1009"

Or use any local checkpoint directory:

export MODEL_PATH=/path/to/X2Streaming-ASR-4B-1009

# Adaptive streaming
python infer.py \
  --config configs/infer.yaml \
  --model_path "$MODEL_PATH" \
  --language zh \
  --audio /path/to/audio.wav

# Full-context offline
python infer.py \
  --config configs/infer_offline.yaml \
  --model_path "$MODEL_PATH" \
  --language en \
  --audio /path/to/audio.wav

Each input line must contain one of audio, wav, or wav_path:

{"audio": "/path/to/first.wav"}
{"wav": "/path/to/second.wav"}

python infer.py \
  --config configs/infer.yaml \
  --model_path "$MODEL_PATH" \
  --language en \
  --input_file audio.jsonl \
  --output_file results.jsonl

The output JSONL contains the audio path, recognized text, inference mode, and optional timestamps or adaptive emit records. A reference transcript is not required.

BACKEND=transformers \
MODE=adaptive \
LANGUAGE=zh \
REPO_ID="$MODEL_PATH" \
bash demo/run.sh

Open http://localhost:7860. LANGUAGE sets the default language tag (zh, en, or cs; default zh); use LANGUAGE=en for English speech.

For vLLM, convert and serve the model first:

python tools/convert_hf_to_mistral.py "$MODEL_PATH"

vllm serve "$MODEL_PATH" \
  --host 0.0.0.0 \
  --port 8000 \
  --tokenizer-mode mistral \
  --config-format mistral \
  --enforce-eager \
  --gpu-memory-utilization 0.9

Then launch the demo proxy:

BACKEND=vllm \
MODE=adaptive \
LANGUAGE=zh \
VLLM_HOST=127.0.0.1 \
VLLM_PORT=8000 \
REPO_ID="$MODEL_PATH" \
bash demo/run.sh

If X2Streaming-ASR is useful for your research, please cite:

@article{lin2026x2streaming,
  title={X2Streaming-ASR: wait when uncertain, emit when ready for streaming ASR},
  author={Lin, Zhiwei and Fu, Kaiqi and Wen, Rime and Liu, Zehan and Qin, Shawn and Gan, Roy and Wang, Hao and Wang, Qian},
  journal={arXiv preprint arXiv:2609.08672},
  year={2026}
}

• Source code: Apache License 2.0
• Model weights: see MODEL_LICENSE.md
• Upstream model: mistralai/Voxtral-Mini-4B-Realtime-2602