Speech Meets ELF: Audio Conditional Continuous-Target Diffusion for Speech Recognition and Translation

Official code for "Speech Meets ELF: Audio Conditional Continuous-Target Diffusion for Speech Recognition and Translation".

Speech-to-text (S2T) systems, both recognition (ASR) and translation (S2TT), are dominated by discrete-token decoders. ELF-S2T takes a different route: it performs S2T as audio-conditioned generation in a continuous text-embedding space. Built on the pretrained ELF (Embedded Language Flows) backbone, the whole denoising trajectory stays continuous and the model commits to discrete tokens only at the final step.

To our knowledge, ELF-S2T is the first adaptation of continuous-target language modelling to speech, and the first diffusion-based S2T model to report translation (S2TT) results.

---

Highlights

• 🌊 Continuous-target S2T. Frozen Whisper-large-v3 + a linear projector prepend the audio condition to the noisy text latent, denoised by ELF in one bidirectional stream — no cross-attention.
• 🎙️ Audio forcing. Supervising the unembed on a heavily noised latent (log-SNR centre -0.5 vs ELF's 0.8) makes the text prior uninformative, so the model must read the speech.
• 🔊 Audio guidance (ACFG). Randomly dropping the audio condition in training enables classifier-free guidance (w=2.0) at inference.
• 📈 Scales with capacity. Quality grows with backbone size; ELF-L reaches 5.69 ASR WER and 28.55 S2TT BLEU.
• 🔍 One unified error cause. ASR garbling and S2TT drift both trace to the same close-distance confusion — a near-but-wrong neighbour of the target.

---

Main results

ELF-S2T (ELF-L, 653.4 M): 5.69 WER on LibriSpeech test-clean (lower better), 28.55 BLEU / 54.91 chrF on CoVoST2 de->en test (higher better).

---

Repository layout

ELF-S2T/
├── elf_torch/           # ELF backbone (PyTorch port of the JAX model)
│   ├── model.py             # ELF-B / ELF-M / ELF-L transformer + flow matching
│   ├── t5_encoder.py        # frozen T5-small encoder (continuous target space)
│   ├── convert_jax_to_pt.py # JAX checkpoint -> torch state_dict (offline)
│   └── sampler.py           # SDE / ODE Euler sampler (ELF reference eval)
├── elf_s2t/             # speech adaptation
│   ├── asr_model.py         # ELF-S2T model: Whisper enc + linear projector + ELF
│   ├── data.py              # Kaldi-style manifest + tar streaming loader
│   ├── train.py             # DDP training (flow + unembed dual objective)
│   ├── eval_utils.py        # online WER/BLEU eval + inline SDE/ODE sampler
│   ├── infer.py             # offline inference (DDP, WER / BLEU / chrF)
│   └── align_format.py      # word-level alignment log for error analysis
├── config/              # training recipes (one YAML per model)
├── scripts/             # launch scripts (torchrun DDP) + offline inference
├── tools/               # weight conversion + dataset manifest builders
├── requirements.txt
└── README.md

---

Setup

pip install -r requirements.txt
# optional: pip install swanlab

Tested with Python 3.10 and PyTorch 2.4 (CUDA 12.1). Multi-GPU training uses torch.distributed.run. Scripts read the interpreter from $ENV_PY (defaults to python), so activate your environment or export ENV_PY=/path/to/python.

Weights

Download the ELF backbone weights and convert the JAX checkpoints once (JAX/flax is only needed for this conversion step):

# ELF backbone (pick the sizes you need)
huggingface-cli download embedded-language-flows/ELF-B-owt --local-dir checkpoints/ELF-B-owt
huggingface-cli download embedded-language-flows/ELF-L-owt --local-dir checkpoints/ELF-L-owt
huggingface-cli download embedded-language-flows/t5_small_encoder_jax --local-dir checkpoints/t5_small_encoder_jax

python elf_torch/convert_jax_to_pt.py --checkpoint checkpoints/ELF-B-owt \
    --out checkpoints/elf_b_owt.pt --model ELF-B --vocab_size 32100 --text_encoder_dim 512
python elf_torch/convert_jax_to_pt.py --checkpoint checkpoints/ELF-L-owt \
    --out checkpoints/elf_l_owt.pt --model ELF-L --vocab_size 32100 --text_encoder_dim 512

# T5 encoder pkl -> pt (avoids a JAX import at training time)
python tools/convert_t5_pkl_to_pt.py \
    --pkl checkpoints/t5_small_encoder_jax/t5_small_encoder_jax.pkl

# T5-small tokenizer used to tokenise targets (loaded offline from this dir)
huggingface-cli download google-t5/t5-small \
    spiece.model tokenizer.json tokenizer_config.json --local-dir checkpoints/t5_small

whisper_path defaults to the HF id openai/whisper-large-v3 (auto-downloaded), override with a local path in the config if you have one.

Pretrained checkpoints

The four trained ELF-S2T models are released on the Hugging Face Hub at ssinon/ELF-S2T. If you only want to run inference (not train), download a checkpoint and skip the training step below. The ELF backbone weights / T5 conversion above are NOT needed for inference, each checkpoint already bundles everything.

# e.g. download the best ASR model into the path the infer script expects
huggingface-cli download ssinon/ELF-S2T asr_elf_l.pt --local-dir outputs/asr_elf_l
# -> outputs/asr_elf_l/asr_elf_l.pt

Data

ASR uses LibriSpeech 960h, S2TT uses CoVoST2 de->en (127k pairs). Build the manifests once:

# LibriSpeech 960h (point --libri_dir at your LibriSpeech tars)
python tools/build_manifest.py --libri_dir data/librispeech960 \
    --out data/librispeech960/manifest.jsonl

# CoVoST2 de->en (fixie-ai/covost2 parquet release)
python tools/build_covost2_manifest.py \
    --src data/covost2/de_en --tar_dir data/covost2/de_en_tars \
    --out data/covost2/de_en/manifest.jsonl --prefix covost2_de_en

Then point the libri_dir / manifest fields in the config YAMLs at your data.

---

Training and inference

# ---- ASR (8-GPU DDP by default; trained from the pretrained ELF backbone) ----
./scripts/run_asr_elf_b.sh
./scripts/run_asr_elf_l.sh

# ---- S2TT (warm-started from the trained ASR model of the same backbone) ----
./scripts/run_st_deen_elf_b.sh
./scripts/run_st_deen_elf_l.sh

# ASR on LibriSpeech test-clean (set LIBRI_DIR to your test-clean tars)
CKPT=outputs/asr_elf_l/asr_elf_l.pt ACFG=2.0 STEPS=128 \
    GPUS=0,1,2,3,4,5,6,7 NPROC=8 BS=8 ./scripts/infer_ls_test_clean.sh

# S2TT on CoVoST2 de->en test (BLEU + chrF via sacrebleu)
CKPT=outputs/st_deen_elf_l/st_deen_elf_l.pt \
    SPLIT=covost2-deen-test ACFG=2.0 STEPS=128 \
    GPUS=0,1,2,3,4,5,6,7 NPROC=8 BS=8 ./scripts/infer_covost2_test.sh

Inference knobs are env vars: ACFG=<float> STEPS=<int>.

---

Citation

@misc{li2026speechmeetselfaudio,
      title={Speech Meets ELF: Audio Conditional Continuous-Target Diffusion for Speech Recognition and Translation}, 
      author={Xuanchen Li and Tianrui Wang and Yuheng Lu and Zikang Huang and Yu Jiang and Chenghan Lin and Chenrui Cui and Ziyang Ma and Xingyu Ma and Chunyu Qiang and Guochen Yu and Xie Chen and Longbiao Wang and Jianwu Dang},
      year={2026},
      eprint={2606.10368},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2606.10368}, 
}

ELF-S2T builds on ELF (Embedded Language Flows), please also cite the original:

@misc{hu2026elfembeddedlanguageflows,
      title={ELF: Embedded Language Flows}, 
      author={Keya Hu and Linlu Qiu and Yiyang Lu and Hanhong Zhao and Tianhong Li and Yoon Kim and Jacob Andreas and Kaiming He},
      year={2026},
      eprint={2605.10938},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2605.10938}, 
}

License

MIT, see LICENSE. ELF-S2T is a derivative of the MIT-licensed ELF codebase.