NVIDIA NeMo Speech 3.0

NVIDIA NeMo Speech 3.0 Release Notes

NeMo Speech 3.0 is the first major release after the repo split and rename to NVIDIA-NeMo/Speech. The repo now focuses on ASR, TTS, audio processing, speaker tasks, and SpeechLM. Non-speech Framework, LLM, VLM, diffusion, export/deploy, evaluator components can now be found under separate repos under NVIDIA-NeMo organization.

While this release brings new major features, the central focus is addressing technical debt: removed 800k deprecated LOC, migration to uv package manager for cleaner installs, stronger model test coverage, reduced the number of dependencies, revamped documentation, lighter containers, and AGENTS.md + agentic skills.

Top-Level Changes and Highlights

Breaking Changes and Migration Notes

• LLM/VLM/diffusion/NLP/export/evaluator collections have been moved out to separate repos.
• Unsupported ASR/TTS models and old tutorials were removed. Use active examples under examples/ and docs under docs/source/.
• uv sync reproduces the supported NeMo Speech stack and may replace Python/PyTorch/CUDA inside .venv. To keep your own stack, install PyTorch first, then use uv pip or pip.
• SpeechLM2 compiled acceleration is optional. Use the supported Docker build for TE/FlashAttention/Mamba/grouped-GEMM/DeepEP performance paths.

SpeechLM2 and NeMo Automodel Backend

SpeechLM2 now supports training SpeechLM models with NeMo Automodel through SALMAutomodel, targeting both dense and MoE LLM backbones such as Nemotron 3.

• SALMAutomodel adds an Automodel-backed SALM path with native LoRA, deferred configure_model() initialization, shard-aware distributed loading, and Automodel-owned mesh creation (#15447).
• AutomodelParallelStrategy supports FSDP2, HSDP, TP, CP, and EP, letting SpeechLM training combine sharded data parallelism, tensor/model partitioning, long-context sequence sharding, and MoE expert routing (#15447, #15648, #15679, #15773).
• Optimized backend support now covers Transformer Engine, FlashAttention, Mamba/state-space kernels, grouped GEMM for MoE experts, and DeepEP for expert-parallel all-to-all communication (#15737, #15758, #15679).
• THD packed-sequence training reduces padding waste in variable-length speech batches and is the supported path for CP long-audio training (#15679).
• Long-audio support was expanded with encoder chunking, activation-checkpointing controls, CP-safe data handling, and fixes around dtype/checkpoint restoration (#15648, #15679, #15686, #15716).
• Export and serving improved with vLLM integration, HF/vLLM-ready checkpoint export, backbone-native chat templates, offline HF export, and buffered SALM inference (#15364, #15520, #15623, #15736).
• Data format support now includes ShareGPT conversations, indexed ShareGPT JSONL, and WebDataset-backed ShareGPT conversations (#15316, #15410).

Nemotron VoiceChat Training Modules

NeMo Speech 3.0 adds the train/eval modules behind Nemotron VoiceChat.

• Added Nemotron VoiceChat speech-decoder modules for the response-generation side of the duplex stack (#15066).
• Added DuplexSTT training and inference for generating agent text from user speech plus text context (#15092).
• Added Duplex/EAR-TTS speech-generation pieces used by VoiceChat, including MagpieTTS decoder integration and semantic-codec training support (#15277, #15524).
• Added the joint NemotronVoiceChat STT + TTS class for validation, offline speech-to-speech inference, and export workflows (#15456).
• Added formatter/reproducibility work for the VoiceChat speech decoder, with faster training and half-precision inference support (#15583).
• Expanded Voice Agent examples for Nemotron Nano v2/v3, MagpieTTS, tool calling, audio logging, and README guidance (#14325, #14704, #15243, #15269, #15318, #15547, #15642).
• Hardened Voice Agent behavior around default Parakeet EOU STT, text aggregation, end-of-bot handling, RTVI messages, empty tools, and logging (#14951, #15068, #15069, #15634).

Note: NemotronVoiceChat class is inference/eval only. Train DuplexSTTModel and DuplexEARTTS/speech-decoder modules separately.

ASR

• Unified prompt-model support now covers multilingual ASR and streaming inference (#15666).
• Streaming ASR gained Unified RNNT inference, batched streaming beam search, simulated chunked transducer decoding, streaming speech translation, and Canary streaming policies (#15522, #15753, #15517, #15132, #14765).
• Transducer decoding is faster and more complete: 2.4x speedup with confidence, buffered/cache-aware confidence, fewer redundant decoding passes, lower streaming memory, and WER fixes (#15315, #15765, #15301, #15148).
• Phrase boosting expanded across transducers, cache-aware RNN-T, CTC/RNN-T/TDT GPU phrase boosting, and cache-aware customization (#15125, #15344, #14277, #14757, #14800).
• New and updated model families include prompt Parakeet Hybrid RNNT/CTC, ASR EOU models, Streaming Sortformer, Canary2 with NFA, and ASR Transformer/Conformer-Transformer encoders (#14561, #14740, #14627, #14121, #15661, #15703).
• Diarization/VAD removed the Pyannote dependency and gained more flexible input handling (#15632, #15184).

TTS

• MagpieTTS received a major refresh: decoder model, refactor, longform inference, unified longform/standard paths, and updated model internals (#15277, #15504, #15210, #15241, #15477, #15375, #15031).
• Added CFG distillation, local transformer CFG distillation, and MagpieTTS MoE support (#15568, #15748, #15370).
• Expanded language/text support: Hindi, Japanese, Arabic char tokenizer, Japanese-English katakana, hi-IN/ko-KR/pt-BR IPA, and Japanese G2P accent support (#15320, #15248, #15614, #15567, #15170).
• Improved EasyMagpie zero-shot disabling and speaker-encoder behavior (#15639, #15503, #15564).
• Codec and evaluation work added semantic-codec training, codec conversion/bandwidth extension, HF audio-codec loading, Frechet Codec Distance, parallel scoring, comparison reports, and deterministic multi-GPU eval (#15524, #15191, #15172, #15223, #15417, #15621, #15427).
• Inference paths now support reference-free inference, dataset selection, clearer inference config, and fp16 Whisper loading (#15213, #15212, #15254, #15680).

Audio

• Added and fixed Conformer U-Net speech-enhancement models (#14442, #14626).
• Added a data-prediction objective for flow-matching speech enhancement (#14749).
• Added streaming mode to SpectrogramToAudio (#14524).
• Fixed BNR 2.0 inference alignment with padded input signals (#15388).
• Reduced duplication and applied small fixes in the audio collection (#15587).

DataLoader and Speech Data Tools

• Lhotse support expanded to Parquet/Arrow embedded audio, AIS batch loading, AIS-hosted input configs, non-tarred S3 audio, temperature re-weighting, randomized tarred shard slicing, and in-manifest channel selection (#15303, #15102, #15538, #14891, #15732, #15200, #14558, #14586).
• Audio-codec data loading gained Lhotse training format support plus follow-up fixes (#15622, #15742).
• Online data augmentation now covers clipping, lowpass, lossy codec augmentation, and saving augmented audio from Lhotse samplers/dataloaders (#14809, #14808).
• Speech Data Explorer gained S3 reading, comparison mode, security fixes, and tutorial updates (#15500, #15137).

Docs

• Documentation was rewritten around speech scope, uv installs, ASR/TTS/SpeechLM2 guides, and cleaner navigation.

Repo Rename and 3.0 Release

• Project identity is now NVIDIA NeMo Speech in NVIDIA-NeMo/Speech (#15783, #15788).
• README/docs now describe the speech-focused repo, current status, and nightly docs (#15127, #15217, #15602, #15766).

Technical Debt Reduction

• Removed LLM, VLM, diffusion, NLP, multimodal, ExportDeploy, evaluator, Automodel, and language-modeling paths (#14095, #14192, #14617, #14677, #14790, #14934, #14964, #15033, #15044, #15357, #15378).
• Cleaned unsupported ASR/TTS modules, old tutorials, old buffered CTC scripts, and deprecated SpeechLM1 scripts (#14660, #15061, #15368, #15507, #15655).
• Removed unsafe pickle paths and hardened target/export handling (#14540, #15065, #15232, #15266, #15288, #15629).

Installation, Packaging, and Containers

• Switched to a uv-first source install: uv sync --extra all --extra cu13 (#15769).
• Added clear bring-your-own Python/PyTorch/CUDA guidance via uv pip or pip, addressing common install feedback (#15769).
• Routed optional SpeechLM2 compiled deps through uv-compatible sources and container builds: Transformer Engine, FlashAttention, Mamba, grouped GEMM, DeepEP (#15737, #15758).
• Refreshed Docker around official PyTorch bases, lighter staged builds, CUDA 12/13 support, and CVE dependency updates (#15638, #15747, #15756, #15761).
• Updated CI/publish flows to install from uv.lock and simplify release builds (#15659, #15668, #15685, #15697).

Tests and Release Hardening

• Added functional init/train-step/inference tests for every supported released model (#15433).
• Added golden-value MoE dispatch tests for grouped GEMM/expert routing (#15698).
• Split L0 tests into ASR GPU/CPU and SpeechLM2 buckets (#15654).
• Hardened streaming ASR, Sortformer, MagpieTTS, codec, and TTS L2/e2e tests (#14416, #14417, #14435, #14823, #15272, #15508, #15584, #15607).
• Refactored release CI and moved to AWS ephemeral runners (#15620, #15668, #15718).

Detailed PR Breakdown

Release Identity and Docs

• #15788 releases r3.0.0; #15783 bumps the next release to 3.0.
• #15127 renames docs/title to NVIDIA NeMo Speech; #15217 and #15602 update repo status/release messaging.
• #15363 and #15460 restructure docs; #15542 refactors ASR docs; #15647 refactors diarization docs; #15745 improves docs visuals.
• #15769 rewrites install docs for uv, bring-your-own stacks, CUDA extras, and pip fallback.
• #15773 documents SpeechLM parallelism; #15738 adds prompt RNNT docs; #15546/#15302 add MagpieTTS docs; #15526 adds streaming ASR inference docs.
• #15744 adds community featured models; #15766 updates the release README.

Technical-Debt Cleanup

• #15378 removes deprecated collections; #15363 starts docs-side transformation.
• #14095, #14677, #14934, #14617, and #14192 remove NeMo1, multimodal/vision, NLP, and language-modeling code.
• #15357, #14790, #15033, #15044, and #14964 remove deprecated LLM/VLM/diffusion tutorials, export-deploy, old Automodel, and evaluator code.
• #15507 cleans old ASR modules; #15368 removes unsupported TTS models/docs; #14660 removes old TTS tutorials; #15061 removes old buffered CTC; #15655 removes SpeechLM1 scripts.
• #15147 updates TRT references; #15146 removes old checkpoint save support; #15211 removes torchaudio usage.
• #15687 applies black formatting; #15656 makes formatting CI check-only with pre-commit hooks.

Install, Packaging, Docker, Dependencies

• #15769 adds uv-first install plus pip/uv-pip fallback for existing PyTorch/CUDA stacks.
• #15734 removes stale uv-base deps; #15758 moves DeepEP/TE git refs to uv sources; #15697 installs CI deps from uv.lock.
• #15638 switches Docker to official PyTorch base; #15747 adds CUDA 12 builds; #15761 moves code copy to the final Docker stage.
• #15737 builds Automodel compiled deps in CI image; #15756 updates CVE-sensitive deps and Dockerfile.
• #15659 unblocks TestPyPI publish; #15668 refactors release workflows; #15685 removes obsolete build workflows.
• #15314 sets torch >= 2.6.0; #15353 documents TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD; #15506/#15018 fix CUDA Python/numba-cuda handling.
• #15630 removes diskcache; #15502 removes protobuf; #15183 removes CUDA bindings pins; #15438 relaxes kaldialign.

Security and Safety

• #15288 fixes command injection and insecure permissions.
• #14540 hardens target resolution.
• #15065 guards trust_remote_code.
• #15266, #15232, and #15629 remove or block unsafe pickle paths.
• #15636 adds SECURITY.md.
• #15500 adds SDE S3/comparison updates plus a security fix.

Tests, CI, Release Reliability

• #15433 adds init/train-step/inference functional tests for released models.
• #15698 adds golden MoE dispatch tests.
• #15654 splits L0 ASR/SpeechLM2 test buckets; #15651 raises GPU speech test timeouts.
• #14823, #14953, and #14184 harden streaming/cache-aware transducer and CUDA-graph tests.
• #14416, #14417, and #14435 complete Streaming Sortformer fixes/tests.
• #15508, #15584, #15272, and #15607 harden TTS/MagpieTTS/codec tests.
• #15620, #15718, #15484, #15519, and #15501 improve CI runners, branch rules, triggering, and e2e layout.
• #15613 adds local dev/PR babysitting support; #15612 adds distributed-log debug support.

SpeechLM2, SALMAutomodel, Scaling, vLLM

• #15447 adds SALM + NeMo Automodel for Nemotron Nano V3, including native LoRA and Automodel-managed distributed setup.
• #15648 expands SALMAutomodel with long-context support, encoder chunking, activation-checkpointing controls, and stability fixes.
• #15679 adds THD packed sequences, context parallelism, CP-safe long-audio training, and related packed-input handling.
• #15773 documents FSDP2, HSDP, TP, PP, CP, and EP strategy usage for SpeechLM training.
• #15737 and #15758 wire compiled Automodel deps, DeepEP, and Transformer Engine into reproducible uv/container builds.
• #15520 adds vLLM support for NeMo SpeechLM, including a registered SpeechLM model/config path.
• #15623 exports vLLM-ready SpeechLM checkpoints with backbone-native chat_template support.
• #15716 adds encoder input chunking for SALM vLLM inference.
• #15736 adds offline HF export for SpeechLM2 checkpoints.
• #15686 preserves perception checkpoint dtype; #15570 fixes Qwen3 SALM LoRA initialization.
• #15316 adds TDT decoder input, ShareGPT format support, and related SALM improvements.
• #15410 adds indexed ShareGPT JSONL and WebDataset formats.
• #15364 adds buffered SALM inference; #15478 updates SpeechLM2 docs and dataloader resampling; #15281 adds multimodal offset-key support.

Nemotron VoiceChat, Duplex Models, Voice Agent

• #15066 implements the Nemotron VoiceChat speech decoder for the response-generation side of duplex speech-to-speech.
• #15092 implements DuplexSTT for speech/text-conditioned agent text generation.
• #15456 adds the joint Nemotron VoiceChat STT + TTS class for validation, offline inference, and export.
• #15583 improves VoiceChat decoder reproducibility, training speed, formatting, and fp16 inference.
• #15277 adds the MagpieTTS decoder model used by the speech-generation stack.
• #15524 adds semantic-codec training support.
• #14325 adds NeMo Voice Agent as the app/integration layer.
• #14704 adds Nemotron Nano v2 support; #15318 adds Nemotron Nano v3 and MagpieTTS improvements.
• #15069 sets Parakeet EOU as default STT for the voice agent.
• #14951 fixes text aggregation, end-of-bot handling, and logging; #15068 fixes missing RTVI bot messages.
• #15243 and #15269 improve tool-calling examples, tool-calling behavior, and logging UX.
• #15634 fixes empty tools; #15642 updates Voice Agent docs.

ASR, Streaming, Prompt Models, Transducers

• #15666 adds unified prompt-model support for multilingual ASR and streaming; #14561 adds Parakeet hybrid prompt support; #15738 adds docs.
• #15522 adds Unified RNNT streaming inference.
• #15753 adds batched streaming beam search; #15315 speeds up transducer decoding 2.4x with confidence; #15765 adds buffered/cache-aware confidence.
• #15517 adds simulated chunked transducer decoding; #15301 removes double decoding; #15148 fixes streaming transducer memory/WER.
• #15125, #15344, #14277, #14800, and #14757 expand phrase boosting and cache-aware customization.
• #15132 adds streaming speech translation; #14765 adds Canary Wait-K/AlignAtt; #14766 adds streaming timestamps.
• #15268 adds Canary timestamp-model restore; #15291 fixes tensor-input timestamps.
• #14740, #15462, #15497, and #15493 add EOU models/metrics and fixes.
• #14627, #14388, #14416, #14417, and #14435 cover Streaming Sortformer train/inference/docs/tests.
• #14905, #14932, and #15025 update Multi-Talker Parakeet streaming docs/notebooks; #14121 adds Canary2 with NFA; #15317 improves short-audio Canary.
• #15632 removes Pyannote from diarization/VAD; #15184 improves diarization inputs; #15573 aligns cpWER with meeteval.
• #15605 removes tail-margin algorithms; #15591 removes PnC flag; #15576 fixes ASR inference pipeline bugs.
• #15661 adds ASR Transformer encoder; #15703 adds Conformer I/O-styled Transformer encoder.

TTS, MagpieTTS, Codecs, Text

• #15277 adds MagpieTTS decoder; #15504 refactors MagpieTTS; #15031 updates the model.
• #15210, #15241, #15477, and #15375 add/unify longform MagpieTTS inference.
• #15568 and #15748 add CFG distillation; #15370 adds MagpieTTS MoE.
• #15320, #15248, #15567, #15614, and #15170 add Hindi/Japanese/IPA/Arabic/Katakana/G2P support.
• #15639, #15503, and #15564 adjust EasyMagpie zero-shot disabling.
• #15191, #15172, #15622, and #15742 improve codec conversion, bandwidth extension, HF codec loading, and Lhotse codec data.
• #15524 adds semantic-codec training; #15223 adds Frechet Codec Distance; #15417 adds parallel scoring; #15621 adds TTS comparison reports.
• #15348 and #15427 improve multi-validation and deterministic multi-GPU eval.
• #15213, #15212, #15254, and #15680 improve reference-free inference, dataset selection, inference config, and fp16 Whisper loading.

Audio

• #14442 adds a Conformer U-Net model for speech enhancement.
• #14626 fixes the Conformer U-Net implementation.
• #14749 adds a data-prediction objective for flow-matching speech-enhancement models.
• #14524 adds streaming mode to SpectrogramToAudio.
• #15388 fixes BNR 2.0 inference alignment with padded input signals.
• #15587 reduces duplication in the audio collection and applies small fixes.

DataLoader and Speech Data Tools

• #15303 adds Parquet/Arrow embedded-audio support via Lhotse.
• #15102 and #15538 add AIS batch loading for ASR audio processing and LhotseSpeechToTextBpeDataset.
• #14891 supports input_cfg.yaml directly from AIS buckets.
• #15732 fixes non-tarred S3 audio in LazyNeMoIterator.
• #15200 adds on-the-fly temperature re-weighting for Lhotse datasets.
• #14558 adds randomized shard slicing for tarred data.
• #14586 adds in-manifest channel selection for multichannel recordings through Lhotse.
• #15316 and #15410 add ShareGPT, indexed ShareGPT JSONL, and WebDataset data formats for SpeechLM2.
• #14294 adds ShareGPT data and a test loader.
• #15622 adds audio-codec Lhotse training format support; #15742 updates and fixes audio-codec Lhotse loading.
• #14809 adds clipping, lowpass, and lossy-codec online augmentations; #14808 adds augmented-audio saving.
• #15500 updates Speech Data Explorer with S3 read support, comparison mode, and security fixes; #15137 updates the SDE tutorial.

Smaller Fixes

• #15437 adds .nemo torchaudio-preprocessor checkpoint migration.
• #15461 auto-detects use_bucketing and validates batch size.
• #15325 fixes TDT fusion scores; #15322 moves transducer fusion models to the base class.
• #15245 speeds .nemo tar extraction/config processing.
• #15409 handles an exp-manager timer race.
• #15238 uses PurePosixPath.
• #15444 replaces assert with ValueError in EncDecMultiTaskModel.
• #15688 fixes validation audio logging; #15699 adds UTMOSv2Calculator.verbose.