v0.27.0rc1

Highlights

This release candidate features 104 merged changes from 52 contributors, including 11 new contributors.

vLLM-Omni v0.27.0rc1 focuses on four major areas: 1) productionizing MiniMax H3 across serving modes, accelerators, quantization paths, and memory-constrained deployments, 2) introducing PersonaPlex full-duplex speech-to-speech serving, 3) advancing scheduler-managed diffusion execution and distributed layerwise offload, and 4) delivering substantial TTS and MiniCPM-o 4.5 performance improvements. The release candidate also rebases the project onto vLLM 0.27.0, adds batched Chat Completions, expands diffusion and video-model coverage, and strengthens CI, documentation, and hardware portability.

Key Improvements

• Expanded MiniMax H3 into a broadly deployable video-and-audio generation stack, with a modular pipeline, aligned official input modes, quality grading through dynamic Cache-DiT loading, TeaCache validation, FP8 and INT8 quantization, fused attention and normalization paths, and optimized distributed execution. (#5720, #5752, #5853, #5840, #5737, #5706, #5801)
• Extended MiniMax H3 hardware coverage across NVIDIA RTX 4090/5090, DGX Spark GB10, RTX PRO 6000, AMD MI350X, Ascend NPU, and Moore Threads MUSA, with platform-specific recipes and correctness fixes. (#5764, #5946, #5863, #5723, #5896, #5837, #5881)
• Added PersonaPlex, a native vLLM port of the Moshi-based full-duplex speech-to-speech model, including duplex serving support for realtime conversational workloads. (#4771)
• Advanced scalable diffusion execution with DLO data-parallel concurrency, scheduler-managed paged KV-cache worker contracts, modular MiniMax H3 support, distilled diffusion LoRA support, and scheduler admission-policy cleanup. (#5864, #5550, #5720, #2783, #5843)
• Improved TTS and speech-generation performance through cached incremental decoding for Qwen3-TTS, CUDA Graph acceleration and batched codec sampling for MiniCPM-o 4.5, TensorRT acceleration for its vocoder, and hot-path optimizations for Voxtral TTS and Step-Audio2. (#5202, #5869, #5792, #5638, #5175, #5067)

Core Architecture & Runtime

• Rebased the project onto vLLM 0.27.0, aligning the engine, scheduler, model-runner, attention, deployment, and platform layers with the vLLM 0.27 release line. (#5976)
• Removed duplicated autoregressive and generation scheduler plumbing and established explicit shared scheduler lifecycle contracts. (#5461)
• Refactored diffusion request admission waiting to clarify scheduler behavior and improve concurrent request handling. (#5843)
• Added worker contracts and RPC plumbing for scheduler-managed paged KV cache, preparing diffusion pipelines such as HunyuanImage3 for scheduler-controlled KV allocation and reuse. (#5550, #5541)
• Added the first Cosmos3 session-memory port for UND text key/value state, continuing the stateful world-model execution work introduced in the previous release. (#4657)
• Read engine arguments directly from VllmOmniConfig, further consolidating pipeline, stage, and engine configuration behind the unified configuration model. (#5678)
• Improved per-replica fault isolation so a single stage failure does not automatically terminate the API server or unrelated replicas. (#4583)
• Removed the legacy --stage-configs-path serve option after completing the migration to registry-backed deployment configuration. (#5647)

Model Support

• Expanded MiniMax H3 support with a modular diffusion pipeline, the official input matrix, quality-grading requests, distilled four-step schedules on NPU, and stricter encoder checkpoint validation. (#5720, #5752, #5853, #5991, #5824)
• Added PersonaPlex, a Moshi-based full-duplex speech-to-speech model with native vLLM execution and duplex serving. (#4771)
• Added LongCat-Video-Avatar-1.5 audio-image-to-video and audio-text-to-video generation. (#4099)
• Added LingBot-Video text-to-image and text-image-to-video generation modes. (#5311)
• Added distilled LoRA support for diffusion models. (#2783)
• Added a Ming Flash Omni TTS adapter and derived TTS model detection from adapter metadata rather than hard-coded model checks. (#5746, #5682)
• Added thinker-only ModelOpt NVFP4 W4A4 checkpoint support for Qwen2.5-Omni. (#5073)
• Improved MOSS-TTS codec v1/v2 detection and synchronized the vendored tokenizer behavior with upstream. (#5635)
• Added LoRA-request handling for non-diffusion models. (#5374)

MiniMax H3 Productionization

• Added online FP8 support and Ascend RainFusion attention with INT8 online quantization. (#5737, #5706)
• Added fused RMSNorm and RoPE optimizations, refined TensorRT-LLM attention support, and restored dynamic RoPE fusion on MUSA. (#5801, #5779, #5881)
• Enabled DLO deployments on RTX 4090 and RTX 5090 and added multi-GPU performance coverage for 4xH100 configurations. (#5764, #5836)
• Added TeaCache support and Cache-DiT validation, including dynamic loading and unloading for quality-grading requests. (#5840, #5853)
• Added packed variable-length attention on NPU to avoid quadratic attention-mask materialization. (#5891)
• Added support for arbitrary video shapes in RainFusion attention. (#6000)
• Improved portability by making conditioned-VAE RNG handling device-aware on MUSA and NPU. (#5703, #5837)
• Added accuracy coverage for T2VA, I2VA, and Ref2VA, including FP8 validation and aligned official reference inputs. (#5709, #5829, #5978)
• Added deployment recipes for ROCm MI350X, DGX Spark GB10, RTX PRO 6000, and RTX 4090. (#5723, #5946, #5863, #5850)

Audio, Speech & Omni Production Optimization

• Added cached incremental decoding for Qwen3-TTS, reducing repeated work during autoregressive generation. (#5202)
• Improved MiniCPM-o 4.5 by reusing FlashAttention unpadding metadata, optimizing Whisper chunk-attention mask construction, batching Talker codec sampling, and adding CUDA Graph support to HiFTGenerator. (#5165, #5382, #5792, #5869)
• Added TensorRT acceleration for the MiniCPM-o Code2Wav vocoder, covering both the DiT estimator and CampPlus components. (#5638)
• Optimized the Voxtral TTS hot path by removing unnecessary host synchronization, caching kernels, and selectively opting out of CUDA Graph execution where appropriate. (#5175)
• Kept Step-Audio2 streaming tokens on device and batched audio-feature length synchronization to reduce device-to-host overhead. (#5067)
• Fixed Qwen3-TTS short-sequence RoPE behavior on NPU by falling back to the supported BSND path. (#5608)
• Improved TTS CI stability by allowing expected branch-count decreases without incorrectly failing the ratchet. (#6008)

Diffusion, Image & Video Generation

• Added MiniMax H3 modular-pipeline support and integrated it with Cache-DiT, TeaCache, DLO, quantization, and multiple attention backends. (#5720, #5840, #5853)
• Fixed DLO AllGather sizing for models with heterogeneous parameter layouts and corrected concurrent data-parallel request execution. (#5802, #5864)
• Documented and strengthened distributed layerwise-offload compatibility across supported execution modes. (#5839)
• Added scheduler-managed paged-KV-cache foundations for HunyuanImage3 through scheduler preparation, worker contracts, and RPC plumbing. (#5541, #5550)
• Added distilled diffusion LoRA support. (#2783)
• Added LongCat-Video-Avatar-1.5 AI2V/AT2V and LingBot-Video T2I/TI2V generation modes. (#4099, #5311)
• Avoided unnecessary attention masks for FLUX.2 and HunyuanVideo 1.5 when the non-varlen path is sufficient. (#4645)
• Fixed BAGEL multimodal RoPE position IDs, KV-only payload transfer behavior, and shared-memory connector coverage. (#5775, #5744, #5898)
• Restored packed-mask-free capability reporting for the TeaCache fake backend. (#5997)
• Normalized NumPy image outputs before saving and corrected image-count validation. (#6031, #5838)

Quantization, Attention & Memory Efficiency

• Added MiniMax H3 online FP8 and NPU INT8 quantization paths. (#5737, #5706)
• Added Qwen2.5-Omni thinker-only ModelOpt NVFP4 W4A4 checkpoint support. (#5073)
• Enabled ModelOpt FP8 inference for Qwen3-Omni on MUSA. (#5671)
• Refreshed FlashInfer attention integration and added Blackwell quantized-attention support for QK16/V8 configurations. (#5344)
• Added GQA and MQA support to the ring-attention SDPA path. (#5255)
• Bounded memory usage during video-frame conversion instead of retaining all converted frames at once. (#5732)
• Added packed variable-length attention for MiniMax H3 on NPU, avoiding quadratic mask materialization for supported workloads. (#5891)

Serving, Frontend & API Behavior

• Added batched Chat Completions, allowing multiple independent chat requests to be submitted through a single frontend request. (#5317)
• Added ComfyUI support for reference-to-video generation, with MiniMax H3 as the initial example. (#5756)
• Added native duplex serving for PersonaPlex. (#4771)
• Improved asynchronous Omni output documentation and clarified when output artifacts are materialized. (#5610)
• Switched reference-video decoding to the vLLM video loader for consistent input handling across serving paths. (#5085)
• Fixed image-count validation and normalized NumPy-backed image outputs before persistence. (#5838, #6031)

Platforms, Distributed Execution & Hardware Coverage

• Expanded MiniMax H3 deployment support across NVIDIA H100, RTX 4090/5090, RTX PRO 6000, DGX Spark GB10, AMD MI350X, Ascend NPU, and Moore Threads MUSA. (#5836, #5764, #5863, #5946, #5723, #5706, #5881)
• Added RainFusion attention, INT8 online quantization, packed var...

v0.26.0

Highlights

This release features 252 merged changes from 96 contributors, including 35 new contributors.

vLLM-Omni v0.26.0 is led by three major additions: 1) MiniMax H3 joint video/audio generation, 2) an experimental full-duplex realtime runtime for MiniCPM-o 4.5, and 3) distributed layerwise diffusion offload with DP multi-concurrency and mmap-backed weight loading. The release also aligns with vLLM 0.26, broadens model and hardware coverage, and advances interactive streaming, diffusion parallelism, TTS performance, quantization, and runtime architecture.

Key Improvements

• Added MiniMax H3 joint video/audio generation, supporting T2VA, first-frame-to-video+audio (FL2VA), and image/audio or multi-video reference-to-video+audio (Ref2VA) through OpenAI-compatible /v1/videos serving. (#5691) NPU and ROCm are also supported in Day-0.
• Introduced an experimental full-duplex realtime runtime for MiniCPM-o 4.5, with native and Realtime-compatible WebSocket paths, streaming audio input/output, cancel and barge-in handling, overlap policy, and playback-aware session state. (#3907)
• Added distributed layerwise offload for multi-device diffusion deployments, combining sharded mmap-backed weight loading, AllGather reconstruction, double-buffered prefetch, and DP multi-concurrency. On the PR's Ascend 910B3 Cosmos3-Nano DP4 workload, cgroup peak memory fell from 178 GB to 47 GB while four requests ran concurrently. (#5397)
• Scalable diffusion serving and performance improvement, with midway prompt updates, asynchronous image outputs, OmniGen2 sequence parallelism, FLUX CFG/VAE parallelism, AllGather-KV attention, TensorRT-LLM diffusion attention, and official LTX multimodal guidance. (#4652, #4981, #3206, #2281, #5292, #4968, #5283, #5148)

Core Architecture & Runtime

• Rebased the project onto vLLM 0.26.0, aligning engine, scheduler, model-runner, deployment, and platform integration with the vLLM 0.26 release line. (#5443)
• Added Phase 1 of composable parallel strategy overlays. An opt-in strategy.yaml can describe TP, DP, PP, EP, and stage-replica degrees while preserving existing deploy configurations and CLI precedence; TP and stage replicas received live end-to-end validation in this phase. (#4281)
• Added the experimental full-duplex runtime and extended it to three-stage MiniCPM-o serving, with session-scoped state and data-plane support for streaming, cancellation, barge-in, overlap handling, and playback acknowledgement. (#3907, #5380, #5613)
• Added the first session-state foundation for autoregressive diffusion world models, then generalized KV-session and prefetch-job capabilities for stateful world-model execution. (#4487, #5271, #4941)
• Migrated legacy stage configurations to the pipeline registry, surfaced stage metadata through VllmOmniConfig, and improved generated deployment configuration. (#5031, #4818, #5343, #5449)
• Modernized multimodal output and diffusion runtime boundaries through payload metadata, MultimodalPayload, output-streaming cleanup, and narrower executor, scheduler, worker, model-runner, connector, and metrics interfaces. (#4922, #4980, #5166, #5215, #5216, #5217, #5218, #5096, #5168)

Model Support

• Added MiniMaxAI/MiniMax-H3 support for joint video and synchronized audio generation. The new pipeline covers T2VA, FL2VA, and Ref2VA workloads and supports synchronous or asynchronous OpenAI-compatible video serving. (#5691)
• Added Krea 2 text-to-image support and OmniGen2 sequence parallelism. (#4730, #3206)
• Expanded world-model and video coverage with Cosmos3 Edge and Distilled, LingBot Video dense and MoE, and MammothModa2-Dev. (#5001, #5035, #5411)
• Added Boogu Image 0.1 Base and Edit and Nemotron Audex support. (#4995, #4976)
• Expanded MiniCPM-o 4.5 with offline and online examples, full-duplex serving, end-to-end coverage, multimodal encoder and startup-memory optimizations, and Ascend NPU talker/vocoder support. (#5222, #3907, #5237, #5188, #5447, #5117)
• Added the MOSS-TTS-Local v1.5 vocoder graph and Fish Speech TTS inference on XPU. (#4929, #4856)
• Fixed model integration and loading across Qwen3-Omni, Ming, SoulX-Singer, HunyuanImage3, and remote-code model paths. (#5086, #5191, #5243, #5210, #5006, #5213)

Audio, Speech & Omni Production Optimization

• Reduced Ming-TTS streaming time to first packet with an initial latent chunk, accelerated its Stage-0 flow head with fused RoPE/QKV plus optional torch.compile and piecewise CUDA Graphs, and cached uploaded speaker embeddings in Stage 0. (#5011, #4942, #5240, #5351)
• Improved Qwen3-TTS throughput and streaming by avoiding per-step hidden-state device-to-host transfers, restoring batched MTP sampling, aligning CUDA Graph capture with async output, caching decoder masks, compiling the pre-transformer, and adding codec chunk ramp-up. (#4879, #4970, #4923, #5107, #5152)
• Improved MOSS-TTS-Local v1.5 with a vocoder graph, dynamic-batch streaming sessions, CUDA Graph-compatible talker execution, and compile support for the talker and codec. (#4929, #5235, #5197, #5530)
• Strengthened MiniCPM-o speech correctness and memory behavior across audio placeholder handling, non-async output, three-stage full-duplex serving, vocoder headroom, implicit connector resolution, and single-GPU generation. (#5116, #5455, #5380, #5621, #5533, #5637)
• Fixed request-local MiMo Audio speech-code routing, VoxCPM2 chunked-prefill length handling, Qwen3-Omni speaker metadata, and chat-completion audio format propagation. (#5070, #5416, #5086, #4718)
• Optimized Qwen3-TTS for Ascend 310P and kept the HiFT vocoder resident on NPU. (#4841, #5242)

Diffusion, Image & Video Generation

• Added distributed layerwise offload for multi-NPU diffusion deployments. Parameter sharding, mmap-backed loading, AllGather reconstruction, two-slot prefetch, and DP multi-concurrency reduce host-memory replication and enable large Cosmos3 workloads that do not fit as fully resident per-rank models. (#5397)
• Expanded distributed diffusion execution with CFG parallelism for FLUX.1-Kontext-dev, VAE patch parallelism for FLUX.2-dev, AllGather-KV sequence-parallel attention, and a TensorRT-LLM diffusion attention backend with Skip-Softmax. (#2281, #5292, #4968, #5283)
• Added layerwise CPU offload for FLUX.2-dev, component/layerwise CPU offload for Cosmos3, configurable diffusion compile granularity, and Cache-DiT support for FLUX.1-Kontext-dev. (#5256, #4695, #4603, #4205)
• Added midway prompt updates for chunked streaming video generation and asynchronous artifact output for diffusion image models. (#4652, #4981)
• Unified the LTX-2 and LTX-2.3 runtime, aligned one-stage multimodal guidance and numerical behavior with the official Lightricks implementation, and generalized CFG parallelism across the guidance plan. (#5147, #5148, #5547)
• Expanded Cosmos3 with Edge/Distilled checkpoints and presets, CPU-offload paths, NPU recipes, ModelOpt FP8 loading, and scheduler/configuration fixes. (#5001, #5313, #5596, #4695, #4978, #5097, #5076, #5176)
• Improved diffusion serving correctness around CFG companion dispatch, explicit guidance_scale=0, MagCache residual application, BAGEL CFG KV transfer, Wan2.2 guidance resolution, video tensor layout, offline Hub behavior, and image output metadata. (#5482, #4999, #5561, #5620, #5615, #5418, #5403, #5619)

Quantization & Memory Efficiency

• Added BitsAndBytes W4 online quantization for diffusion transformers and Transformer Engine online FP8 for the FLUX.2-dev Mistral text-encoder component. (#5037, #5136)
• Improved ModelOpt checkpoint compatibility with Cosmos3 FP8 loading and NVFP4 scale-tensor remapping. (#5076, #5087)
• Added packed-parameter support with HSDP and fixed component quantization initialization. (#5088, #5103)
• Fixed Quack FP8 GEMM behavior under inference_mode() plus torch.compile() and skipped the path when scale tensors are not populated. (#5153, #5262)
• Moved GGUF diffusion-model support from vLLM-Omni core to the out-of-tree vllm-project/vllm-gguf-plugin. (#4769)

Serving, Frontend & API Behavior

• Added native /v1/duplex and Realtime-compatible /v1/realtime?duplex=1 WebSocket entrypoints for the experimental MiniCPM-o 4.5 full-duplex runtime. (#3907)
• Added session.prompt_update steering for streaming video generation and asynchronous output for diffusion image generation. (#4652, #4981)
• Improved realtime lifecycle handling by allowing input commit without closing the speech WebSocket, stopping disconnected sessions, flushing connector tails, and making runtime-control payloads serializable. (#5517, #5388, #5414, #5613)
• Returned diffusion metrics from /v1/images/generations, corrected per-stage timing/token statistics, and normalized diffusion request extras in chat serving. (#5278, #4974, #5171)
• Improved API validation by rejecting unsupported output modalities, preserving caller sampling parameters, honoring explicit guidance_scale=0, and rejecting incompatible completions requests for thinker+talker models when remote code is unavailable. (#4720, #4115, #4999, #5525)

Platforms, Distributed Execution & Hardware Coverage

• Added the distributed layerwise offload path validated on Ascend 910B3, and aligned Ascend NPU integration and CI with vLLM 0.26. (#5397, #5490)
• Expanded Ascend model coverage and recipes for MiniCPM-o 4.5, Qwen3-TTS, Cosmos3 Nano/Super, Qwen3-Omni, Wan2.2, Qwen-Image-Edit, and HunyuanImage3. (#5117, #4841, #4978, #5097, #5339, #5167, #5436)
• Added CosyVoice3 support on Moore Threads MUSA. (#5164)
• Added Fish Speech TTS on XPU and deli...

v0.26.0rc1

Highlights

This release candidate features 145 merged pull requests from 71 contributors, including 21 new contributors.

vLLM-Omni v0.26.0rc1 aligns the project with the vLLM 0.26 release line and delivers major improvements across streaming and realtime multimodal serving, diffusion and world-model execution, TTS performance, model coverage, quantization, hardware acceleration, runtime architecture, and CI infrastructure.

This release introduces a full-duplex realtime runtime with a MiniCPM-o 4.5 demo, midway prompt updates for streaming video generation, and session-state infrastructure for autoregressive diffusion world models. It expands model support with Cosmos3 Edge and Distilled checkpoints, LingBot Video, Boogu Image, MammothModa2-Dev, MOSS-TTS-Local v1.5, and broader MiniCPM-o 4.5 integration. It also adds new diffusion parallelism, offloading, attention, compilation, and caching capabilities while substantially improving Ming-TTS, Qwen3-TTS, MOSS-TTS, Qwen3-Omni, and MiniCPM-o performance and correctness.

Key Improvements

• Aligned with the vLLM 0.26 release line, including the vLLM 0.26.0 rebase and compatibility updates across XPU, NPU, CUDA, ROCm, diffusion FusedMoE, deployment configuration, and CI images. (#5443, #5066, #5115, #5167)
• Introduced full-duplex and interactive streaming capabilities, adding a full-duplex realtime runtime, a MiniCPM-o 4.5 demo, midway prompt updates for streaming video generation, and improved realtime session lifecycle handling. (#3907, #4652, #5383, #5388)
• Expanded diffusion and world-model support, adding Cosmos3 Edge and Distilled checkpoints, LingBot Video dense and MoE variants, Boogu Image, MammothModa2-Dev, and session-state infrastructure for autoregressive diffusion world models. (#5001, #5035, #4995, #5411, #4487)
• Improved diffusion performance and scalability, adding CFG parallelism for FLUX Kontext, VAE patch parallelism and layerwise CPU offload for FLUX.2-dev, AllGather-KV sequence-parallel attention, TensorRT-LLM diffusion attention, configurable compile granularity, and Cache-DiT support. (#2281, #5292, #5256, #4968, #5283, #4603, #4205)
• Accelerated TTS streaming and generation, reducing Ming-TTS time to first packet, optimizing its Stage-0 flow head and speaker caching, improving Qwen3-TTS compilation and chunk scheduling, and enabling CUDA Graph execution for MOSS-TTS-Local v1.5. (#5011, #4942, #5240, #5351, #5107, #5152, #5197)
• Strengthened multimodal and speech correctness, fixing Qwen3-Omni speaker metadata and audio-video cache behavior, MiniCPM-o audio output and placeholder handling, realtime connector flushing, audio format propagation, and per-request speech-code routing. (#5086, #5308, #5116, #5455, #5414, #4718, #5070)
• Expanded quantization capabilities, adding BitsAndBytes W4 for diffusion transformers, Transformer Engine FP8 for FLUX.2-dev Mistral, Cosmos3 ModelOpt FP8 loading, NVFP4 scale remapping, and HSDP packed-parameter support. (#5037, #5136, #5076, #5087, #5088)
• Modernized core runtime and pipeline architecture, migrating legacy stage configurations to the pipeline registry, moving diffusion outputs to payload metadata, cleaning up diffusion executor, scheduler, worker, and model-runner interfaces, and consolidating connector and metrics infrastructure. (#5031, #4922, #4980, #5215, #5216, #5217, #5218, #5096, #5168)

Core Architecture & Runtime

• Rebasing the project onto vLLM 0.26.0 aligned the runtime with the latest vLLM release line. (#5443)
• Added a session-state manager for autoregressive diffusion world models, establishing the first phase of persistent world-model sessions. (#4487)
• Generalized autoregressive diffusion KV-session capabilities and added support for prefetched KV jobs in the model runner. (#5271, #4941)
• Added the full-duplex realtime runtime and a MiniCPM-o 4.5 demonstration application. (#3907)
• Migrated legacy stage configurations to the pipeline registry and made stage metadata available through VllmOmniConfig. (#5031, #5343)
• Updated configuration construction to return VllmOmniConfig and fixed missing sequence_parallel_size values in generated deployment configurations. (#4818, #5449)
• Refactored diffusion outputs around payload metadata and moved multimodal payload accumulation into MultimodalPayload. (#4922, #4980)
• Refactored diffusion output streaming and cleaned up executor, scheduler, worker, and model-runner interfaces. (#5166, #5215, #5216, #5217, #5218)
• Cleaned up Omni connector runtime and transfer backends and fixed terminal async-chunk processor flushing. (#5096, #5414)
• Centralized metrics helpers and split the benchmark serving CLI into more focused modules. (#5168, #5206)
• Removed obsolete scheduler, diffusion-hook, model-loader, configuration, and example code. (#5312, #5270, #5199, #5201, #5335)

Model Support

• Added support for Cosmos3 Edge and Cosmos3 Distilled checkpoints, together with an Edge deployment recipe and updated scheduler behavior. (#5001, #5313, #5176)
• Added dense and MoE support for LingBot Video. (#5035)
• Added support for Boogu Image 0.1 Base and Boogu Image 0.1 Edit. (#4995)
• Added support for MammothModa2-Dev and migrated its examples to the shared x_to_text.py interface. (#5411, #5454)
• Added support for the MOSS-TTS-Local v1.5 vocoder graph. (#4929)
• Expanded MiniCPM-o 4.5 support with offline and online examples, end-to-end tests, Ascend NPU talker and vocoder support, and multiple memory and performance optimizations. (#5222, #5237, #5117, #5130, #5188, #5228, #5447)
• Added Fish Speech TTS inference support on XPU. (#4856)
• Added a Cosmos3 Nano recipe for Ascend NPU hardware. (#4978)
• Fixed SoulX-Singer pipeline registration and several remote-code loading paths. (#5210, #5006, #5213)
• Fixed Ming processor registration with Transformers 5.x and newer. (#5243)

Audio, Speech & TTS Serving

• Reduced Ming-TTS streaming time to first packet by emitting an initial latent chunk earlier. (#5011)
• Accelerated the Ming-TTS Stage-0 flow head with fused RoPE and QKV operations, optional torch.compile, and piecewise CUDA Graph execution. (#4942)
• Added speaker-embedding caching for uploaded Ming-TTS reference audio and moved speaker extraction and caching into Stage 0. (#5240, #5351)
• Improved Qwen3-TTS performance by caching decoder masks and compiling the pre-transformer path. (#5107)
• Added configurable Qwen3-TTS codec chunk ramp-up for smoother streaming startup. (#5152)
• Added mode-aware reference-audio artifact readiness for Qwen3-TTS. (#5157)
• Refactored MOSS-TTS-Local v1.5 talker execution for CUDA Graph compatibility and added a single-80GB deployment configuration. (#5197, #4761)
• Fixed Qwen3-Omni speaker metadata during non-async handoff and restored audio output for MiniCPM-o when async chunking is disabled. (#5086, #5455)
• Fixed MiniCPM-o 4.5 audio placeholder pooling, Hugging Face cache-path resolution, and single-GPU startup memory usage. (#5116, #5301, #5447)
• Fixed request-local speech-code routing for MiMo Audio under continuous batching. (#5070)
• Propagated Higgs Audio V3 sampling parameters to the sampler and adjusted an overly strict Higgs Audio V2 streaming quality threshold. (#5406, #5232)
• Kept the HiFT vocoder on Ascend NPU instead of unnecessarily transferring it to the CPU. (#5242)
• Fixed audio output format handling in chat completions. (#4718)

Diffusion, Image & Video Generation

• Added CFG parallel execution for FLUX.1-Kontext-dev. (#2281)
• Added Cache-DiT support for FLUX.1-Kontext-dev and began reorganizing diffusion cache integrations. (#4205, #5226)
• Added FLUX.2-dev layerwise CPU offloading and VAE patch parallelism. (#5256, #5292)
• Added AllGather-KV sequence-parallel attention and a TensorRT-LLM diffusion attention backend with Skip-Softmax support. (#4968, #5283)
• Added configurable diffusion compilation granularity. (#4603)
• Extended Cosmos3 CPU offloading to component-level and layerwise execution paths. (#4695)
• Added Cosmos3 Edge and Distilled checkpoints and fixed configuration and scheduler compatibility for these models. (#5001, #5239, #5176)
• Added dense and MoE variants of LingBot Video and support for Boogu Image Base and Edit models. (#5035, #4995)
• Added midway prompt updates for streaming video generation. (#4652)
• Added per-video audio masks for Qwen Omni. (#4656)
• Fixed image CFG behavior in HunyuanImage and preserved image payloads when reasoning metadata is present. (#4752, #5238)
• Fixed Helios Cholesky failures caused by non-positive-definite matrices and unavailable CPU LAPACK paths. (#4921)
• Unified the LTX-2 and LTX-2.3 pipeline runtime. (#5147)
• Migrated the replica data-parallel Wan2.2 example to deployment configuration and added a replica-DP video DiT recipe and benchmark. (#5267, #5052)

Quantization, Kernels & Performance

• Added BitsAndBytes W4 online quantization for diffusion transformers. (#5037)
• Added Transformer Engine online FP8 support for the FLUX.2-dev Mistral component. (#5136)
• Fixed loading of ModelOpt FP8 Cosmos3 checkpoints and remapped ModelOpt NVFP4 scale tensors. (#5076, #5087)
• Added support for packed quantized parameters with HSDP. (#5088)
• Initialized component-level quantization state consistently. (#5103)
• Fixed FP8 Quack GEMM execution under inference_mode() and torch.compile() and disabled it when required scale tensors are unavailable. (#5153, #5262)
• Added configurable diffusion compile granularity and expanded diffusion performance configurations. **(#46...

v0.25.0rc1

Highlights

This release candidate features 32 merged pull requests from 22 contributors, including 5 new contributors.

vLLM-Omni v0.25.0rc1 aligns the project with the vLLM 0.25 release line and delivers improvements across composable parallel execution, diffusion and image generation, TTS performance and correctness, model integration, quantization extensibility, endpoint behavior, testing, and documentation.

This release adds Krea 2 text-to-image support, sequence parallelism for OmniGen2, and the first phase of composable parallel strategy overlays. It also improves Qwen3-TTS performance and batching, strengthens Ming-TTS, MOSS-TTS, Voxtral, and VoxCPM2 serving, migrates additional models and examples to standardized registries and model_extras, and moves GGUF diffusion quantization support into an out-of-tree plugin.

Key Improvements

• Aligned with the vLLM 0.25 release line, including the vLLM v0.25.0 rebase and compatibility fixes for custom IR operations, RMSNorm, stage payloads, deployment configuration, and model test levels. (#5042, #5009, #5015, #5008)
• Introduced composable parallel strategy overlays, providing the first phase of unified TP, DP, and stage-replica configuration for distributed Omni workloads. (#4281)
• Expanded diffusion and image-generation support, adding Krea 2, sequence parallelism for OmniGen2, Hugging Face kernel-package integration, improved hub-kernel loading, and standardized Ming-flash-omni-2.0 examples. (#4730, #3206, #4926, #4977, #4835)
• Improved Qwen3-TTS performance and batching, eliminating unnecessary per-step device-to-host transfers, restoring batched MTP sampling, aligning CUDA Graph capture with async output, and adding Gumbel-max code-predictor sampling. (#4879, #4970, #4923, #4261)
• Strengthened TTS and speech correctness, fixing out-of-vocabulary stop-token masking, direct speaker-embedding handling, MOSS-TTS codec resolution, Voxtral feedback and short-ASR checks, and VoxCPM2 remote-code loading. (#4971, #5005, #4760, #4954, #4984)
• Improved model and pipeline integration, migrating MammothModa2 and OmniVoice to the pipeline registry and fixing Qwen3-Omni video metadata handling when audio from video is enabled. (#4647, #4959)
• Expanded quantization extensibility, migrating GGUF diffusion-model support to an out-of-tree plugin. (#4769)
• Improved serving resilience and validation, adding endpoint rejection support, fixing LoRA argument forwarding, correcting RoPE shape handling, and expanding compact diffusion and image-to-video test coverage. (#4762, #4936, #4655, #4524, #3573)

Core Architecture & Runtime

• Added the first phase of composable parallel strategy overlays, enabling TP, DP, and stage-replica settings to be composed through a unified execution strategy. (#4281)
• Added a mechanism for endpoints to reject unsupported requests before execution, improving routing behavior and frontend error handling. (#4762)
• Migrated MammothModa2 and OmniVoice to the pipeline registry, consolidating model discovery and pipeline configuration. (#4647)
• Removed redundant functions and logging as part of the first phase of runtime cleanup. (#4986)
• Fixed stale CI expectations following recent stage-payload and deployment-configuration changes. (#5015)
• Restored vllm_c IR operation priority and torch.nn.RMSNorm behavior required by Qwen-Image after upstream compatibility changes. (#5009)

Model Support

• Added support for the Krea 2 text-to-image diffusion model. (#4730)
• Added sequence parallelism support for OmniGen2, improving distributed diffusion execution. (#3206)
• Migrated MammothModa2 and OmniVoice to the pipeline registry. (#4647)
• Migrated Ming-flash-omni-2.0 image-generation examples to the standard model_extras configuration style. (#4835)
• Fixed Qwen3-Omni video metadata handling when use_audio_in_video is enabled. (#4959)
• Fixed RoPE handling for inputs that could otherwise produce shape mismatches. (#4655)

Audio, Speech & TTS Serving

• Improved Qwen3-TTS performance by allowing hidden-pooler payloads to opt out of per-step hidden-state device-to-host transfers. (#4879)
• Removed the default Qwen3-TTS seed configuration that prevented batched MTP sampling. (#4970)
• Aligned Qwen3-TTS talker MTP CUDA Graph capture with Qwen3-Omni behavior and added async-output compatibility. (#4923)
• Added Gumbel-max sampling for Qwen3-TTS code-predictor outputs. (#4261)
• Fixed Qwen3-TTS engine crashes by dropping out-of-vocabulary stop IDs from minimum-token masking. (#4971)
• Fixed Ming-TTS voice handling so uploaded direct speaker embeddings are used correctly. (#5005)
• Improved MOSS-TTS codec compatibility by resolving the decode method across remote-code and vendored tokenizer implementations. (#4760)
• Fixed Voxtral TTS feedback handling and short-ASR validation. (#4954)
• Fixed VoxCPM2 performance-server loading when models require trust_remote_code. (#4984)

Diffusion & Image Generation

• Added Krea 2 text-to-image model support. (#4730)
• Added sequence parallelism for OmniGen2 diffusion inference. (#3206)
• Added support for the Hugging Face kernels package in native diffusion attention backends. (#4926)
• Improved Hugging Face Hub kernel discovery and loading in flash_attn_hub. (#4977)
• Migrated GGUF diffusion quantization support to an out-of-tree plugin. (#4769)
• Fixed LoRA argument forwarding in the offline text-to-image example. (#4936)
• Added a reusable tiny-diffusion-model testing pattern for lightweight and deterministic diffusion tests. (#4524)
• Restored operation priority and RMSNorm compatibility required by Qwen-Image. (#5009)

Quantization, Kernels & Dependencies

• Migrated GGUF diffusion-model support into an out-of-tree quantization plugin, reducing core coupling and improving extensibility. (#4769)
• Added Hugging Face kernel-package support for native diffusion attention backends. (#4926)
• Improved remote kernel loading behavior in flash_attn_hub. (#4977)
• Updated safetensors to the 0.8.0 release. (#4713)

Serving, Frontend & API Behavior

• Added endpoint rejection support so incompatible or unsupported requests can be rejected through a standardized mechanism. (#4762)
• Fixed LoRA argument propagation in offline text-to-image serving examples. (#4936)
• Fixed Qwen3-Omni video metadata propagation for requests using audio embedded in video. (#4959)
• Fixed direct speaker-embedding handling for Ming-TTS voice requests. (#5005)
• Corrected the MiniCPM-o 4.5 curl-based TTS example and clarified chat_template_kwargs usage. (#4950)

CI, Testing & Documentation

• Added image-to-video offline example documentation and test coverage. (#3573)
• Added a tiny diffusion model pattern for faster and more focused diffusion testing. (#4524)
• Fixed core-model CI level selection for non-diffusion models. (#5008)
• Updated invalid-layer DFX tests to use the supported 2–10 range. (#5047)
• Fixed CI failures caused by stale expectations after stage-payload and deployment-configuration changes. (#5015)
• Updated the WeChat community group QR code. (#4939)

What's Changed

• [Refactor] Migrate Ming-flash-omni-2.0 Image-gen examples with model_extras by @yuanheng-zhao in #4835
• [Bugfix] Fix LoRA arguments passing in offline text-to-image script by @SamitHuang in #4936
• [BugFix] fix possible shape mismatch when using ROPE by @Semmer2 in #4655
• Update WeChat group QR code by @david6666666 in #4939
• [Refactor] Migrate MammothModa2 and OmniVoice to pipeline registry by @zwhzzz0821 in #4647
• [Diffusion][Feature] Add SP for Omnigen2 by @zhangj1an in #3206
• [Test] Add image-to-video offline example docs and tests by @loveysuby in #3573
• [Bugfix] MOSS-TTS codec: resolve decode method across remote-code and vendored tokenizers by @IneshReddy249 in #4760
• Support HuggingFace kernels package for native diffusion attention backends by @SamitHuang in #4926
• docs(recipe): fix MiniCPM-o 4.5 curl TTS example and clarify chat_template_kwargs usage by @amy-why-3459 in #4950
• [Core / Bugfix] Add Mechanism for Endpoint Rejection by @alex-jw-brooks in #4762
• Fix Voxtral TTS feedback and short ASR checks by @napleon-liu in #4954
• [Bugfix] Drop out-of-vocabulary stop ids from min-tokens masking (qwen3-tts min_tokens engine crash) by @linyueqian in #4971
• [Perf][Qwen3-TTS] Skip per-step hidden-state D2H via hidden pooler payload opt-out by @linyueqian in #4879
• [Perf][Qwen3-TTS] Drop default seed from qwen3_tts.yaml to restore batched MTP sampling by @linyueqian in #4970
• [Diffusion] Improve HuggingFace hub kernel loading in flash_attn_hub by @SamitHuang in #4977
• [Tests] Tiny Diffusion Model Pattern by @alex-jw-brooks in #4524
• Bump safetensors to the 0.8.0 release by @oglok in #4713
• [Perf][Qwen3-TTS] Align ...

v0.24.1

Highlights

This release resolves performance regression of Qwen-Image model according to issue #4964 . And it also fixes the default operator priority for RMSNorm.

What's Changed

• [BugFix] Restore vllm_c IR op priority and torch.nn.RMSNorm for Qwen-Image (Same As #5009) by @NumberWan in #5017

Full Changelog: v0.24.0...v0.24.1

v0.24.0

Highlights

This release features 285 commits from 112 contributors, including 40 new contributors.

vLLM-Omni v0.24.0 brings a large set of improvements across TTS, speech, diffusion, image generation, video generation, robot-policy serving, quantization, runtime architecture, request batching, cache behavior, CI, and hardware backend support.

This release expands production-ready multimodal serving coverage with new and improved models such as Higgs Audio V3, Qwen3-TTS, Ming-omni-tts, IndexTTS2, MOSS-TTS-local-v1.5, Step-Audio2, SDXL, Cosmos3, GR00T-N1.7, HunyuanImage3, DreamZero, Wan2.2 / Wan S2V, SenseNova-U1, JoyAI-VL-Interaction, and more. It also includes the vLLM 0.23 and 0.24 rebases, significant Omni stage runtime refactoring, diffusion request-level batching, async output materialization, improved streaming paths, and broad CUDA / ROCm / XPU / NPU coverage.

Key Improvements

• Aligned with the vLLM 0.24 release line, including the vLLM 0.23.0 and v0.24.0 rebases, release-branch updates, and compatibility fixes across model loading, graph behavior, quantization, and serving paths. (#4286, #4709)
• Expanded TTS and audio model support, adding or improving Higgs Audio V3, Qwen3-TTS, CosyVoice, Fish Speech, VoxCPM2, Voxtral, Ming-omni-tts, IndexTTS2, MOSS-TTS-local-v1.5, Step-Audio2, soulx-singer, Aura, and Stable-Audio-Open. (#4169, #4204, #3838, #4664, #4341, #464, #3862, #4257, #3664)
• Made speech serving more production-ready, with SSE audio streaming, word-level timestamps, Qwen3-TTS correctness fixes, token accounting, WebSocket input handling, streaming-tail fixes, cross-request audio-corruption fixes, and better voice-clone reference-audio behavior. (#4034, #4490, #4679, #4673, #4731, #4706, #4415, #4429)
• Strengthened diffusion, image, and video generation, including major improvements for Cosmos3, HunyuanImage3, DreamZero, BAGEL, Wan2.2, Wan S2V, LTX-2.3, SDXL, Qwen-Image, Qwen-Image-Edit, MammothModa2, GLM-Image, and SenseNova-U1. (#4331, #4379, #4514, #4627, #4041, #4213, #4098, #4667, #4739, #4474, #4293, #4299, #3956, #4188)
• Improved core Omni runtime and serving infrastructure, including the Omni stage runtime and distributed replica control plane, orchestrator output-path optimization, async output materialization, diffusion request-level batching, HF-config-based pipeline resolution, structured config classes, and configurable video storage backends. (#3855, #4527, #4476, #4079, #3760, #4425, #2531)
• Expanded quantization and memory-efficiency coverage, including ModelOpt FP8 for video-generation models, Qwen3-Omni NVFP4 W4A4 on Blackwell, Qwen-Image AutoRound W4A16, ROCm MXFP4 online quantization, HSDP + FP8 compatibility, CacheDiT cleanup, prefix-cache fixes, and improved memory metrics. (#3305, #4025, #4528, #3588, #4104, #4494, #2527, #4449, #4106, #4477)
• Improved hardware backend support, with fixes and features for CUDA, ROCm, XPU, and Ascend NPU, including NPU graph behavior, custom-op registration, diffusion-worker initialization, XPU CI fixes, platform-agnostic device selection, and ROCm test stability. (#4454, #4712, #4386, #4674, #4451, #4496, #4380, #4574, #4821, #4848)

Core Architecture & Runtime

• Refactored the Omni stage runtime and distributed replica control plane, providing a stronger foundation for multistage omni serving and distributed stage coordination. (#3855)
• Reduced orchestrator bottlenecks by separating inter-stage outputs from client-facing outputs. (#4527)
• Added async Omni output materialization so generated multimodal artifacts can be surfaced through more flexible serving flows. (#4476)
• Added request-level batching for diffusion pipelines, improving throughput and serving efficiency for compatible diffusion workloads. (#4079)
• Introduced HF-config-based pipeline resolution and eager pipeline registry behavior, improving automatic pipeline selection and startup behavior. (#3760)
• Added structured VllmOmniConfig classes and restored deploy-config-based pipeline resolution in StageConfigFactory. (#4425, #4729)
• Migrated existing pipelines to SupportsComponentDiscovery / ModuleDiscovery, improving offload discovery and nested HSDP DiT sharding support. (#3076, #3456)
• Extracted diffusion output formatting boundaries and removed dead custom stage input hooks to simplify stage-processing internals. (#4407, #4531)
• Fixed multi-replica stage identity and cleaned up redundant deploy configuration. (#4410)
• Aligned Omni request input signatures with vLLM and fixed multimodal generation-stage output construction. (#4568, #4579)

Model Support

• Added support for bosonai/higgs-audio-v3-tts-4b and optimized Higgs Audio V3 serving paths. (#4169, #4204)
• Added or improved TTS model families including Ming-omni-tts dense 0.5B, Ming-omni-tts MoE 16.8B-A3B, IndexTTS2, MOSS-TTS-local-v1.5, Step-Audio2, soulx-singer, Aura, and fine-tuned Qwen3-TTS language support. (#2906, #4341, #3838, #4664, #4804, #464, #3862, #4257, #4210)
• Added or enabled diffusion and image/video model coverage for SDXL, GR00T-N1.7, Cosmos3, HunyuanImage3, DreamZero, Wan2.2 / Wan S2V, LTX-2.3 I2V, JoyAI-VL-Interaction, MammothModa2, and SenseNova-U1. (#4331, #3798, #4266, #4041, #4154, #4276, #4458, #4739, #4575, #4299, #4188)
• Added DROID policy server support for Cosmos3 OpenPI and GR00T-N1.7 OpenPI serving, expanding robot-policy serving coverage. (#4282, #3798)
• Migrated or refreshed examples for Cosmos3, Helios, Magi human, VACE, MammothModa2-Preview, AudioX, Qwen Image, Stable-Diffusion-3.5, Voxtral TTS, and Qwen3-TTS. (#4581, #4569, #4572, #4648, #4691, #4607, #4236, #4052, #4051, #4026)

Audio, Speech & TTS Serving

• Improved Qwen3-TTS serving correctness across custom voice mode, WebSocket input handling, no-async-chunk payload splitting, multi-token replay on resume, seeded residual MTP sampling, token accounting, Code2Wav CUDA Graph output length, and full-payload completion behavior. (#4650, #4731, #4870, #4559, #4869, #4889, #4673, #4525, #4465)
• Added streaming TTS word-level timestamps via forced alignment. (#4034)
• Added SSE stream_format support for audio speech streaming and made speech streaming default to SSE. (#4490, #4679)
• Fixed Code2Wav async-chunk streaming tail drop, reducing boundary clicks and time-compression artifacts. (#4706)
• Improved CosyVoice and CosyVoice3 paths, including TensorRT optimization, reference-text instruction templating, and prompt handling fixes. (#4168, #4756)
• Fixed Fish Speech serving issues including Gradio default voice handling, prefix-cache collision, and KV-cache unpacking after vLLM compatibility changes. (#3941, #4008, #4428)
• Improved MOSS-TTS stability and performance with eager initialization, CUDA Graph support, cross-request audio-corruption fixes, talker micro-optimizations, loading shims, streaming support, and WER accuracy fixes. (#3230, #4157, #4415, #4230, #4398, #4804, #4859)
• Improved VoxCPM2 quality, hardware adaptation, KV-cache settings, unified decode graph optimization, and test stability. (#4319, #4310, #4374, #4279, #4606, #4878)
• Updated audio and TTS recipes, baselines, and docs for Qwen3-TTS, VoxCPM, Voxtral, Stable-Audio-Open, AudioX, and Ming-omni-tts. (#4026, #4521, #4051, #4774, #3664, #4607, #4324)

Diffusion, Image & Video Generation

• Improved Cosmos3 generation, transfer, quality, distributed behavior, testing, and documentation, including v2v support, transfer paths, regression fixes, I2V conditioning optimization, skipped unused conditioning latents, regional compile, sequence-parallel sound latent padding, OpenPI serving, and L2 serving tests. (#4266, #4379, #4514, #4627, #4467, #4614, #4485, #4678, #4282, #4535)
• Improved HunyuanImage3 correctness and performance with DiT grouped step batching, more resolution support, AR RGB conversion alignment, CoT truncation fixes, streaming CoT display, sync removal, timestep scalar-sync avoidance, KV prefetch, MoE fixes, and bridge request batching fixes. (#4041, #4004, #4502, #4260, #4148, #4401, #4363, #4448, #4893, #4849)
• Advanced DreamZero serving with TP and cross-attention cache fixes, CUDA Graph support, torch.compile, DiT caching, and engine-level KV-cache management groundwork. (#4154, #4213, #4534)
• Improved BAGEL generation with denoising schedule fixes, batched CFG forward reimplementation, CFG-parallel mixin reuse, memory metrics, and SP denoise fixes. (#4509, #4098, #4768, #4477, #4328)
• Improved Wan2.2 / Wan S2V with graph-break fixes, CacheDiT + Ulysses fixes, S2V API support, SP and HSDP support, Wan VAE spatially sharded decode, and VACE cache / lifecycle fixes. (#4053, #3927, #3394, #4276, #4458, #4620, #4667)
• Added LTX-2.3 I2V support, VAE decode parallelism, resident auxiliary modules, and tensor-parallel gated-attention fixes. (#4739, #4277, #4144, #4439)
• Improved image-generation serving with file-style image response_format, default image-generation limits, Qwen-Image RoPE fixes, Qwen-Image-Edit performance fixes, Qwen-Image-Layered compatibility, and Ming-flash-omni image-generation batching fixes. (#1673, #3381, #4474, #4293, #4777, #4837)
• Improved SenseNova-U1 with LoRA support, CFG parallelism, fused RMSNorm + 3D RoPE kernels, TeaCache, and pipeline fixes. (#3971, #4188, #4669, #4164, #4326)

Quantization, Cache & Memory Efficiency

• Added ModelOpt FP8 support for Wan2.2 and HunyuanVideo-1.5 video generation, and propagated quantization configuration into HunyuanVideo-1.5 I2V transformer layers. (#3305, #4245)
• Added Qwen3-Omni NVFP4 W4A4 serving on Blackwell and fixed Thinker lm_head prefix handling so NVFP4 exclude lists ar...

v0.24.0rc1

Highlights

This release candidate features 158 merged changes from 82 contributors, including 21 new contributors.

vLLM-Omni v0.24.0rc1 is a release candidate aligned with the vLLM 0.24 release line. It expands diffusion, video, TTS, and robot-serving coverage while tightening the multistage runtime, request batching, quantization, cache behavior, and hardware backend support. This release candidate is intended to validate the vLLM 0.24 rebase and the new production-serving paths before the final cut.

Key Improvements

• Aligned with the vLLM 0.24 release line, including the vLLM 0.24.0 rebase and the new v0.24.0rc1 release branch/tag baseline. (#4709)
• Expanded model and modality coverage, adding or improving SDXL, GR00T-N1.7 with OpenPI serving, IndexTTS2, MOSS-TTS-local-v1.5, Ming-omni-tts MoE 16.8B-A3B, soulx-singer, Aura, and JoyAI-VL-Interaction. (#4331, #3798, #3838, #4664, #4341, #3862, #4257, #4575, #4623)
• Strengthened diffusion, image, and video generation, with major Cosmos3, HunyuanImage3, DreamZero, BAGEL, Wan, streaming-video, and image-response improvements. (#4379, #4514, #4627, #4467, #4614, #4041, #4213, #4098, #4667, #3737, #1673)
• Made audio and TTS serving more production-ready, including Qwen3-TTS correctness and streaming fixes, word-level timestamps, SSE audio streaming, CosyVoice3 prompt handling, MOSS-TTS stability, and new TTS model support. (#4429, #4465, #4525, #4673, #4731, #4034, #4490, #4679, #4756, #4415, #3838)
• Improved core runtime and serving infrastructure, with the Omni stage runtime control plane, orchestrator output-path fixes, async output materialization, diffusion request-level batching, HF-config-based pipeline resolution, structured config, and video storage backends. (#3855, #4527, #4476, #4079, #3760, #4425, #2531)
• Expanded quantization, cache, and hardware coverage, including Qwen3-Omni NVFP4 W4A4 on Blackwell, Qwen-Image AutoRound W4A16, HSDP+FP8 compatibility, CacheDiT cleanup, prefix-cache fixes, memory metrics, and NPU/ROCm/XPU fixes. (#4025, #4528, #3588, #4494, #2527, #4449, #4106, #4477, #4454, #4712, #4574)

Core Architecture & Runtime

• Refactored the Omni stage runtime and distributed replica control plane, then reduced orchestrator bottlenecks by separating inter-stage outputs from client-facing outputs. These changes improve the runtime foundation for multistage omni serving. (#3855, #4527)
• Added async Omni output materialization and request-level batching for diffusion pipelines, giving serving paths a cleaner way to handle generated artifacts and batch compatible diffusion work at the request level. (#4476, #4079)
• Improved pipeline and configuration resolution with HF-config-based pipeline selection, eager pipeline registry behavior, structured VllmOmniConfig classes, deploy-config restoration in StageConfigFactory, and component discovery for offload. (#3760, #4425, #4729, #3076)
• Cleaned up diffusion and stage boundaries by extracting output formatting, removing dead custom stage input hooks, fixing multi-replica stage identity, and aligning multimodal generation output construction. (#4407, #4531, #4410, #4579)

Model Support

• Added or enabled new model families and serving paths, including SDXL, GR00T-N1.7 with OpenPI serving, IndexTTS2, MOSS-TTS-local-v1.5, Ming-omni-tts MoE 16.8B-A3B with CFM CUDA Graph, soulx-singer, Aura non-async-chunk serving, and JoyAI-VL-Interaction streaming interaction serving. (#4331, #3798, #3838, #4664, #4341, #3862, #4257, #4575)
• Migrated or refreshed examples for Helios, Magi human, VACE, MammothModa2-Preview, AudioX, Cosmos3, and Qwen image recipes, making more model entries follow the standard task-example and model_extra pattern. (#4569, #4572, #4648, #4691, #4607, #4581, #4236)
• Improved existing model behavior across HunyuanImage3, LTX-2.3, Qwen-Image, Qwen-Image-Edit, MammothModa2, Fish Speech, Voxtral, and SenseNova-U1. (#4416, #4439, #4474, #4293, #4299, #4428, #4380, #4188)

Audio, Speech & Omni Production Optimization

• Improved Qwen3-TTS serving correctness across custom voice handling, degenerate full-payload completion, Code2Wav CUDA Graph output length, WebSocket input handling, and /v1/audio/speech token accounting. (#4429, #4465, #4525, #4731, #4673)
• Added word-level timestamps through a forced aligner and expanded speech streaming behavior with SSE stream-format support and SSE as the default speech-streaming mode. (#4034, #4490, #4679)
• Stabilized other speech and audio paths, including MOSS-TTS cross-request audio correctness, MOSS-TTS talker performance, MOSS-TTS loading compatibility, CosyVoice3 reference-text templating, Higgs Audio v3 lazy codec loading, and Fish Speech KV-cache unpacking for vLLM 0.23.0 compatibility. (#4415, #4230, #4398, #4756, #4368, #4428)
• Updated TTS and audio recipes and baselines for Qwen3-TTS, VoxCPM, Voxtral, Stable-Audio-Open, AudioX, and related recipe references. (#4026, #4521, #4051, #3664, #4607, #4567)

Diffusion, Image & Video Generation

• Improved Cosmos3 quality, transfer behavior, performance, and distributed correctness, including Cosmos3 transfer, v2v quality, regression fixes, I2V conditioning transfer optimization, skipped unused I2V conditioning latents, HSDP regional compile, sequence-parallel sound latent padding, and Cosmos3 L2 serving tests. (#4379, #4514, #4627, #4467, #4614, #4485, #4678, #4535)
• Improved HunyuanImage3 performance and correctness with DiT grouped step batching, AR RGB conversion alignment, sync removal, timestep scalar-sync avoidance, prefetch KV, and streaming CoT display for AR generation. (#4041, #4502, #4401, #4363, #4448, #4148)
• Advanced DreamZero and BAGEL generation paths with DreamZero TP and cross-attention cache fixes, CUDA Graph/torch.compile/DiT caching support, BAGEL denoising schedule fixes, reimplemented batched CFG forward, and CFG-parallel mixin reuse. (#4154, #4213, #4509, #4098, #4768)
• Improved Wan and video-serving behavior with Wan2.2 graph-break and CacheDiT/Ulysses fixes, Wan2.2-VACE-Fun cache and lifecycle fixes, Wan VAE spatially sharded decode, Wan S2V SP/HSDP support, and streaming diffusion video output. (#4053, #3927, #4667, #4620, #4276, #4458, #3737)
• Expanded image-generation serving and correctness with file-style image response_format, Qwen-Image RoPE and edit-path performance fixes, GLM-Image CFG-parallel dtype fixes, MammothModa2 text-to-image fixes, and SenseNova-U1 CFG parallel, fused RMSNorm+3D RoPE, and TeaCache support. (#1673, #4474, #4293, #3956, #4299, #4188, #4669, #4164)

Quantization & Memory Efficiency

• Added Qwen3-Omni NVFP4 W4A4 serving on Blackwell and fixed Thinker lm_head prefix handling so NVFP4 exclude lists are honored. (#4025, #4528)
• Expanded diffusion/image quantization coverage with Qwen-Image AutoRound W4A16, Qwen2.5-Omni AutoRound loading fixes, HSDP plus FP8 online quantization compatibility, and Cosmos3-Nano/Super online FP8 validation docs. (#3588, #4781, #4494, #4393, #4584)
• Improved cache and memory behavior by simplifying CacheDiT integration, fixing DFlash prefix-cache corruption, avoiding per-step blocking writes in OmniTensorPrefix, adding Bagel memory metrics, and using module discovery for nested HSDP DiT sharding. (#2527, #4449, #4106, #4477, #3456)

RL, Serving & Integrations

• Added DROID policy server support for Cosmos3 OpenPI and GR00T-N1.7 pipeline support with OpenPI serving, extending robot-policy serving coverage in this release line. (#4282, #3798)
• Improved frontend and OpenAI-compatible serving behavior with configurable video storage backends and TTL, correct HTTP status codes for audio voice endpoints, vLLM-aligned Omni request signatures, and fixes for multimodal generation-stage outputs. (#2531, #3969, #4568, #4579)
• Added async output materialization and streaming diffusion-video output so artifact-producing requests can be surfaced through more flexible serving flows. (#4476, #3737)

Platforms, Distributed Execution & Hardware Coverage

• Improved Ascend NPU coverage with model-runner unpacking fixes, OmniMRotaryEmbedding support, diffusion-worker Ascend config initialization, custom-op registration, and restored pre-#9572 graph behavior by capping NPU CUDA-graph mode to piecewise. (#4454, #3609, #4386, #4712, #4674)
• Improved ROCm, CUDA, and XPU stability through Voxtral and diffusion test fixes, XPU marker fixes, and replacement of hardcoded CUDA device selection with platform-agnostic APIs. (#4380, #4574, #4451, #4496)
• Expanded distributed diffusion paths with HSDP/SP-related fixes and features across Cosmos3, Wan S2V, Wan VAE, and nested DiT sharding. (#4452, #4678, #4276, #4458, #4620, #3456)

CI, Benchmarks & Documentation

• Strengthened rebase, ready/merge, and nightly coverage with full E2E rebase pipelines, fixed pipeline upload behavior, nightly L2/L3 E2E groups, split diffusion X2V tests, Cosmos3 L2 serving tests, and reduced ready-CI duration. (#4478, #4532, #4693, #4734, #4744, #4535, #4354)
• Updated performance baselines and reliability coverage for VoxCPM, Qwen3-TTS, HunyuanImage3, VoxCPM2, and related perf JSONs. (#4289, #4521, #4600)
• Improved contributor tooling and docs with a vLLM-Omni test-agent skill, code-quality guidance for the PR precheck skill, bug-report field IDs, CUDA custom Docker build guidance, and refreshed model recipes. (#4434, #4697, #4605, #1386, #4567)

What's Changed

• [BugFix] Fix HunyuanImage3 size match issue by @Semmer2 in #4416
• [Perf] Restore parallel stage initialization for AR+DiT pipelines by @zengchuang-hw in #3641
• [Doc] validate Cosmos3-Nano online FP8 b...

v0.23.0rc1

Highlights

This release candidate features 79 commits from 68 contributors, including 12 new contributors.

vLLM-Omni v0.23.0rc1 is a release candidate aligned with upstream vLLM v0.23.0. It focuses on validating the next release line by expanding TTS/audio model coverage, improving speech serving latency and correctness, strengthening diffusion/image/video generation paths, and broadening quantization and hardware backend readiness across CUDA, Blackwell, ROCm, NPU, and XPU.

This release candidate is intended to validate the vLLM 0.23 integration, the refreshed TTS serving adapter path, Higgs Audio V3 and Qwen3-TTS production behavior, and diffusion/video quantization and platform coverage before the final v0.23.0 release.

Key Improvements

• Rebased to upstream vLLM v0.23.0, refreshing the base runtime for the v0.23 release line. (#4286)
• Expanded speech and TTS model coverage, adding Higgs Audio V3 TTS, Step-Audio2, Ming-omni-tts dense 0.5B, broader Qwen3-TTS language support, and LoRA support for SenseNova-U1. (#4169, #464, #2906, #4210, #3971)
• Improved TTS serving performance and reliability, with Higgs Audio V3 reference-audio caching and prefix caching, Qwen3-TTS hot-path optimizations and prefix-cache fixes, TensorRT acceleration for CosyVoice, and CUDA Graph support for MOSS-TTS. (#4200, #4199, #4204, #3689, #4317, #4168, #4157)
• Strengthened diffusion, image, and video generation, including Cosmos3 video-to-video, WAN2.2-S2V image+audio server API support, HunyuanImage3 resolution and latency improvements, LTX-2.3 VAE decode parallelism, and Wan2.2 sequence-parallel performance fixes. (#4266, #3394, #4004, #4333, #4277, #3763)
• Expanded quantization and hardware coverage, including ModelOpt FP8 for Wan2.2 and HunyuanVideo-1.5, Blackwell FP8 GEMM fused-bias kernels, XPU sage attention, NPU VoxCPM2 support, and Ascend 310P support for Qwen3-TTS. (#3305, #4245, #4241, #3785, #4310, #4283)
• Improved runtime architecture and developer experience, with the TTS serving adapter framework, multimodal output-channel separation, audio-in-video refactoring, Diffusers pipeline cleanup, and stronger CI/test gating. (#4330, #2744, #3566, #1932, #4313, #4365)

Core Architecture & Runtime

• Rebased vLLM-Omni to upstream vLLM v0.23.0, keeping the Omni runtime aligned with the latest upstream release line. (#4286)
• Introduced the TTS serving adapter framework and migrated TTS models onto the refreshed adapter path, making speech model serving easier to extend and maintain. (#4330)
• Continued multimodal output processor work by separating multimodal output channels, improving the architecture for heterogeneous text/audio/video outputs. (#2744)
• Cleaned up diffusion pipeline loading by removing dead legacy pipeline.yaml loading paths and duplicate diffusion logic, and by simplifying DiffusersPipelineLoader. (#4023, #1932)
• Refactored audio-in-video implementation and added the MoriIO transfer engine, improving maintainability for multimodal data routing and media-transfer workflows. (#3566, #1742)
• Refined guardrail error handling with explicit 400-level error behavior for invalid requests. (#4297)

Model Support

• Added support for bosonai/higgs-audio-v3-tts-4b, bringing Higgs Audio V3 TTS into the vLLM-Omni serving stack. (#4169)
• Added Step-Audio2 support and Step-Audio R1 reasoning parser support. (#464, #2846)
• Added Ming-omni-tts dense 0.5B pipeline support and follow-up compatibility fixes for Mapping-style input checks. (#2906, #4397)
• Expanded Qwen3-TTS language support for fine-tuned checkpoints and added non_streaming_mode for Qwen3-TTS base models during online inference. (#4210, #4198)
• Added LoRA support for SenseNova-U1. (#3971)
• Added more resolution support for HunyuanImage3.0. (#4004)
• Added Cosmos3 video-to-video generation and Cosmos3-Nano baselines. (#4266, #4301)
• Added WAN2.2-S2V server API support for image + audio input workflows. (#3394)

Audio, Speech & Omni Production Optimization

• Improved Higgs Audio V3 serving with LRU caching for voice-clone reference-audio encoding, Stage-0 prefix caching enabled by default, and serving-path optimizations. (#4200, #4199, #4204)
• Fixed Higgs Audio V3 Stage0 talker ramp-down and buffer-state crashes, improving stability for long-running speech serving. (#4219)
• Optimized Qwen3-TTS hot paths with prefix-cache OOM guards, orchestrator/talker micro-optimizations, and fixes for prefix-cache corruption and cross-request codes_ref leakage. (#3689, #4317, #4373)
• Improved Qwen3-TTS Gradio streaming TTFP by using audio/pcm. (#4346)
• Optimized CosyVoice TTFP and throughput using TensorRT. (#4168)
• Added CUDA Graph support for the MOSS-TTS codec decoder and made MOSS-TTS-Nano eager-init compatible with load_format: dummy. (#4157, #3230)
• Fixed Fish Speech serving issues, including a Gradio default-voice 400 error and prefix-cache collisions from missing cache_salt. (#3941, #4008)
• Improved VoxCPM2 robustness with KV-cache pinning for smaller GPUs and fixes for concurrent speech quality. (#4279, #4319)

Diffusion, Image & Video Generation

• Added Cosmos3 video-to-video generation support and Cosmos3-Nano baseline coverage. (#4266, #4301)
• Added WAN2.2-S2V server API support for image+audio generation workflows. (#3394)
• Improved HunyuanImage3 behavior with offline CoT fixes, CoT truncation fixes, stream-mode accuracy fixes, additional resolution support, and a prepare_attention_mask optimization that reduces end-to-end latency. (#4174, #4260, #4265, #4004, #4333)
• Improved HunyuanVideo-1.5 quantization propagation so I2V transformer FP8 layers can be enabled correctly. (#4245)
• Improved LTX-2.3 production behavior by keeping auxiliary modules resident by default, adding VAE decode parallelism, and fixing RMSNorm identity-weight registration. (#4144, #4277, #4278)
• Optimized Wan2.2 sequence-parallel behavior by skipping attention masks for zero-padded SP sequences to avoid the varlen path. (#3763)
• Improved Lance text-to-image and image-to-image performance. (#4214)
• Fixed BAGEL SP denoise indentation behavior. (#4328)

Quantization & Memory Efficiency

• Added ModelOpt FP8 support for Wan2.2 and HunyuanVideo-1.5 video generation. (#3305)
• Propagated quantization configuration into HunyuanVideo-1.5 I2V transformer paths to enable FP8 layers. (#4245)
• Defaulted Blackwell FP8 GEMM to the quack CuteDSL fused-bias kernel, improving the default quantized execution path on Blackwell. (#4241)
• Pinned VoxCPM2 KV cache to reduce OOM risk on smaller GPUs. (#4279)

Platforms, Distributed Execution & Hardware Coverage

• Added XPU sage_attn backend support. (#3785)
• Removed CUDA hardcoding in Cosmos3 XPU paths and made VLLM_VIDEO_SYNC_TIMEOUT tunable. (#4360)
• Updated DreamZero to support non-CUDA hardware paths. (#4399)
• Added NPU support for VoxCPM2 and adapted the VoxCPM2 audio encoder for non-CUDA backends. (#4310, #4374)
• Adapted Qwen3-TTS for Ascend 310P. (#4283)
• Added ROCm CI group/env features. (#4208)

Reliability, Tooling & Developer Experience

• Fixed chunk-transfer zombie cleanup on every scheduler tick to keep engine-core alive after request aborts. (#3774)
• Removed the pydub dependency for Python 3.13 compatibility. (#4035)
• Surfaced all-rank diffusion RPC failures so distributed diffusion errors are easier to diagnose. (#4403)
• Added automatic cleanup for generated audio files in tests and expanded realtime invalid-parameter coverage. (#4294)
• Improved CI efficiency with diff-aware L2/L3 gating, skipped unrelated merge/ready tests, single-GPU queue migration, and temporary skips for unstable OOM/MOSS-TTS-Nano cases. (#4291, #4313, #4365, #4311, #4391)
• Added Voxtral TTS tests. (#3738)

CI, Benchmarks & Documentation

• Updated README and supported model documentation for TTS and diffusion categories. (#4233, #4300)
• Added a Stable Diffusion 3.5 recipe for 1× RTX A6000 48GB. (#4052)
• Added CUDA verification notes for inclusionAI/Ming-omni-tts-0.5B. (#4324)
• Added failure-mode documentation and cleaned up the PR template. (#3926, #4336)
• Added Claude skills for precheck-PR and quantization workflows. (#4216, #4252)

What's Changed

• [TTS][New Model] support bosonai/higgs-audio-v3-tts-4b by @yuekaizhang in #4169
• [Perf][Higgs-Audio-V3] LRU cache for voice-clone ref-audio encode by @linyueqian in #4200
• [Perf][Higgs-Audio-V3] Turn on Stage-0 prefix caching by default by @linyueqian in #4199
• [Doc] Add Stable-Diffusion-3.5 recipe for 1x RTX A6000 48GB (#2645) by @yangyonggit in #4052
• [Bugfix] Fish Speech Gradio hardcoded default voice causes 400 error by @nagelanping in #3941
• [Model] Support languages added by fine-tuned Qwen3-TTS checkpoints by @n0n4m39911 in #4210
• [Quant] ModelOpt FP8 for Wan2.2 & HunyuanVideo-1.5 video-gen by @lishunyang12 in #3305
• [BugFix] fix hunyuan image3 offline cot by @BLANKETusers in #4174
• [skip ci] docs: update WeChat QR code by @david6666666 in #4242
• [Refactor]Refactoring audio_in_video implementation by @amy-why-3459 in #3566
• [Doc] Add precheck-pr Claude Code skill by @hsliuustc0106 in #4216
• [Perf] Keep LTX2.3 auxiliary modules resident by default by @mglyn in #4144
• [Fix][HiggsAudioV3] Fix ramp-down off-by-one crash ...

v0.22.0

Highlights

This release features 339 commits from 124 contributors, including 52 new contributors.

vLLM-Omni v0.22.0 is an omnimodal world-model release aligned with the vLLM 0.22 release line. It provided Day-0 support for Nvidia Cosmos 3 world-model support for text/image/audio/video/action input/output, and broadens speech and multimodal model coverage, and improves production serving across multistage runtime, OpenPI robot serving, diffusion acceleration, quantization, and hardware backends.

Key Improvements

• World model support, with Cosmos3 model day-0 support, sound generation, action modality, and DreamZero integration with CFG parallel plus OpenPI serving. (#3454, #4073, #4102, #2162, #3673)
• Expanded quantization and hardware coverage, including Blackwell diffusion attention backends, W4A16 (Intel autoround), FP8/INT8, MXFP4, MXFP8, ModelOpt mixed FP8/NVFP4, batched ModelOpt FP8, ROCm AITER, Intel XPU, and Ascend NPU updates. (#3353, #3059, #3700, #3902, #3578, #3570, #3782, #3943, #4155, #3079, #3015, #3419, #3511, #2325)
• Made audio and TTS serving more production-ready, with Qwen3-TTS, Qwen3-Omni, VoxCPM2, Fish Speech S2 Pro, OmniVoice, async audio input, custom voices, ref-context cache, and high-concurrency improvements. (#3662, #3492, #3322, #3592, #4054, #3882, #3773, #3336, #3614)
• RL integration with veRL-Omni, with Qwen-Image, Bagel, SD 3.5, WAN 2.2. (#3915, #2973, #3818, #3610)
• Aligned with the vLLM 0.22 release line, including the vLLM 0.21 and 0.22 rebases, dependency compatibility updates, release image builds, and PyPI upload support. (#3530, #3891, #4022, #3428, #3667)

Core Architecture & Runtime

• Integrated OmniCoordinator into the stage engine pipeline and continued the communication-layer refactor across non-async omni paths, improving multistage orchestration, request routing, and model-runner reuse. (#3569, #2677, #3719, #3476)
• Hardened stage and diffusion lifecycle behavior with worker dead detection, cleanup fixes, safer subprocess shutdown, SIGINT cleanup for NCCL/ZMQ resources, master-port selection fixes, and diffusion prefetch protection for newer transformers shard-resolution behavior. (#3214, #3494, #3751, #3872, #3803, #4076)
• Improved request and scheduler correctness through unified diffusion request identity, prefix-cache and token-history fixes, streaming finish reasons, Qwen3-Omni sampling alignment with transformers, and deterministic media-path handling in mixed-modality examples. (#3744, #3665, #3681, #3374, #4137, #3355)
• Added TrackingArgumentParser and refreshed configuration behavior around recursive engine-arg merging, deploy-config field allowlisting, concrete entrypoint typing, and single-stage/multistage test coverage. (#3369, #3009, #3483, #3139)

Model Support

• Added Cosmos3 support across model execution, recipes, tests, and accuracy coverage, including base model support, sound generation, and action modality support. (#3454, #4073, #4102)
• Added DreamZero world-model integration with CFG parallel, OpenPI serving, deployment configs, online examples, OpenPI client helpers, and source-parity tests. (#2162, #3673)
• Added or expanded omni and multimodal model support for MiniCPM-o 4.5, Lance, MOSS-TTS, GLM-TTS, Higgs Audio v2, Covo-Audio-Chat, HiDream-I1-Full, Ming-flash-omni-2.0 image generation, SenseNova U1, and Qwen3-Omni Thinker LoRA for RL training. (#3642, #4067, #3710, #3420, #3141, #3762, #2293, #2572, #2875, #3319, #3915)
• Improved model-family behavior across Qwen-Image, Qwen-Image-Edit, BAGEL, HunyuanImage3, HunyuanVideo 1.5, FLUX.2-dev, LTX-2/LTX-2.3, DreamID-Omni, Helios, Ovis image, MiMo-Audio, and Ming-flash-omni. (#3608, #3219, #3933, #3728, #3857, #3979, #3244, #3621, #3905, #3265, #3470, #3876, #3686, #4080)

Audio, Speech & Omni Production Optimization

• Optimized Qwen3-TTS for high-concurrency serving with precomputed custom voices, ref-context cache, restored cross-request Code2Wav batching, persistent prompt-embedding helpers, reduced CUDA Graph buckets, and compatibility fixes for newer transformers versions. (#3662, #3492, #3322, #3992, #3932, #3880)
• Improved Qwen3-Omni performance and correctness with TTFP optimization, sampling alignment with transformers, prefix-cache correctness, long-output correctness tests, torch.compile accuracy fixes, and streaming-input fixes after the v0.22 rebase. (#4054, #4137, #3665, #3539, #3885, #4085)
• Improved Fish Speech S2 Pro, VoxCPM2, OmniVoice, GLM-TTS, Higgs Audio v2, and MOSS-TTS serving paths through high-concurrency decode work, Triton/CUDA Graph acceleration, voice clone serving, reproducible seeds, nonverbal tags, and broader offline/online examples. (#3773, #3882, #3336, #3668, #3968, #3141, #3762, #3420)
• Added audio SLO metrics, cross-stage transfer metric families, audio streaming continuity metrics, and per-stage/per-replica metric wrapping for upstream vllm:* metrics. (#3576, #3618)

Diffusion, Image & Video Generation

• Added and expanded diffusion parallel execution with Wan2.2 pipeline parallelism, HunyuanImage3 VAE parallelism, HunyuanVideo 1.5 USP plus VAE patch parallel, LTX-2.3 CFG parallel, BAGEL VAE parallel, and HunyuanVideo/HunyuanImage3 NPU performance work. (#2322, #3091, #3979, #3905, #3982, #3178)
• Expanded diffusion acceleration with CacheDiT for Helios, DreamID-Omni, SenseNova U1, and LTX-2; prompt-embedding caching; MagCache; step-wise LoRA; and CacheDiT-related correctness fixes. (#3470, #3265, #3906, #3621, #2962, #1287, #3639, #3219)
• Improved image and video generation correctness and serving behavior across HunyuanImage3, Qwen-Image, Qwen-Image-Edit, Flux2 Klein, GLM-Image, SD3, SenseNova U1, and /v1/videos, including long-prompt/device fixes and safer bf16 video frame conversion before NumPy output. (#4145, #3933, #4074, #3711, #3717, #3451, #3949, #4114)
• Improved diffusion serving and benchmark behavior with endpoint routing for image edits, benchmark endpoint naming, output comparison tooling, performance quality gates, and stage-level benchmark statistics. (#3693, #3137, #3175, #3851, #3628)

Quantization & Memory Efficiency

• Added broader diffusion quantization support, including Wan2.2 W4A16, GLM-Image W4A16, LTX-2 online FP8/INT8, DreamID-Omni online FP8/INT8, NPU MXFP4 online/offline quantization, XPU MXFP8, ModelOpt mixed FP8/NVFP4 and batched ModelOpt FP8 serving support. (#3353, #3059, #3700, #3902, #3578, #3782, #3570, #3943, #4155)
• Added quantization quality and trajectory comparison tooling for diffusion outputs, improved quantization benchmark handling for omni outputs, and expanded quality-gate coverage for FP8 Z-Image and related diffusion tests. (#3175, #3653, #3929)
• Improved memory and cache behavior through Qwen-Image text encoder cleanup, prompt-embedding cache support, custom pipeline sleep memory release fixes, global CUDA graph pool reuse, BAGEL per-step sync removal, and AR prefix hidden-state CPU staging deduplication. (#3608, #2962, #3818, #3361, #3987, #3734)

RL, Serving & Integrations

• Added DreamZero/OpenPI serving and a realtime OpenPI robot serving API, including online DreamZero examples, OpenPI client helpers, connection tests, and serving tests. (#2162, #3673)
• Added Qwen3-Omni Thinker LoRA support for RL training and improved custom pipeline argument handling, sleep/wakeup memory behavior, and multistage deployment coverage. (#3915, #2973, #3818, #3610)
• Improved OpenAI-compatible serving behavior for image edits, speech generation, realtime audio, chat/multistage generation, invalid parameter handling, stream finish reasons, and frontend audio engine errors. (#3693, #2849, #3614, #3374, #3652, #3316)
• Added Yuanrong TransferEngine connector support for NPU and improved connector/runtime infrastructure around chunk transfer, memory pools, local-rank handling, distributed KV flow, and multi-replica GPU device mapping. (#3180, #3569, #3740, #4132)

Platforms, Distributed Execution & Hardware Coverage

• Expanded Blackwell diffusion support with CUDNN attention, FlashInfer attention auto-routing, and SageAttention3 backend support for GB200/B200/RTX 5090/PRO 6000/DGX Spark class systems. (#3079, #3015)
• Improved ROCm coverage with AITER GroupNorm, AITER backend support for ring attention, and v0.22-era ROCm CI fixes. (#3419, #3511, #3946)
• Improved Intel XPU coverage with CosyVoice3 support, MXFP8 support through the vLLM main-repo method, diffusion attention defaults, Docker/CI updates, v0.22 rebase fixes, and Wan2.2 S2V RoPE/cache_dit optimization. (#2325, #3782, #3525, #3675, #4059, #4062)
• Improved Ascend NPU coverage with Wan2.2 MXFP4 quantization, HunyuanImage3 FA-FP8, GLM-Image stage configs and HCCL runtime environment fixes, Yuanrong connector support, sampler/runtime fixes, and v0.22 ModelRunner updates. (#3578, #3540, #3235, #3180, #3517, #4130)

CI, Benchmarks & Documentation

• Unified the release pipeline around a NIGHTLY=1 option, added x86_64/aarch64 image builds, enabled twine upload to PyPI, refreshed Docker bases, and updated CUDA/ROCm/XPU installation docs for the current release line. (#3428, #3667, #3859, #4059)
• Added or improved reliability, invalid-parameter, nightly parity, accuracy, and performance coverage for Cosmos3, DreamZero, HunyuanImage3, HunyuanVideo 1.5, GLM-Image, BAGEL, VoxCPM2, Qwen3-Omni, Wan2.2, MOSS-TTS, and multistage deployment. (#3454, #2162, #3790, #3852, #3451, #2175, #4055, #3729, #4097, #3610)
• Improved benchmarking and observability infrastructure with audio SLOs, cross-stage transfer metrics, modality metrics, Prometheus/stat-logger tests, audio-streaming continuity metrics, diffusion benchmark endpoint routing, optional baseline assertion...

v0.22.0rc1

Highlights

This release candidate features 179 commits from 88 contributors, including 28 new contributors.

vLLM-Omni v0.22.0rc1 is a broad release candidate focused on aligning with the vLLM 0.22 release line, expanding speech and diffusion model coverage, and improving production serving for multistage omni workloads. It strengthens the runtime around stage orchestration, async audio and streaming paths, diffusion caching, quantization, and multi-backend deployment. This release candidate is intended to validate the vLLM 0.22 rebase and the new model/runtime coverage before the final cut.

Key Improvements

• Aligned with the vLLM 0.22 release line, including the main rebase and release pipeline improvements for image builds and PyPI publishing. (#3891, #3428, #3667)
• Expanded speech and omni model coverage, adding GLM-TTS, Higgs Audio v2, Qwen3-Omni Thinker LoRA support for RL training, voice-clone serving for OmniVoice, and new deployment recipes for Fish Speech S2 Pro and Qwen Image Edit. (#3141, #3762, #3915, #3668, #3323, #3684)
• Broadened image and video generation support, with HiDream-I1-Full, Ming-flash-omni-2.0 image generation, improved Qwen/Hunyuan/GLM/BAGEL paths, and stronger support for HunyuanVideo 1.5. (#2572, #2875, #3933, #3728, #3979)
• Improved diffusion acceleration and parallel execution, including Wan 2.2 pipeline parallelism, HunyuanImage3 VAE parallelism, step-wise LoRA, CacheDiT coverage, prompt-embedding cache, and MagCache. (#2322, #3091, #3639, #3470, #3265, #3906, #2962, #1287)
• Made TTS serving more production-ready, with Qwen3-TTS high-concurrency optimization, precomputed custom voices, ref-context caching, Code2Wav batching and compatibility fixes, Fish Speech S2 Pro serving improvements, and OmniVoice CUDA Graph/Triton acceleration. (#3662, #3492, #3322, #3880, #3932, #3773, #3336)
• Expanded quantization and hardware coverage, including W4A16, online FP8/INT8, MXFP4, MXFP8, ModelOpt mixed FP8/NVFP4, Blackwell diffusion attention backends, ROCm AITER support, Intel XPU coverage, and Ascend NPU improvements. (#3353, #3059, #3700, #3902, #3578, #3570, #3782, #3079, #3015, #3419, #3511, #2325)

Core Architecture & Runtime

• Integrated OmniCoordinator into the stage engine pipeline and improved async audio/chunk request handling, including correct completion behavior without pad-token injection, audio streaming input for async chunks, and request-id aliasing fixes. (#3569, #3614, #3613, #3953)
• Hardened diffusion and multistage lifecycle behavior with worker dead detection, cleanup fixes, subprocess exit handling, SIGINT cleanup for NCCL/ZMQ resources, and safer master-port selection for parallel launches. (#3214, #3494, #3751, #3872, #3803)
• Improved scheduling and cache correctness across Qwen3-Omni, prefix caching, token history, offline/online alignment, and distributed stage-0 multimodal cache routing. (#3681, #3665, #3506, #3740, #3885)
• Refined configuration and stage startup behavior, including recursive engine-arg merging, deploy-config field allowlisting, and migration of Ming-flash-omni image-generation deploy configs. (#3009, #3483, #3975)

Model Support

• Added GLM-TTS and Higgs Audio v2 support with offline/online serving examples, deploy configs, tests, and client/demo coverage. (#3141, #3762)
• Added HiDream-I1-Full and Ming-flash-omni-2.0 image-generation support, plus recipes and deploy guidance for Qwen Image Edit, GLM-Image, Helios, Fish Speech S2 Pro, and Voxtral TTS. (#2572, #2875, #3684, #2950, #3114, #3323, #3498)
• Added Qwen3-Omni Thinker LoRA support for RL training and improved long-output correctness, streaming helpers, and torch.compile accuracy behavior. (#3915, #3539, #3885)
• Improved existing model paths across Qwen-Image, Qwen-Image-Edit, BAGEL, HunyuanImage3, Ovis image, SenseNova U1, LTX-2.3, MiMo-Audio, and Ming-flash-omni. (#3608, #3219, #3933, #3728, #3857, #3876, #3691, #3854, #3686, #3975)

Audio, Speech & Omni Production Optimization

• Optimized Qwen3-TTS for high-concurrency serving with precomputed custom voices, ref-context cache, cross-request Code2Wav batching, persistent prompt-embedding helpers, reduced CUDA Graph buckets, and compatibility fixes for newer transformers versions. (#3662, #3492, #3322, #3992, #3932, #3880)
• Improved speech serving correctness and streaming behavior, including speech-endpoint finish reasons, async chunk continuity metrics, uploaded-voice handling, short Code2Wav chunk handling, and prompt-length estimation for Qwen3-TTS reference codes. (#2849, #3618, #3523, #3687, #3940)
• Improved Fish Speech S2 Pro and OmniVoice production paths with high-concurrency serving, Triton kernel fusion, CUDA Graph acceleration, voice clone support, reproducible seed support, and removal of hardcoded default voice assumptions in examples. (#3773, #3336, #3668, #3829)
• Stabilized MiMo-Audio and shared TTS components, including voice instability fixes, batching follow-ups, common activation refactors, bf16/fp16 Triton fixes, and reusable talker/model runner paths across GPU and NPU. (#3686, #3817, #3886, #3472, #3476)

Diffusion, Image & Video Generation

• Added and expanded diffusion parallel execution with Wan 2.2 pipeline parallelism, HunyuanImage3 VAE parallelism, LTX-2.3 CFG parallel support, and HunyuanVideo 1.5 USP plus VAE patch parallel support. (#2322, #3091, #3905, #3979)
• Expanded diffusion acceleration with CacheDiT for Helios, DreamID-Omni, SenseNova U1, and LTX-2, prompt-embedding caching, MagCache, and step-wise LoRA support. (#3470, #3265, #3906, #3621, #2962, #1287, #3639)
• Improved image/video generation correctness and performance across HunyuanImage3, HunyuanVideo, Qwen-Image, Qwen-Image-Edit, BAGEL, Flux2 Klein, GLM-Image, LTX-2.3, SenseNova U1, and Ovis. (#3630, #3694, #3768, #3857, #3844, #3219, #3933, #3680, #3711, #3717, #3059, #3854, #3691, #3876)
• Improved diffusion serving and benchmark behavior by routing image edit workloads to the edits endpoint, renaming the diffusion benchmark backend to endpoint, adding output comparison tools, and strengthening diffusion performance optimization quality gates. (#3693, #3137, #3175, #3851)

Quantization & Memory Efficiency

• Added broader diffusion quantization support, including Wan2.2 W4A16, GLM-Image W4A16, LTX-2 online FP8/INT8, DreamID-Omni online FP8/INT8, NPU MXFP4 online/offline quantization, XPU MXFP8, and ModelOpt mixed FP8/NVFP4 for image generation. (#3353, #3059, #3700, #3902, #3578, #3782, #3570)
• Added quantization quality and trajectory comparison tooling for diffusion outputs, improved quantization benchmark handling for omni outputs, and expanded quality-gate coverage for FP8 Z-Image and related diffusion tests. (#3175, #3653, #3929)
• Improved memory behavior through Qwen-Image text encoder cleanup, prompt-embedding cache support, custom pipeline sleep memory release fixes, and CUDA graph pool reuse in VoxCPM2 and Ming-flash-omni paths. (#3608, #2962, #3818, #3361)

RL, Serving & Integrations

• Added Qwen3-Omni Thinker LoRA support for RL training and improved custom pipeline argument handling, sleep/wakeup behavior, and multistage serving tests. (#3915, #2973, #3818, #3610)
• Improved OpenAI-compatible serving behavior for image edits, speech generation, realtime and chat paths, server-control reliability, invalid parameter handling, and frontend audio engine error handling. (#3693, #2849, #3316, #3652, #3680)
• Added Yuanrong TransferEngine connector support for NPU and improved connector/runtime infrastructure for chunk transfer, memory pools, local-rank handling, and distributed KV flow. (#3180, #3569, #3740)

Platforms, Distributed Execution & Hardware Coverage

• Expanded Blackwell diffusion support with CUDNN attention, FlashInfer attention auto-routing, and SageAttention3 backend support for GB200/B200/RTX 5090/PRO 6000/DGX Spark class systems. (#3079, #3015)
• Improved ROCm coverage with AITER GroupNorm and AITER backend support for ring attention, plus ROCm CI/version updates. (#3419, #3511, #3659)
• Improved Intel XPU coverage with CosyVoice3 support, MXFP8 support through the vLLM main-repo method, diffusion attention defaults, Docker/CI updates, and XPU-specific test fixes. (#2325, #3782, #3525, #3675, #3718, #3761, #3994)
• Improved Ascend NPU coverage with Wan2.2 MXFP4 quantization, HunyuanImage3 FA-FP8, GLM-Image stage configs and HCCL runtime environment fixes, Yuanrong connector support, and sampler/runtime fixes. (#3578, #3540, #3235, #3180, #3517)

CI, Benchmarks & Documentation

• Unified the release pipeline around a NIGHTLY=1 option, added x86_64/aarch64 image builds, enabled twine upload to PyPI, and refreshed Docker bases for the current release line. (#3428, #3667, #3859)
• Added or improved reliability, invalid-parameter, nightly parity, accuracy, and performance coverage for Qwen-Image, Qwen-Image-Edit, HunyuanImage3, HunyuanVideo 1.5, BAGEL, VoxCPM2, Qwen3-Omni, Wan2.2, and multistage deployment. (#3502, #3652, #3670, #3795, #3852, #3849, #2175, #3864, #3729, #3610)
• Improved benchmarking infrastructure with audio-streaming continuity metrics, diffusion benchmark endpoint routing, optional baseline assertions, perf JSON updates, and repo-wide benchmark documentation. (#3618, #3693, #3695, #1939)
• Refreshed docs and recipes for quantization, diffusion performance, CosyVoice3 online serving, GLM-Image, Helios, Qwen Image Edit, VACE, and CUDA image commands. (#3764, #3851, #3748, #2950, #3114, #3684, #3584, #3836)

Note

• The release includes compatibility work for newer dependency versions, including LTX-2 connector handling with diffusers==0.38.0 and Qwen3-TTS Code2Wav compatibility with `transf...