Skip to content

Diffusers 0.40.0: New pipelines, tensor-parallel support, improved CLI, and more

Latest
Latest

Choose a tag to compare

View all tags
@sayakpaul sayakpaul released this 20 Aug 14:53
· 2 commits to main since this release
v0.40.0
d035dcd

Tip

This release features several new pipelines, including LTX2.5, MiniMax H3, and Wan Animate 2. We're also graduating Modular Diffusers out of the experimental phase and announcing its stable support. Additionally, this release includes minimal support for tensor-parallel. There's a lot more that went down in this release. So, please consult the notes for details.

New Pipelines

MiniMax-H3

MiniMax-H3 generates video and its soundtrack together. A single transformer denoises one packed sequence containing the text conditioning, the conditioning media, and the target video and audio latents — there is no separate vocoder and no post-hoc audio pass. Its conditioner is a Qwen3VLForConditionalGeneration whose unnormalized 50th-decoder-layer hidden state is read instead of the last one.

MiniMax-H3 is integrated as Modular Diffusers blocks only — MiniMaxH3Blocks and their MiniMaxH3ModularPipeline are the whole integration. The conversion ships both checkpoint partitions in one repository and exposes three workflows (t2va, fl2va, ref2va) that can be pruned at from_pretrained time so only that task's components are declared and downloaded.

MiniMax Music 3

MiniMax Music 3 produces complete songs up to five minutes long from lyrics and a music description, with expressive vocals and long-range structure. It is a hybrid of an autoregressive and a diffusion stage: an 8B Qwen3-based global language model predicts one semantic audio token per frame while a small depth decoder fills in seven residual RVQ codebooks, and their fused hidden states condition a 2.4B flow-matching transformer that produces Flow-VAE latents in overlapping chunks. A DAC-style decoder turns the latents into 44.1 kHz stereo audio.

Stable Audio 3

Stable Audio 3 is a text-to-audio model from Stability AI that generates high-quality stereo audio at 44.1 kHz. It uses a rectified-flow DiT conditioned on a frozen T5Gemma text encoder (via cross-attention) and on duration (a float embedded by StableAudio3DurationEmbedder and used for adaptive layer norm), and decodes with the SAME (Semantically-Aligned Music Encoder) autoencoder, AutoencoderSAME.

Three pipelines ship: StableAudio3Pipeline, StableAudio3AudioToAudioPipeline, and StableAudio3InpaintPipeline.

Thanks to @buffett0323 for the contribution (#14119).

LTX-2.5

LTX-2.5 reuses the existing LTX2Pipeline / LTX2VideoTransformer3DModel / AutoencoderKLLTX2Video classes — there is no separate pipeline class. The user-visible difference is the text encoder: LTX-2.5 is paired with a Gemma 4 (gemma4_unified) checkpoint, loaded automatically from a converted LTX-2.5 repo.

Lightricks/LTX-2.5-Diffusers ships both the distilled DiT (transformer/) and the full/SFT DiT (transformer_full/), plus everything two-stage generation needs. Alongside the checkpoint support, this release adds:

  • LTX2VideoDiffusionDecoderModel and LTX2VideoDiffusionDecodePipeline — a second video decoder over the same latent space, so latents are interchangeable between decoders.
  • A duration_head that predicts shot length from the text-connector output, so num_frames is auto-predicted by default when the loaded pipeline has one.
  • Prompt enhancement through a separate off-the-shelf google/gemma-4-E2B-it checkpoint (enable_prompt_enhancement=True).
  • LTX25AutoBlocks for Modular Diffusers (#14453).

Wan-Animate-2

Wan-Animate-2 by the Alibaba Wan Team animates a reference character image with the motion of a driving video. The driving video is processed in fixed-length segments: each segment runs a reference-extraction pass that caches the driving segment's K/V in every transformer layer, denoises against that cache, and is decoded inside the loop, because the next segment conditions on the previous segment's decoded tail frames.

Two presets are available — the base checkpoint samples with classifier-free guidance, and the distilled checkpoint samples in few steps without it. Guidance is owned by the pipeline's guider component, so there is no guidance_scale argument.

Thanks to @kelseyee for authoring the integration (#14413).

JoyAI-Image-Edit-Plus

JoyAI-Image-Edit-Plus extends the JoyAI-Image family (an 8B MLLM paired with a 16B MMDiT) to multi-image instruction-guided editing. It accepts 1–5 reference images plus a text instruction and composes elements from the references into a new image.

Thanks to @tangyanf for the contribution (#14032).

Cosmos 3 follow-ups

Cosmos 3 landed in 0.39.0 and gets substantially more coverage in this release:

Thanks to @yzhautouskay and @atharvajoshi10 for the contributions.

Modular Pipeline Support

Modular Diffusers is no longer marked experimental (#14525) — the API warning has been dropped.

Important

Please try out Modular Diffusers and let us know about your feedback!

Core Library

Tensor parallelism

Tensor-parallel inference is now supported for model inference on CUDA and AWS Neuron (Trainium/Inferentia), exposed through the same public API already used for context parallelism:

from diffusers import TensorParallelConfig

pipe.transformer.enable_parallelism(config=TensorParallelConfig(mesh=tp_mesh))

Sharding is model-agnostic and driven from a flat _tp_plan, which has been added to the FLUX.1, FLUX.2 and Qwen-Image DiTs. Check out the docs for more details.

New quantization backends

Attention, kernels and offloading

CLI

The diffusers-cli was reworked for agentic use (#13966) and then cleaned up (#14381): modular_model_index.json is written when saving a custom block so ModularPipeline.from_pretrained can load and run custom blocks as pipelines, auto CPU offload works for Modular pipelines, workflow can be passed to Modular pipelines, and output saving handles multimodal output (e.g. LTX video frames + audio) and batched video.

Skills are now installed through the CLI rather than the Makefile (#14454):

diffusers-cli skills list
diffusers-cli skills add <skill name>

🚨 Breaking changes and deprecations

  • JAX/Flax support has been removed (#14169) — roughly 12k lines deleted. All Flax* classes and the flax extras are gone.
  • Mixed-rank LoRAs without alpha keys now load at their intended scale (#14409). Previously get_peft_kwargs took lora_alpha from the first entry of the rank dict and never revisited it, so every module whose rank differed from the first key's rank got an arbitrary, key-order-dependent scale. Ranks are now mirrored into the alphas when a checkpoint brings no alpha information (the diffusers/PEFT convention: alpha == rank, scale 1.0). Adapters with a declared alpha keep it, and uniform-rank adapters are unaffected. Existing mixed-rank, no-alpha LoRAs will now produce different (correct) results.
  • torch_dtype is deprecated in favour of dtype (#14205, #14313), following transformers. torch_dtype still works but warns, and will be removed in 1.0.0. A torch.dtype alias was added for the docs.
  • DDUF is deprecated (#14141). dduf_file warns and will be removed in 0.41.0.

Security

Training

LoRA

Notable fixes

Tests and infrastructure

A large chunk of this release is test modernization: pipeline tests continue migrating to the new mixin structure (Wan, Qwen-Image, FLUX.2, CogVideoX, Stable Diffusion, and the LoRA pipeline tests), tests/others, training tests, and attention-processor tests moved to pytest, model-level and pipeline-level quantization tests were standardized, and an output_shape property was introduced in the pipeline tests. The agent-facing docs and skills under .ai/ were expanded to cover tests, model implementation, and blockset conventions.

All commits

Contributors

kashif, apolinario, and 40 other contributors
Assets 2
Loading