Skip to content

Release: v5.16.0

Choose a tag to compare

View all tags
@Cyrilvallez Cyrilvallez released this 26 Aug 12:35
· 11 commits to main since this release
v5.16.0
93d1bcf
This commit was signed with the committer’s verified signature.
Cyrilvallez Cyril Vallez
SSH Key Fingerprint: O743lyKbz6cYCDq5uTRdzyLHAo4JjdtgAF9Q3yVhAQA
Verified
Learn about vigilant mode.

Release v5.16.0

New Model additions

Qwen4-Exp

image

Qwen4-Exp builds on Qwen3.5's hybrid text and multimodal architecture with three key components: GatedResidual (GR), Qwen Sparse Attention (QSA), and Per-Layer Embedding (PLE).

GR is a Qwen-developed residual architecture that combines Hyper-Connection with GatedNorm. It mixes multiple residual streams with fine-grained elementwise gating before each attention and Mixture-of-Experts (MoE) block, then controls how much of the block output is injected back into each stream.

QSA uses multiple query heads to score compressed key blocks, selects the most relevant contiguous token blocks, and keeps the incomplete trailing block uncompressed. This block-level selection reduces indexing overhead and improves memory locality for long sequences. Combined with Gated DeltaNet, QSA makes Qwen4-Exp the first hybrid architecture to integrate linear and sparse attention, substantially improving inference efficiency for long-context workloads.

PLE enriches selected decoder layers with layer-specific lexical features derived from hashed token n-grams and a dilated depthwise convolution.

Links: Documentation

GraniteSpeech5

image

Granite Speech 5.0 Turbo CTC is a lightweight (~470M parameters) conformer encoder for automatic speech recognition, trained with Connectionist Temporal Classification (CTC) on BPE targets. It is a fast, encoder-only member of the Granite Speech family: transcription requires a single forward pass followed by greedy CTC decoding, with no autoregressive decoder.

Architecturally, it extends the Granite Speech conformer CTC encoder with:

  1. Frame stacking + block-wise time subsampling: the feature extractor stacks pairs of log-mel(+delta) frames (2x), and the first two conformer blocks each subsample time by 2 through a stride-2 depthwise convolution (with a mean-pooled residual), for a total 8x time reduction at 10 ms mel hop.

  2. Block attention with Shaw's relative positional embeddings: attention is computed over fixed-size blocks (the sequence is right-padded to a whole number of blocks, with padded frames masked out), using separate bias-free query/key/value projections.

  3. Self-conditioned CTC: the CTC posteriors of the middle layer are projected and fed back into the hidden states, and the CTC head is shared between this mid-layer self-conditioning and the final prediction.

Links: Documentation

Step3p7

Step-3.7-Flash was proposed in Step 3.7 Flash by StepFun. It is a 198B-parameter sparse Mixture-of-Experts vision-language model, pairing a 196B-parameter MoE language backbone with a 1.8B-parameter vision encoder for native image understanding.

StepFun hasn't published a technical report for Step-3.7-Flash, so the details below are drawn from the released checkpoint's configuration rather than a paper.

  • Sparse MoE decoder: all but the first 3 decoder layers route through a MoE block of 288 routed experts (top-8 per token) plus a single shared expert. The router scores experts with a sigmoid and a learned per-expert bias instead of an auxiliary load-balancing loss, the same strategy as DeepSeek-V3.
  • Gated attention: each attention layer adds an extra projection whose sigmoid output gates the attention output per head, before the output projection — the same Gated Attention mechanism used in Qwen3-Next. A subset of layers use fewer heads and a sliding window instead of full attention.
  • Multi-token prediction: some checkpoints ship extra decoder layers trained for multi-token prediction, which [~GenerationMixin.generate] can use for speculative decoding via use_mtp=True.
  • Vision encoder: a SigLIP-style ViT with 2-D rotary position embeddings and a learned per-layer scale on the attention and MLP branches. Its output is downsampled 4x by two stride-2 convolutions before a linear projector maps it into the text model's hidden size.
  • Dynamic image tiling: instead of a fixed tile grid, the image processor picks its tiling window from each image's own aspect ratio, producing one downscaled global view plus zero or more local high-resolution crops per image.

Links: Documentation

CohereCompass

CohereCompass is the base architecture for small, specialized (vision-)language models trained by Cohere.

Links: Documentation

ESMC and ESMFold2

ESMC and ESMFold2 are new state-of-the-art protein language and folding models from BioHub. ESMC is trained with a masked language modeling objective, and it can be easily transferred to sequence and token classification tasks for proteins. Checkpoints exist in various sizes, from 300M parameters up to 6B parameters. It works as a drop-in replacement for older ESM-2 and ESM-3 models, with significantly higher accuracy.

ESMFold2 is a state-of-the-art protein folding model which produces high accuracy predictions. It uses an iterated diffusion approach that is significantly different from the original ESMFold, offering huge improvements in accuracy for more complex structures.

Links: Documentation ESMC, Documentation ESMFold2

Breaking changes

The legacy tensor-parallel implementation has been replaced with a DTensor-native backend, so users relying on the previous TP API for inference or training must migrate to the new DTensor-based interface.

attn_implementation="sdpa" dispatch is now properly supported for wav2vec2_conformer, wav2vec2-bert, and SeamlessM4T/v2 models, which may change initialization behavior for users who previously worked around this limitation.

FuyuProcessor no longer returns the image_patch_indices output, so any code that depends on this field must be updated to remove references to it.

Cache

Several cache-related bugs were fixed in this release, including an off-by-one error in the sliding window cache, Whisper speculative decoding cache corruption, CpmAnt use-cache failures, Qwen2.5-Omni/Qwen3-Omni-MoE generation with compilable caches, and compressed-tensors loading for KV-cache-only quantized models. Documentation was also added for cache token removal using negative values, and per-layer cache configuration support (allowing models to use different cache settings per layer) was introduced.

Generation

This release fixes several generation bugs across multiple models, including Whisper speculative decoding issues (UnboundLocalError, cache corruption, speed regression, and left-padded batch position IDs), broken image generation in Emu3, garbage output in OLMo/GPTNeoX, and Qwen2.5-Omni/Qwen3-Omni-MoE generation with compilable caches. Additionally, logit distributions for candidate generators using sampling are now aligned by returning logits after applying logit processors.

  • [Whisper] Fix speculative decoding: preserve cleared suppress tokens through super().generate() (#48108) by @ydshieh in [#48108]
  • [Whisper] Fix decoder position IDs for left-padded batches in longform generation (#48028) by @ydshieh in [#48028]
  • [serge] Fix 2 integration tests for model generation failing with import_or_config (other (2)) (#48061) by @sergereview[bot] in [#48061]
  • Align logit distributions for CandidateGenerators using sampling (#48007) by @Cyrilvallez in [#48007]
  • [GPTNeoX] Fix post_processor not overridden when loading from pretrained (OLMo garbage generation) (#47988) by @ydshieh in [#47988]
  • [emu3] 🦮 Black Labrador is back! Fix image generation broken since #37033 (#47948) by @ydshieh in [#47948]

Attention

Several attention-related bug fixes were made in this release, including correcting a SigLIP2 documentation typo, fixing Flash/SDPA attention dispatch tests for xcodec2 and ROCm RDNA GPUs, resolving a GPT2 cross-attention mask being silently discarded, and enabling SDPA support declaration in TimmWrapper. Per-layer cache configuration and attention-mask selection support was also introduced, allowing models with heterogeneous layer configurations to use distinct sliding_window, attention_chunk_size, and number_of_conv_states values per layer.

Quantization

Quantization improvements include adding NVFP4 quantization support via HF kernels (enabling on-the-fly BF16 weight quantization with ~50% memory reduction), and fixing several bugs: reverting a regression in is_quantization_compressed that caused incorrect module layouts for packed-format checkpoints, fixing CLIP weight initialization failures with quantized checkpoints, and restoring KV-cache quantization setup for KV-cache-only quantized models.

  • Revert "[Quantization]: Refactor is_quantization_compressed for format-based detection" (#48072) by @subin9 in [#48072]
  • feat: add nvfp4 quantization (#47883) by @drbh in [#47883]
  • [DeepSeekV2] Fix integration tests OOM: use device_map=auto instead of 8-bit quantization (#47991) by @ydshieh in [#47991]
  • Fix CLIP _init_weights when a child module carries quantized weights (#47921) by @Bluear7878 in [#47921]

Parallelization

Introduced a naive pipeline parallel inference engine supporting tied/untied weight embeddings with seamless generate() integration, while restoring backward compatibility for the tensor-parallel API with a deprecation cycle for tp_plan in from_pretrained(). Additionally fixed a model parallel bug in the BLT model affecting beam search.

Kernels

Kernel support was improved with documentation updates highlighting supported models, a fix for export crashes on kernel-decorated functions by adding a is_torchdynamo_exporting guard, and the default Flash Attention 2 hub kernel version was bumped to v3 to resolve compatibility issues with newer PyTorch versions.

Bugfixes and improvements

Significant community contributions

The following contributors have made significant changes to the library over the last release:

  • @tarekziade
    • CI: gate the hunyuan-moe slow test (#48330)
    • CI: fix muse OOMs (#48284)
    • ignore mlinter ci file (#48267)
    • Post two CI badges on a PR: CPU PR CI and GPU run-slow (#48190)
    • Assign a reviewer even when a codeowner has left, and route models by modality (#48085)
    • Fix gpt_oss runs on GPU (#48118)
    • Moving mlinter to 0.1.4 (#47918)
    • unpin pytest in the examples_torch deps (#48023)
    • Let the GPU verify caller turn on the memory probe (#48001)
    • Potential fix for code scanning alert no. 267: Artifact poisoning (#47949)
    • Scan a diff in trufflehog, not the whole repo history (#47945)
  • @dkrisman
    • Add an opt-in per-frame pixel cap (cap_pixels_per_frame) to the Qwen3-VL video processor (#48071)
  • @jiqing-feng
    • Cpmant fix use cache (#48013)
    • [xcodec2] Fix flex attention and flash dispatch tests (#48244)
    • Always tie embeddings for LongT5 and Pop2Piano (#47620)
    • Fix CpmAnt loading: size lm_head to vocab_size (#48012)
    • Fix Qwen2.5-Omni / Qwen3-Omni-MoE generation with a compilable cache (#47872)
    • Bump default flash-attn2 hub kernel version to v3 (#47863)
    • Declare sdpa support in TimmWrapper (#47939)
  • @ydshieh
    • Fix AutoTokenizer returning TokenizersBackend for DeepSeek-R1-Distill-Qwen models (#48211)
    • [Gemma4] Investigate flaky test_generation_beyond_sliding_window_1_eager (#48236)
    • [Gemma4] Fix stale expected values in integration tests (#48233)
    • [TableTransformer, PI0] Fix stale expected values and OOM in integration tests (#48198)
    • Fix stale expected values in integration tests (cuda sm_86 / Aug04 regressions) (#48171)
    • [EsmFold2] Fix stale expected distogram logit values (#48182)
    • Fix Apr 05 integration test regressions (cuda sm_86) (#48170)
    • Fix integration test expected values for cuda sm_86 (Mar 15 regressions) (#48168)
    • [LLaVA] Fix pixtral integration tests for cuda sm_86 (#48166)
    • [Qwen2.5-Omni] Update stale expected values for cuda sm_86 (#48164)
    • [Mistral3] Fix batched integration tests: padding_side=left + update expected values (#48161)
    • [InternVL] Fix stale expected values for Llama integration tests (cuda sm_80) (#48153)
    • Retry transient network errors (RemoteDisconnected) in github_utils (#48124)
    • [Whisper] Fix speculative decoding: preserve cleared suppress tokens through super().generate() (#48108)
    • Enable mlinter findings artifact for inline PR reviews (#48117)
    • Delete old mlinter review comments before posting new ones (#48107)
    • Accept artifact dir as argument in post_mlinter_review.py (#48106)
    • Fix mlinter artifact path (#48088)
    • [Whisper] Fix decoder position IDs for left-padded batches in longform generation (#48028)
    • [PE] Skip test_sdpa_can_dispatch_on_flash for TimmWrapper-backed models (#48064)
    • [Gemma3] Update integration test expected values for A10G (#48036)
    • [CircleCI] Enable CI for private forks, no-op for public repo (#48056)
    • [Gemma3n] Update integration test expected values for A10G + torch 2.13 (#48035)
    • [Florence2] Fix two integration test failures caused by torch 2.13 and auto-dtype (#48031)
    • [ModernVBERT] Fix integration test checkpoint (404 since April) (#48009)
    • [Whisper] Fix speculative decoding: UnboundLocalError, cache corruption, and speed regression (#48000)
    • [Whisper] Fix batch decode_with_timestamps in WhisperTokenizer.decode() (#47997)
    • [Whisper] Fix integration test failures on A10G (dtype, stale values, API changes) (#47995)
    • [DeepSeekV2] Fix integration tests OOM: use device_map=auto instead of 8-bit quantization (#47991)
    • [OLMoE] Update expected logits for A10G and add torch.no_grad() (#47989)
    • [OLMo] Fix OOM in logits tests by adding torch.no_grad() (#47986)
    • [GPTNeoX] Fix post_processor not overridden when loading from pretrained (OLMo garbage generation) (#47988)
    • [AXK1] Fix expected logits for CUDA A10G (#47980)
    • [Gemma] Update expected values for A10G (#47976)
    • [emu3] 🦮 Black Labrador is back! Fix image generation broken since #37033 (#47948)
    • Remove stale (None, None) fallback in qwen2_5_vl batch_different_resolutions test (#47972)
    • [tests] Fix expected output for Qwen2.5-VL batch_wo_image on CUDA (#47968)
    • CI: add vllm-test-init and vllm-test-transformers jobs on dedicated runners (#47934)
    • fix: correct checkpoints, config annotations, and create_dummy_models improvements (#47902)
    • Add CI_CPU_MEMORY_LIMIT_GB to check_failed_tests workflow (#47884)
    • Use tiny Hub checkpoint in Qwen3ASR processor test (#47833)
  • @eustlb
  • @eladsegal
    • Fix DynamicCache reconstruction during ExecuTorch export (#47900)
    • Support per-layer cache configuration and attention-mask selection (#47901)
  • @YangKai0616
    • 🚨[wav2vec2] Support attn_implementation=sdpa dispatch (#46196)
  • @drbh
    • feat: add nvfp4 quantization (#47883)
  • @Priyans-Lathiya
    • docs: use relative paths for README language menus and add fa/ro entries (#47777)
  • @itazap
    • [new model] step 3.7 (#46658)
  • @calpt
    • [CohereCompass] Minor docs fixes (#47903)
    • Add CohereCompass modeling (#47878)

Contributors

tarekziade, dkrisman, and 49 other contributors
Assets 2
Loading