Skip to content

Release v5.24.0

Latest

Choose a tag to compare

@Cyrilvallez Cyrilvallez released this 21 Nov 12:28
· 0 commits to main since this release
eeee110

Release v5.24.0

New Model additions

Inkling 2 Mini

image

Inkling 2 Mini is a multimodal model from Thinking Machines Lab that accepts text and image inputs and generates text outputs. It is a smaller sibling of Inkling 2 Small, added in v5.23.0. It has 24 decoder layers instead of 48, and compensates with a looping mechanism in which layers 6 to 17 run three times. A learned message-passing mechanism mixes the residuals of the previous pass into the attention and mixture-of-experts inputs of each loop layer. The decoder has a hidden size of 2304 and routes each token to 8 of 128 experts. The image encoder, tokenizer and processor are the same as Inkling 2 Small's.

A research checkpoint (TML-d24-varloop-research) has the same architecture but supports a variable number of loop passes. Fewer passes are faster, and more passes improve quality. Both checkpoints contain NVFP4 and MXFP8 quantized layers, which are dequantized at load time.

Links: Documentation | Blog Post | HF Blog Post

Apertus 1.5

Apertus 1.5 is a multimodal model from the Swiss AI Initiative that accepts text, images and audio and generates text. It extends the Apertus language model with the EMU3.5 vision tokenizer for images and WavTokenizer for audio. The EMU3.5 tokenizer is an image-only VQGAN with the same encoder as Chameleon's, and it assigns codes by the largest inner product with the codebook. The media tokenizers turn images and audio into discrete tokens, which the language backbone processes alongside text in a shared input vocabulary. The output vocabulary contains only text tokens, so image and audio generation are not supported.

Links: Documentation

WavTokenizer

WavTokenizer encodes 24 kHz mono audio into discrete tokens using a single codebook, at 40 or 75 tokens per second depending on the checkpoint, and reconstructs audio from those tokens. WavTokenizerModel encodes and decodes, and WavTokenizerEncoderModel only encodes. Speech, music/audio and unified checkpoints are available. The 40-token-per-second variant is the audio tokenizer of Apertus 1.5. Training the codec is not supported.

Links: Documentation | Paper

Dory

Dory is a looped (latent-recurrent) language model from NVIDIA. It uses a normalized-GPT architecture with interleaved sliding-window and full attention layers. A shared block of layers is applied repeatedly, so Dory can spend more compute per token. The loop count can be changed at inference time on the same checkpoint to trade speed against quality.

Links: Documentation | Paper (nGPT)

HY4-VL

HY4-VL (Hunyuan V4 Vision-Language) combines the HYV4 mixture-of-experts text backbone with a vision encoder for image-text and video-text generation. It supports dynamic-resolution visual inputs and uses multimodal rotary position embeddings (mRoPE) to represent temporal and spatial positions. It builds on the Qwen2-VL design and shares its vision patch merger with HunYuan-VL.

Links: Documentation

Omnilingual ASR

Omnilingual ASR is a suite of multilingual speech-recognition models from Meta FAIR that covers more than 1,600 languages. Each model pairs a Wav2Vec2-style audio encoder with one of two heads:

  • CTC (OmniASRForCTC): a linear projection to the vocabulary, decoded non-autoregressively with greedy CTC.
  • Audio language model (OmniASRForConditionalGeneration): audio embeddings are projected into a Llama decoder that generates the transcription. It optionally takes a language code (e.g. "eng_Latn"), which generally improves transcription quality.

Both variants come in 300M, 1B, 3B and 7B sizes.

Links: Documentation | CTC Documentation | Paper

Lara Mini

Lara Mini is a multilingual base language model from Translated with 7.2B total and 0.6B active parameters per token. It is a hybrid of linear-time recurrent layers, sliding-window attention and a few global attention layers, each followed by a fine-grained mixture of experts. The weights are released under the CC BY-NC-SA 4.0 license (non-commercial use) in a gated repository.

Links: Documentation | Paper

SarvamMLA

SarvamMLA is the architecture of Sarvam-105B, a 105B-parameter mixture-of-experts language model from Sarvam AI. It combines Multi-head Latent Attention (MLA) with sparse MoE routing: 128 routed experts with 8 active per token, plus 1 shared expert. It uses sigmoid routing with group-based top-k selection and YaRN rotary embeddings for contexts up to 131K tokens. The architecture is the same as DeepSeek-V3, with a custom configuration.

Links: Documentation

SarvamMoE

SarvamMoE is the architecture of Sarvam-30B, Sarvam AI's 30B-parameter mixture-of-experts model. It uses grouped-query attention with QK normalization, and 128 routed experts with 6 active per token, plus 1 shared expert. The architecture is the same as GLM-4-MoE, so the auto classes load it with the GLM-4-MoE implementation and only SarvamMoEConfig is specific to SarvamMoE.

Links: Documentation

EXAONE

EXAONE covers the EXAONE 3.0, EXAONE 3.5 and EXAONE Deep language models from LG AI Research. Their architecture is the same as Llama, so the auto classes load them with LlamaModel and LlamaForCausalLM, and only ExaoneConfig is specific to EXAONE. The released checkpoints are renamed to the Llama names when loading and back when saving, so they no longer need trust_remote_code=True.

Links: Documentation

Model updates

  • Ling 3.1 Flash now loads natively through BailingHybrid, like Ling 3.0 in v5.20.0 (#49737) by @icavan
  • Beam gains a BeamForTokenClassification head (#50634) by @AymenKallala

Breaking changes

Cache renaming: past_key_values / cache_params → cache (#50585, #50730)

past_key_values and cache_params are renamed to cache across the library. The old names still work with a deprecation warning; see the dedicated section below.

Attention models used to call their cache past_key_values, while Mamba-style and linear-attention models called it cache_params. Hybrid models therefore had to juggle two names for the same object. There is now a single name, cache, everywhere: model forward signatures, generate, prepare_inputs_for_generation, the masking utilities, and every ModelOutput.

# Before
outputs = model(input_ids, past_key_values=past_key_values, use_cache=True)
next_cache = outputs.past_key_values

# Now
outputs = model(input_ids, cache=cache, use_cache=True)
next_cache = outputs.cache
  • Fully backward compatible for now: past_key_values= / cache_params= are renamed to cache automatically, with a one-time deprecation warning. Model forward and prepare_inputs_for_generation accept the old names until v5.40.0, and generate accepts them until v6.0.0. Passing both an old name and cache raises a ValueError.
  • Outputs too: outputs.past_key_values, outputs.cache_params and outputs["past_key_values"] still return the cache. Output classes can also still be built with the old field names (e.g. CausalLMOutputWithPast(past_key_values=...)), so most remote-code models keep working unchanged.
  • Custom code: models defined outside the library should rename their arguments to cache. Layers now call cache.update(key_states, value_states, layer_idx, kwargs=kwargs), so custom Cache subclasses that override update should accept the extra kwargs argument.

🚨 peft>=0.22.0 is now required.

📚 Docs: https://huggingface.co/docs/transformers/main/en/kv_cache

Multimodal Outputs

Generative multimodal models now return shared output classes (MultimodalBaseModelOutputWithPast, MultimodalCausalLMOutputWithPast, and their MoE variants) instead of one output class per model. Code that checks for a model-specific output class should switch to these.

modeling_attn_mask_utils.py has been removed. Every function in it had been deprecated in favor of transformers.masking_utils. Remote-code models that still import _prepare_4d_causal_attention_mask or AttentionMaskConverter from it must move to transformers.masking_utils.

Other things

LayoutLMv3 (and the PP-DocLayoutV2 reading-order model that shares its code) now go through the attention interface, so SDPA and flex attention are supported. The relative position bias is built once per forward pass and passed to attention as position_bias.

  • 🚨 LayoutLMv3: support SDPA through the attention interface, build the relative position bias once (#50688) by @14H034160212

Cache

Beyond the rename, this release adds static-cache support to assisted decoding and fixes generate with flex attention and a static cache. StaticCache now allocates values with the value states' dtype, and the cache is kept under activation checkpointing when gradients are off. The is_linear / is_croppable properties are fixed for hybrid linear/full-attention models, BigBird block-sparse decoding works with a KV cache, and KV sharing is restored for the Gemma 4 assistants.

Generation

Generation fixes cover sampled beam search (candidates are sorted again before finalization), constrained sampling (batch IDs are preserved), generate on MPS with encoder-decoder models, and TextStreamer, which no longer drops spaces or lags on text without spaces such as Chinese, Japanese and Korean. StopStringCriteria initializes faster, and its cache now actually hits.

Continuous batching

Continuous batching now works under torch.inference_mode() and finds sharded key/value projections under any decoder path. The per-token memory estimate scales with the TP size, warmup handles async and static inputs correctly, and an out-of-bounds error in the top-k processor is fixed.

  • [CB] Scale the per-token memory estimate by the TP size (#48566) by @qgallouedec
  • Continuous batching: find the sharded key/value projections under any decoder path (#50366) by @qgallouedec
  • Fix continuous batching under torch.inference_mode() (#50863) by @tarekziade
  • [CB] Re-order the warmup block so async warms both sets of inputs (#50868) by @remi-or
  • Wait for each warmup forward before the next one rewrites the static inputs (#50353) by @qgallouedec
  • Fix out-of-bounds error in continuous batching top-k processor (#50282) by @khush-patel-09
  • [CB] Replace test prompts with less ambiguous ones (#50876) by @remi-or

Chat templates, parsing and serving

Chat parsing gets a new python-call-list content parser, better handling of bare keys and duplicate XML arguments, and three further parser fixes. apply_chat_template accepts message objects, and assistant masks are correct with left truncation. transformers serve handles a missing Hub cache, lists loaded models in /v1/models, keeps local paths containing @, and keeps assistant content when tool_calls is empty.

Pipelines

Multimodal pipelines now accept media-only inputs and string assistant prefills, keep Mistral prefills in Voxtral, stop mutating generation kwargs, and return chat messages as input_text when returning tensors. pipeline also respects a device_map that targets a single device.

  • fix(pipelines): support media-only inputs in AnyToAnyPipeline (#50856) by @BlazeSR10hawk
  • fix(pipelines): support string assistant prefill in multimodal full-t… (#50679) by @BlazeSR10hawk
  • fix: preserve Mistral prefill in Voxtral and multimodal pipelines (#50624) by @BlazeSR10hawk
  • fix(pipelines): avoid mutating multimodal generation kwargs (#50786) by @BlazeSR10hawk
  • Return the chat messages as input_text when multimodal pipelines return tensors (#50816) by @burrows99
  • Fix pipeline ignoring device_map when it targets a single device (#50645) by @wanhakim

Vision

Vision fixes cover Muse Glimmer's embedding order, GotOCR2 and Emu3 image sizes, Kimi video inputs, NemotronH Omni video forwarding, SAM3 RoPE precision in bf16, and DETR-family configs with dilation=None. SigLIP2 and LFM2-VL now process all images at once with tensor ops. Flash Attention is turned off for multimodal models whose image mask it would drop.

Audio

Audio fixes cover the OmniASR CTC blank token, Qwen3 Omni audio kwargs, Qwen3-ASR's truncation=False default, PeAudioVideo loading with timm 1.0.31, and the audio training examples.

Quantization

Dequantized models are now saved without the quantizer's weight conversions, and the unquantized weights of a dequantized FP8 checkpoint are cast to the model's dtype. The quantization docs also get a standard template for each method.

  • Save dequantized models without the quantizer's weight conversions (#50271) by @qgallouedec
  • Cast the unquantized weights of a dequantized FP8 checkpoint to the model's dtype (#50595) by @qgallouedec
  • Update inkling2 quant config (#50829) by @guarin
  • [docs] Standard template for quantization methods (#50848) by @stevhliu

Parallelization

FSDP2 modules are resharded before the optimizer is created, pipeline parallelism spreads layers more evenly, and TP sharding of tied weights is fixed.

Exporters

Exported ONNX, OpenVINO and ExecuTorch programs load without trust_remote_code. Pickled .pt2 programs, which can run arbitrary code when loaded, now require trust_remote_code=True.

Bugfixes and improvements

  • Skip flaky Kosmos2 test_generate_from_multimodal_encoder_outputs (#51049) by @ydshieh
  • Fix flaky DeepseekV32 / Whisper / HiggsAudioV2 CUDA graph capture failures (GC inside the capture, torch 2.15) (#50899) by @ydshieh
  • [docs] Fix the custom ResNet example in the customizing models guide (#50999) by @AlexxRussell
  • [serge] Fix 2 integration tests for model seamless_m4t_v2 failing with output_mismatch (list output differs (2)) (#48553) by @sergereview[bot]
  • [serge] Fix 2 integration tests for model videomae failing with other (other (2)) (#49287) by @sergereview[bot]
  • Always ensure correct dtype for _keep_in_fp32_modules_strict (#50975) by @Cyrilvallez
  • Fix DataCollatorWithFlattening crash on empty examples (#50984) by @fruitstash
  • Require transformers-mic 0.1.3 for make run-mic (#51019) by @tarekziade
  • Dory tests: build the recurrent-layer configs through the model tester (#51000) by @tarekziade
  • Move older-style testers of 129 more models onto static small-config defaults (#50976) by @tarekziade
  • Fix UMT5 decoder scaling for untied checkpoints (#50950) by @MonikaDvorackova
  • Move older-style testers of 123 popular models onto static small-config defaults (#50972) by @tarekziade
  • Fix typos in the English model and decoding guides (#50974) by @Ultroncn
  • [serge] Fix 2 integration tests for model ministral3 failing with other (other (2)) (#49336) by @sergereview[bot]
  • [serge] Fix 2 integration tests for model regnet failing with output_mismatch (tensor values differ (2)) (#47616) by @sergereview[bot]
  • [serge] Fix 2 integration tests for model efficientnet failing with output_mismatch (tensor values differ (2)) (#47840) by @sergereview[bot]
  • [serge] Fix 2 integration tests for model pvt_v2 failing with output_mismatch (tensor values differ (2)) (#49243) by @sergereview[bot]
  • Follow the SentencePiece model when converting a bare tokenizer.model (#50555) by @hmellor
  • [serge] Fix 4 integration tests for model pvt failing with output_mismatch (tensor values differ (4)) (#47750) by @sergereview[bot]
  • Migrate the remaining 95 testers to static small-config defaults (#50944) by @tarekziade
  • Migrate testers for 20 popular models to static small-config defaults (#50916) by @tarekziade
  • Use hf-internal-testing for the tiny Dbrx and Bloom models and the example images (#49993) by @albertvillanova
  • Return an empty list from batch_decode for an empty batch (#50873) by @zaidwhy
  • [LFM3] Hub causal-conv1d kernels and packed sequences (#50732) by @paulpak58
  • Fix NewTokenizer example in custom tokenizers guide (#50840) by @phatstraws
  • [Apertus1.5] Add A10G expectations for audio and batch generation integration tests (#50924) by @ydshieh
  • Check python_multipart before multipart in is_multipart_available (#50664) by @RyanMullins
  • feat(ci): scan the docker images with Wiz before they are pushed (#49150) by @rtrompier
  • Remove now useless keys_to_ignore_at_inference in all configs (#50911) by @Cyrilvallez
  • [SGLang CI] Fix the VLM teardown error on CPU (follow-up of #50913) (#50914) by @ydshieh
  • Set _skip_keys_device_placement directly inside PreTrainedModel (#50909) by @Cyrilvallez
  • [SGLang CI] Deselect a GPU-only VLM test (#50913) by @ydshieh
  • [vLLM CI] Fix the FireRedASR2 and MTP test failures (missing dependency, GPU-only test) (#50912) by @ydshieh
  • Fix Reformer LM head bias not applied since the weight-loading refactor (#50811) by @ydshieh
  • Fix error message and remove duplicate code in convert_slow_tokenizer.py (#50077) by @samyuktahegde
  • Fix first-subword label assignment in LayoutXLM, LayoutLMv2, and MarkupLM tokenizers (#50455) by @14H034160212
  • Skip added-token trie scan when no token can match (#50781) by @TheNetherWatcher
  • [CI report] Upload the vLLM / SGLang CI reports to the Hub and post a summary to Slack (#50896) by @ydshieh
  • Use lifespan instead of on_event in pipeline web server guide (#50858) by @phatstraws
  • [CI] docs-check fixes (#50279) by @stevhliu
  • Expose literal tester overrides and merged small-config defaults (#50462) by @tarekziade
  • [SGLang CI] Add failure report job (shared with vLLM CI) (#50882) by @ydshieh
  • [Apertus 1.5] Guard behind slow for gated repo (#50884) by @vasqu
  • avoid running this wf from fork (#50694) by @tarekziade
  • bench test reproducibility (#50504) by @tarekziade
  • Fix timestamps (#50503) by @tarekziade
  • [SGLang CI] Test against the latest published SGLang CPU release (#50878) by @ydshieh
  • Keep AFMoE expert biases in FP32 on BF16 load (#50687) by @annekethvij
  • Fix seeded collator generator selection (#50861) by @danarkhipov
  • Remove no_grad decorator on all _init_weights (#50831) by @Cyrilvallez
  • Remove no_grad decorator on RoPE forward (#50828) by @Cyrilvallez
  • Update expected logits slice for Qwen2MoeIntegrationTest::test_model_a2_7b_logits on A10G (#50845) by @ydshieh
  • Check for an empty batch in convert_to_tensors without flattening it (#49942) by @jayzuccarelli
  • [serge] Fix 2 integration tests for model qwen2_moe failing with other (other (2)) (#49480) by @sergereview[bot]
  • [serge] Fix 1 integration test for model flint failing with other, raising RuntimeError (other (1)) (#50283) by @sergereview[bot]
  • [serge] Fix 5 integration tests for model qwen3_vl_moe failing with other, raising RuntimeError (other (5)) (#50145) by @sergereview[bot]
  • Fixing training CI again (#50747) by @SunMarc
  • Fix Mistral Large 4 LeRoPE training gradients (#50727) by @taking-lying-flat
  • Keep MiMo-V2-Flash's router in float32 in bfloat16 models (#50722) by @francesco-bertolotti
  • Fix NanoChat inheriting an incompatible rotary kernel (#50823) by @taking-lying-flat
  • Fix CwmIntegrationTest GPU OOM on multi-GPU runners (cap per-GPU max_memory at 70%) (#50824) by @ydshieh
  • Mark OmDetTurboModelTest::test_batching_equivalence as flaky (topk query selection) (#50815) by @ydshieh
  • [docs] Fix imports that fail on main in four English docs (#50817) by @burrows99
  • Fix duplicate JetMoE router logits in auxiliary loss (#50658) by @taking-lying-flat
  • docs: updates to security policy in SECURITY.md (#50711) by @Michellehbn
  • [serge] Fix 2 integration tests for model utils failing with output_mismatch (list output differs (2)) (#49407) by @sergereview[bot]
  • Fix OOM in Mamba2IntegrationTest by using MemoryCleanupMixin (#50808) by @ydshieh
  • [docs] Fix the any-to-any guide example using the removed infer_device (#50792) by @burrows99
  • Scale the padding-free position_ids test tolerance to the logits (#50697) by @EvanZhuang
  • [docs] Fix PairClassificationPipeline example using the removed self.framework (#50789) by @burrows99
  • [serge] Fix 2 integration tests for model hrm_text failing with output_mismatch (tensor values differ (2)) (#50006) by @sergereview[bot]
  • [serge] Fix 6 integration tests for model bloom failing with output_mismatch (list output differs (4), other (2)) (#50469) by @sergereview[bot]
  • [serge] Fix 2 integration tests for model ministral failing with output_mismatch (tensor values differ (2)) (#50146) by @sergereview[bot]
  • [serge] Fix 1 integration test for model deepseek_vl_hybrid failing with output_mismatch (list output differs (1)) (#49581) by @sergereview[bot]
  • [serge] Fix 2 integration tests for model ovis2 failing with output_mismatch (list output differs (2)) (#50233) by @sergereview[bot]
  • [docs] Replace examples using pipeline tasks removed in v5 (#50625) by @rickygrosvenor-pramanick
  • [CI] Fix external links (#50685) by @stevhliu
  • Update A10 expected outputs of FA2 integration tests (#50738) by @ydshieh
  • Revert mark linguistic generated (#50735) by @vasqu
  • Update dev version (#50725) by @vasqu

Full Changelog: v5.23.0...v5.24.0