Skip to content

Release 5.17.0

Latest
Latest

Choose a tag to compare

View all tags
@vasqu vasqu released this 09 Sep 15:42
· 5 commits to main since this release
v5.17.0
856157a

Release v5.17.0

New Model additions

HYV4

image

Hy4-Preview is a 780B-parameter mixture-of-experts language model that activates 49B parameters per
token. Each MoE layer holds 256 routed experts plus one always-active shared expert and routes every
token to 8 of them. The context window is 1M tokens.

The architecture combines four features:

  • Multi-head Latent Attention (MLA) compresses keys and values into a low-rank latent
    (kv_lora_rank) that kv_b_proj expands back to one key/value per query head.
  • DeepSeek Sparse Attention (DSA) selects index_topk keys per query with a lightweight indexer.
    Following IndexShare, only the layers marked "full"
    in indexer_types run an indexer; "shared" layers reuse the previous full layer's selection.
  • Gated MLA with learnable attention sinks, where each head owns a sink logit that participates
    in the softmax and contributes no value, as in GPT-OSS.
  • Independent Hyper-Connections (iHC) replace the plain residual path with hc_mult parallel
    residual streams that are collapsed before, and redistributed after, every sublayer.

The implementation does not execute the multi-token prediction (MTP) layers. Released checkpoints
keep those weights so that other runtimes can use them for speculative decoding; they are ignored
at load time.

Links: Documentation

VibeVoice

image

VibeVoice is a novel framework for synthesizing high-fidelity, long-form speech with multiple speakers by employing a next-token diffusion approach within a Large Language Model (LLM) structure. It's designed to capture the authentic conversational "vibe" and is particularly suited for generating audio content like podcasts and multi-participant audiobooks.

Links: Documentation

NeoMME

NeoMME is a family of efficient 260M and 800M parameter multimodal-native multilingual foundation encoders from H Company. It processes multilingual text tokens and raw image patches in a single bidirectional Transformer encoder, without a separately pretrained vision tower or causal language model.

NeoMME-Retriever is a model fine-tuned from the NeoMME backbone for visual document retrieval with joint late-interaction and dense objectives. It takes text queries and documents (text or page screenshots) and produces multi-vector embeddings for MeanMaxSim scoring (late-interaction) and mean-pooled embeddings for cosine similarity (dense).

Links: Documentation

Fun-ASR-Nano

Fun-ASR-Nano is an 800M-parameter end-to-end speech recognition model developed by Alibaba DAMO Academy's FunAudioLLM team. It achieves state-of-the-art performance on Chinese, English, and Japanese ASR benchmarks while being significantly smaller than comparable models.

Key features are

  • Chinese, English, and Japanese, including 7 Chinese dialects and 26 regional accents
  • Hotword customization for domain-specific vocabulary
  • Native punctuation output (no separate punctuation model needed)

Links: Documentation

KimiLinear

Kimi Linear is a hybrid linear attention architecture from Moonshot AI, introduced in
Kimi Linear: An Expressive, Efficient Attention Architecture.

At its core is Kimi Delta Attention (KDA), a refinement of Gated DeltaNet
that gives each key channel its own forget gate, so the recurrent state decays per channel instead of per head. KDA is
used in most layers; every fourth layer keeps a full-attention block that reuses DeepSeek-V3's Multi-head Latent
Attention (MLA), and the feed-forward blocks are DeepSeek-V3-style MoE with a shared expert.

Links: Documentation

Canary

Canary-1B-v2, a fast, robust multilingual model for Automatic Speech Recognition (ASR) and Speech-to-Text Translation (AST):

Canary reuses the Fast Conformer encoder from Parakeet (loaded through [ParakeetEncoder] / [ParakeetEncoderConfig]) and pairs it with a Transformer decoder that uses fixed sinusoidal positional embeddings, cross-attention to the encoder outputs and tied input/output embeddings. The task is selected through a decoder prompt prefix built by [CanaryProcessor] of the form <|startofcontext|> <|startoftranscript|> <|emo:undefined|> <source_lang> <target_lang> <pnc|nopnc> <|noitn|> <|notimestamp|> <|nodiarize|>, where source_lang == target_lang selects transcription and otherwise selects translation.

Links: Documentation

Breaking changes

Vision rotary embeddings (2D/3D) have been standardized into a unified RoPE frequency computation module, so users with custom vision models relying on attention-layer-level or model-specific RoPE grid interleaving logic must migrate to the new centralized modeling_rope_utils.py implementation.

Generation

Generation improvements include a performance optimization that avoids unnecessary accelerator synchronization on every decode step (reducing per-step overhead), and a fix to prevent unconditional downloading of remote hub files during generation. Several correctness fixes were also applied, including enforcing auto-compile cache checks for encoder-decoder models, standardizing past_key_values naming in AfMoE, and resolving flaky export and integration test failures.

  • [Generate] Avoid unconditionally downloading remote hub file (#48620) by @vasqu in [#48620]
  • [generate] stop synchronizing the accelerator on every decode step (#47975) by @SunMarc in [#47975]
  • [AfMoE] Standardize past_key_values argument naming across forward and generate (#48430) by @shenhuaqingshi in [#48430]
  • Fix MTP generation test regex gate for escaped layer ignore keys (#48003) (#48262) by @Noxtimo in [#48262]
  • fix(generation): Enforce the auto-compile cache check for encoder-decoder models (#48364) by @harshaljanjani in [#48364]
  • [serge] Fix 4 integration tests for model generation failing with output_mismatch (list output differs (4)) (#48133) by @sergereview[bot] in [#48133]
  • [VibeVoice] Skip generate export tests (flaky) (#48396) by @ydshieh in [#48396]

Cache

Fixed several cache-related bugs, including a quantized cache issue in VibeVoice, incorrect rejection of non-static cache implementations in VoxtralRealtime, missing auto-compile cache checks for encoder-decoder models, and a silent failure when paged attention is called without a cache. Documentation was also updated to clarify ContinuousBatchingConfig usage and sliding window model limitations.

Kernels

Kernel support was improved with fixes for nested FLA kernel imports when only fla-core is installed, a warning when hub-kernel functions silently fall back to slower pure-PyTorch reference implementations, and the ability to register standalone functions (e.g., RoPE) in KernelConfig with optional non-inheritance of default mappings. Additional fixes include corrected repository paths for ESMFold2 kernels and updated documentation for KernelConfig customization.

Quantization

Fixed several quantization bugs, including a quant cache issue in VibeVoice, incorrect FP8 embedding handling for Qwen models, missing FP8 tensor parallelism layer overrides, and unnecessary MXFP4 weight dequantization on XPU devices.

Bugfixes and improvements

Significant community contributions

The following contributors have made significant changes to the library over the last release:

  • @ydshieh
    • [fix] Update stale expected strings in HunYuanVL integration tests (#48646)
    • [fix] Update stale golden values and fix expected_logits shape in FlavaForPreTraining integration tests (#48639)
    • [tests] Fix integration test golden values broken by fast image processor default (PR #41388) (#48637)
    • [KimiLinear] Fix test_cpu_offload: set num_local_experts=4 in model tester (#48624)
    • Fix GPU memory teardown in CLI serve tests (#48618)
    • [fix] Fix how we read package versions - triggered by torch 2.14+ (#48615)
    • [Docker] Upgrade CPU torch to <=2.14.0, torchcodec to <=0.16.0 (#48614)
    • Add PR comment CI for AMD (MI300) (#48065)
    • [MiniCPMV4_6] Update test_small_model_vision_generation_batch expected output (value drift) (#48406)
    • Avoid print to stdout that fails the job check_failed_tests job (#48391)
    • [VibeVoice] Skip generate export tests (flaky) (#48396)
    • [LongcatFlash] Fix test_longcat_generation_cpu: use device_map="cpu" to avoid MoE disk offload issue (#48377)
    • [Qwen3VLMoe] Update test_small_model_integration_test_batch expected output (value drift) (#48376)
    • [ONNX] Skip affected models on torch 2.13 (two dynamo regressions) (#48191)
    • Retry get_daily_ci_runs on stale GitHub API cache (#48374)
    • [conftest] Use get_cpu_ram_total_gib for psutil patch (cgroup-aware) (#48290)
    • Fix flaky test_training_gradient_checkpointing for BigBirdPegasus (fp noise filter) (#48332)
    • [qwen4_exp] disable torch/onnx export tests due to data-dependent control flow (#48345)
    • [debug] Trace previous CI run selection in get_previous_daily_ci.py (#48338)
  • @LauraGPT
    • Add Fun-ASR-Nano model (#46180)
  • @tarekziade
    • Fix AttributeError in gradient_checkpointing_enable(offload=True) (#48590)
    • Compress the agent conventions file and document two modular pitfalls (#48586)
    • CI: Point test fixtures at hf-internal-testing copies we already host (#48521)
    • Retire test_multi_gpu_data_parallel_forward (#48508)
    • QA: Add noisy comment checker (#48484)
    • skip mtp slow tests for now (#48328) (#48329)
    • Bump transformers-mlinter to 0.1.5 and clear the new findings (#48259)
  • @remi-or
    • [Fix] Fix A10 expectations for a test (#48454)
    • Kimi linear (#48250)
    • [Fix] Use dedicated helpers for DeepGEMM and SonicMoE tests (#48523)
    • [Fix] Sparse TikToken tokenizers silently fail (#48446)
    • [Improvement] Make gated delta rule more explicit (#47625)
    • [CB] Fix wrong device scoping (#48370)
    • [CB] Fail faster (#48334)
  • @ArthurZucker
  • @harryjulian
    • Add support for NeuCodec (#47143)
  • @delock
    • Batch Rebalance Data Sampler (#47340)
  • @harshaljanjani
    • fix: Add DEIMv2 attribution (#48448)
    • model: Add NVIDIA Canary-1B-v2 to Transformers (#46825)
    • fix(generation): Enforce the auto-compile cache check for encoder-decoder models (#48364)
    • fix(models): Drop the position-indexed token type lookup in RoPE encoders (#48407)
  • @tonywu71
    • Add NeoMME and NeoMME-Retriever (#47992)
  • @Dovis01
    • [Docs]: Update GLM 5.3 (#48401)
    • [Docs] Change 5.3 Flash pos in toc (#48366)
    • [Glm 5.3 Flash] GLM 5.3 Flash Support (#48342)
  • @pengzhiliang
    • Implement VibeVoice (#40546)

Contributors

tarekziade, ydshieh, and 46 other contributors
Assets 2
Loading