v0.19.8 Patch Release

What's Changed

• Deprecate leftovers of compression library by @hwchen2017 in #8535
• Update version.txt after 0.19.7 release by @loadams in #8555
• Deprecate loco zero++ by @hwchen2017 in #8534
• Deprecate sparse_gradients by @pengdurice in #8504
• docs: stop Codex false-alarm on DCO sign-off by @delock in #8561
• Raise Modal sandbox timeout to absorb full-suite runtime by @jinyouzhi in #8571
• Derive the autocast test backend and scope the bf16 gate to NCCL by @delock in #8559
• Skip heavy CI for docs-only PRs touching root markdown by @delock in #8560
• Deprecate APEX amp by @hwchen2017 in #8564
• Document sub_group_size and correct its default to 1e9 by @shravanibnikam in #8567
• Deprecate legacy curriculum learning by @hwchen2017 in #8566
• [muon] Per-head Newton-Schulz for attention projections by @alanhuangyoo in #8384
• Upgrade yapf to v0.43.0 to fix EOFError failure in CI by @jinyouzhi in #8574
• [AutoTP] Add support for vocab-parallel LM head and related configurations by @jinyouzhi in #8309
• Remove Progressive Layer Dropping (PLD) support by @hwchen2017 in #8562
• Support Muon in BF16_Optimizer with FP32 gradient accumulation by @0z5a in #8526
• Register native pinned host memory with the CANN runtime for NPU by @VenusTZZ in #8532
• implement register_host_memory and unregister_host_memory for supa by @frozenleaves in #8537
• Remove elastic training by @pengdurice in #8563
• Use int64 program ids in the SwiGLU and AutoEP fused restore Triton kernels by @pengdurice in #8591
• Describe the GPTBigCode and Yuan shared-QK layouts by @Achyuthan-S in #8575
• Fix HybridEngine OPT generation with modern cache interfaces by @0z5a in #8523
• [muon] Per-head Muon for linear-attention layers: read head geometry from the owning module by @alanhuangyoo in #8436
• [AutoTP] Support eager row-parallel output head training by @0z5a in #8538
• Release AutoEP router-logit caches on checkpoint replay exits by @yh0903 in #8581
• Fix memory leak and NULL-deref in shm_initialize by @tritsystem in #8565
• Fix silent gradient-bucket drop on CPU in ZeRO-1/2 non-contiguous reduction by @delock in #8382
• Add opt-in AutoEP Python GC policy by @yh0903 in #8451
• [AutoTP] Add optional Liger vocabulary-parallel cross entropy by @0z5a in #8540
• Overlap AutoEP split planning with token packing by @yh0903 in #8359
• Normalize Gram Newton-Schulz inputs before casting to fp16 by @vineethsaivs in #8533
• Remove the older TiledLoss and UlyssesSPFwdLossBwdWithLogits by @vineethsaivs in #8587
• Wait for all async all-gathers in coalesced ZeRO-3 parameter gather by @delock in #8539
• Skip narrow dtypes in the mixed-precision cast by @sfc-gh-abkulkarni in #8416
• AutoEP: drop the redundant float32 copy in the default combine by @pengdurice in #8621
• Remove stale compression test from HPU CI by @nathon-lee in #8616
• Skip fp16 ZeroPP tests on accelerators without fp16 support by @jinyouzhi in #8610
• AutoEP: share one DeepEP buffer across MoE layers instead of one per layer by @yh0903 in #8557
• Implement graph operations for NPU accelerator by @VenusTZZ in #8605
• Add opt-in regional compilation around AutoEP layers by @yh0903 in #8380
• Fix pipe tests failures on CPU by @jinyouzhi in #8611
• Run Muon once per optimizer step under ZeRO-3, not once per micro-batch by @alanhuangyoo in #8600
• Fix 2 CPU failures in cpu-torch-latest by @jinyouzhi in #8629
• Fix Muon optimizer under ZeRO CPU offload and bound gather buffers by @jinyouzhi in #8464
• Loosen the late-iteration gradient tolerance in the checkpointing test by @delock in #8648
• Add affine AutoEP checkpoint placement by @jinyouzhi in #8544
• Fix SequenceTiledCompute backward for empty trailing shards by @taking-lying-flat in #8434
• fix: Remove the unused ROCm Apex clone, build, and PYTHONPATH setup. by @nathon-lee in #8617
• Avoid repeated full gradient clearing in ZeRO-3 subgroup updates by @aprylewu in #8625
• Stage Muon's momentum once per partition in ZeRO-1/2 by @alanhuangyoo in #8633
• A zero-element parameter loses its shape when it is bound to a flat buffer by @alanhuangyoo in #8467
• Fix Muon aspect-ratio scale truncated when muon_update recompiles by @alanhuangyoo in #8641
• Deprecate and remove graph harvesting by @hwchen2017 in #8579
• Replay AutoEP's cached DeepEP dispatch in the layout it was built with by @yh0903 in #8636
• Keep ZenFlow Adam step counters in float32 by @vineethsaivs in #8496
• [AutoTP] Implement VocabParallelEmbedding and enhance tied model support by @jinyouzhi in #8582
• docs: add Simplified Chinese translation (README.zh.md) by @JasonYeYuhe in #8498
• fix(rollout): reject shared prefill with continuous batching by @modelpath-dev in #8583
• AutoEP: Add activation registry by @pengdurice in #8650
• Skip ZeRO DeepCompile backward hooks for AutoSP by @jinyouzhi in #8667
• Add DualPipeV by @phoeenniixx in #8576
• Implement FusedLion, FusedLamb, and Quantizer ops for NPU accelerator by @VenusTZZ in #8627
• Fix Triton compatibility for FlashAttention and MatMul autotuning by @hwchen2017 in #8679
• fix(rollout): forward top-k sampling configuration by @nathon-lee in #8676
• Skip fp16-config tests on accelerators without fp16 support by @delock in #8684
• Consolidate PRs for docs and typos by @hwchen2017 in #8689
• Fix ZeRO-3 Muon all-gather for mixed parameter shapes by @jinyouzhi in #8628
• Reject amsgrad in DeepSpeedCPUAdam instead of ignoring it by @vineethsaivs in #8694
• Fix FPDT K/V gradient buffers under grouped-query attention by @jinyouzhi in #8665
• Check AutoSP sequence length and head count divisibility by @jinyouzhi in #8675
• Validate the AutoSP SP/DP mesh when registering process groups by @jinyouzhi in #8674
• Use expert gradients, not expert weights, in the FP16_Optimizer MoE grad norm by @yh0903 in #8635
• Restore a shard from the map the layer published by @Achyuthan-S in #8622
• AutoEP: one program per token in the fused weighted restore forward by @yh0903 in #8678
• Nightly full-suite run with classified failure triage by @delock in #8606
• Keep Muon's update at full size under fp16 loss scaling by @alanhuangyoo in #8655
• Add Fengchun Hua (hipudding) to COMMITTERS.md by @PKUWZP in #8718
• Add opt-in fused AutoEP DeepEP row weighting by @yh0903 in #8644
• AutoEP: MiniMax-M3 preset by @pengdurice in #8651
• AutoEP: reorder expert rows with gathers on the collective path by @yh0903 in #8671
• Fix Ulysses SP registration for string model paths and gloo collectives by @delock in #8702
• Harden ZeRO-1/2 offload gradient storage lifetime and stream ordering by @jeffra in #8632
• Give rollout cache fakes the per-layer config Transformers now reads by @sfc-gh-truwase in #8730
• Add an opt-in fused Triton RoPE for Hugging Face split-half models by @yh0903 in #8663
• feat(rollout): align decode fronts and trim dead zones by @nathon-lee in #8497
• Fix autotuner crash when an experiment produces no metric by @Rakshit-gen in #8421
• Raise Modal nightly Sandbox timeout to 2h by @delock in #8746
• Keep invalid group-norm sentinels from becoming a finite 1.0 step by @sfc-gh-truwase in #8638
• Pin Modal torch-latest to OCI and cover PR open events by @tohtana in #8770
• Add AWS capacity fallback for torch-latest by @tohtana in #8778
• [BUG] Replace deprecated npu_apply_adam(_w) with torch._fused_adam(w)_ on NPU by @Ginray in #8640
• Release each BF16 gradient once immediate_grad_update has accumulated it by @yh0903 in #8654
• Stream AWS fallback logs while commands run by @tohtana in #8783
• Assert offload-state removal only when offload leaves the accelerator by @delock in #8738
• [AutoTP] Support tied embeddings for row-parallel output-head training by @jinyouzhi in #8680
• Keep the buffers a model names for fp32 in fp32 under ZeRO-3 by @pengdurice in #8699
• Fix DataAnalyzer.run_map crashing when specific_threads is set by @Rakshit-gen in #8758
• feat(rollout): add OPSD response token loss primitives by @nathon-lee in #8677
• Use default fp32 tolerances in AutoTP training tests by @hwchen2017 in #8796
• Guard the shm inference all-reduce on the host ISA by @delock in #8701
• Reduce unnecessary CI work for documentation changes and consolidate agent instructions in AGENTS.md by @hwchen2017 in #8794
• Avoid circuler import caused by Hybrid Engine and OPT model by @tohtana in #8737
• AutoEP + AutoTP folding under ZeRO-3 by @pengdurice in #8670
• Add an opt-in fused Triton RMSNorm for Llama-style Hugging Face models by @yh0903 in #8645
• Log the skipped optimizer step on gradient overflow by @sfc-gh-abkulkarni in #8417
• [UC] Plan a scale-1 affine shard-to-shard transfer without rebuilding the parameter by @0z5a in #8623
• Preserve non-floating buffers in ZeRO checkpoint conversion by @aprylewu in #8515
• [Bug Fix] Fix CPUAdagrad updates with nonzero weight decay by @hwchen2017 in #8781
• [checkpoint] Refuse a map whose pieces write the same shard address by @0z5a in #8624
• [AutoSP] Honor seq_dim when AutoSP shards inputs by @jinyouzhi in #8720
• [AutoSP] Fix tensor truthiness in get_node_shape_meta by @jinyouzhi in #8760
• Add an opt-in chunked causal-LM loss that keeps no full-vocabulary FP32 tensor by @yh0903 in #8653
• [AutoSP] Rebuild query-sharded attention masks before SDPA by @jinyouzhi in #8761
• Replay AutoEP's DeepEP dispatch layout in non-reentrant checkpoint recompute by @yh0903 in #8731
• Add an opt-in fused gate+up grouped GEMM for AutoEP experts by @yh0903 in #8733
• Add a pr-checklist agent skill and an AI contribution policy by @PKUWZP in #8744
• AutoEP: do not slice DeepEP buffers that already hold exactly the arrived rows by @yh0903 in #8747
• Use C++20 for XPU op builders on torch 2.12+ by @jinyouzhi in #8803
• [bug fix] ZeRO2: preserve fp32 grad accumulation precision in partition storage by @stas00 in #8749
• Reduce chunked cross-entropy memory test footprint by @tohtana in #8824
• [CI] Run torch-latest only on AWS by @tohtana in #8825
• Build CPU optimizer ops without CUDA and fix the CPU Lion SIMD update by @hwchen2017 in #8813
• Add a nightly multi-rank CPU suite by @delock in #8806
• Respect BF16 overflow-check opt-out in ZeRO-1/2 by @XFDG in #8818
• Speed up import deepspeed by deferring heavy imports and import-time work by @hwchen2017 in #8807
• Use C++20 for NPU op builders on torch 2.12+ by @VenusTZZ in #8816
• [fix] NPU: RoPE correctness, inverse-frequency caching, and KV cache preallocation in the inference op by @VenusTZZ in #8776
• Release GIL while destroying AIO handles by @tohtana in #8832
• Add vocabulary-parallel logprobs, entropy and KL ops by @jinyouzhi in #8791
• Describe the CodeGen fused-QKV layout as a block selection by @Achyuthan-S in #8706
• Add vocabulary-parallel fused linear cross entropy by @jinyouzhi in #8792
• [CI] Restore Modal-first torch-latest routing by @tohtana in #8849
• Fix TestLionConfigs on the CPU accelerator and run it on every runner by @hwchen2017 in #8833
• Add deepspeed.comm.pg_utils.is_parallel for optional parallel groups by @jinyouzhi in #8815
• Release process-group caches before CPU/Gloo worker shutdown by @tohtana in #8846
• Add KLXPU (KUNLUNXIN XPU) accelerator by @GuCarpenter in #8799
• Enable CUDA Graph for continuous rollouts by @nathon-lee in #8732

New Contributors

• @shravanibnikam made their first contribution in #8567
• @VenusTZZ made their first contribution in #8532
• @tritsystem made their first contribution in #8565
• @sfc-gh-abkulkarni made their first contribution in #8416
• @aprylewu made their first contribution in #8625
• @JasonYeYuhe made their first contribution in #8498
• @modelpath-dev made their first contribution in #8583
• @phoeenniixx made their first contribution in #8576
• @Ginray made their first contribution in #8640
• @XFDG made their first contribution in #8818
• @GuCarpenter made their first contribution in #8799

Full Changelog: v0.19.7...v0.19.8