v0.19.6 Patch Release

What's Changed

• Fix checkpoint rank selection for Ulysses sequence parallelism by @sfc-gh-thonguyen in #8226
• Return a copy from OnebitLamb.get_lamb_coeffs by @vineethsaivs in #8227
• Update version.txt after 0.19.5 release by @loadams in #8243
• Implement the documented per-param-group lists in OneCycle by @vineethsaivs in #8201
• Keep the elastic batch size within max_train_batch_size by @vineethsaivs in #8237
• [DeepCompile] Fix KeyError on frozen parameters in ZeRO-3 by @roycho96 in #8214
• [DeepCompile] Fix staticmethod handling on Python 3.9 by @baremetaldevx86 in #8240
• Upgrade pip in Python install-smoke jobs by @tohtana in #8247
• (1/2) Implementing Compiler Pass for AutoTP by @therealnaveenkamal in #8204
• Fix DeepCompile last use for unconsumed waits by @tohtana in #8254
• Add configurable sum gradient reduction by @zupengwang in #8232
• Add native (DeepNVMe) host-memory pinning backend for accelerators by @sfc-gh-truwase in #8211
• Add compile.offload_activation_pin_memory for DeepCompile activation offload by @sfc-gh-truwase in #8258
• Fix int32 overflow in Triton grouped-GEMM expert offset by @PKUWZP in #8261
• NCCL and other backend PG timeout 30min -> 10min by @stas00 in #8253
• Fix ZeRO-3 crash in AutoTP universal-checkpoint metadata by @PKUWZP in #8270
• Add CUDA graph support for HybridEngine generation by @PKUWZP in #8271
• Fix docstring Args entries that name a parameter the function does not take by @darkdi in #8223
• Keep parameter dtype through ZeRO-3 weight quantization by @adityasingh2400 in #8215
• [AutoTP] Complete uneven sharding and universal checkpoint support by @jinyouzhi in #8185
• Route FPDT and checkpoint writer pins through accelerator pin_memory by @sfc-gh-truwase in #8257
• Add fused triton kernel for swiglu by @hwchen2017 in #8244
• Make pass contracts cover the passes DeepCompile actually schedules; Renaming files by @pengdurice in #8251
• Non-reentrant activation checkpoint CPU offload by @sfc-gh-truwase in #8282
• Enable activation offloading by @pengdurice in #8255
• [typo] Add raise for RuntimeError by @jinyouzhi in #8281
• Fix ZeRO-1/2 with zero-sized parameters by @fwerkor in #8280
• docs: note async cpu_checkpointing perf and expandable_segments by @sfc-gh-truwase in #8287
• Fix ZeroDivisionError in compute_elastic_config return_microbatch on non-0.2 elasticity by @ebarkhordar in #8286
• Fix repeated FlopsProfiler metric accumulation by @baremetaldevx86 in #8246
• fix: stop DeepSpeedConfig writing max_grad_norm back into the caller's config dict by @ebarkhordar in #8289
• Enable DeepSpeed support on Apple Silicon (MPS) with ZeRO Stage 1-3 by @PKUWZP in #8293
• fix(engine): resolve inference workspace attribute lookup by @nathon-lee in #8288
• [AutoTP] Convert HF embedding_rowwise tp_plan entries to SKIP specs by @ebarkhordar in #8294
• Fix zero reduce bucket size validation by @fwerkor in #8266
• Register native pinned host memory with CUDA for GPU DMA by @sfc-gh-truwase in #8283
• Drop the documented grad_hooks ZeRO option, which does not exist by @vineethsaivs in #8242
• Skip zero-sized parameters in HP fragment mapping by @fwerkor in #8298
• perf(rollout): add HybridEngine rollout profiling by @nathon-lee in #8295
• Route isend/irecv to nonblocking backend methods and stage them as async on MPS by @PKUWZP in #8303
• Fix device mismatch in test_gate_up_partition_covers_the_whole_weight by @alanhuangyoo in #8308
• Import print_dist in auto_tp by @alanhuangyoo in #8311
• Reject invalid ZenFlow ratio and update interval boundaries by @tandede in #8274
• Avoid redundant copies in MPS P2P staging by @FU-max-boop in #8314
• Fix AutoTP metadata updates for unsharded modules by @zupengwang in #8299
• [Apple Silicon Support Phase 1] Add Metal FusedAdam kernel and CPU Adam build for Apple Silicon by @PKUWZP in #8300
• Enable optimized Adam backend for MuonWithAuxAdam optimizer by @jinyouzhi in #8278
• feat(opsd): share prompt prefill across rollout samples by @nathon-lee in #8296
• stop allocating per-element temporaries in the overflow check by @stas00 in #8325
• Fix shared loss gradient accumulation by @tohtana in #8245
• Isolate DeepCompile list-schedule test ops by @tohtana in #8319
• Fix ZeRO-3 synchronization during OPSD rollout by @LiRunGuo in #8264
• [AutoTP] Fix training lm_head routing by @gaoxiaomo in #8302
• Route ZeRO/SuperOffload pin sites through accelerator pin_memory by @sfc-gh-truwase in #8256
• Honor adam_w_mode in the CPU multi_tensor_adam binding by @PKUWZP in #8307

New Contributors

• @sfc-gh-thonguyen made their first contribution in #8226
• @baremetaldevx86 made their first contribution in #8240
• @darkdi made their first contribution in #8223
• @fwerkor made their first contribution in #8280
• @alanhuangyoo made their first contribution in #8308
• @tandede made their first contribution in #8274
• @FU-max-boop made their first contribution in #8314
• @LiRunGuo made their first contribution in #8264
• @gaoxiaomo made their first contribution in #8302

Full Changelog: v0.19.5...v0.19.6