Skip to content

v0.1.12

Latest
Latest

Choose a tag to compare

View all tags
@FoolPlayer FoolPlayer released this 09 Sep 03:57
· 8 commits to main since this release
v0.1.12
fd99abf
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Highlights

  • DeepSeek-V4, end to end — training support (#840), TileLang kernels (#912), Liger kernel path (#946), MFU accounting (#944), Ulysses sequence parallel (#949) and context parallel for attention, compressors and indexer (#1111, #1131), the Lightning Indexer KL objective (#1134) with its TileLang teacher-distribution kernel (#1109), FlashMLA cuDNN DSA backend (#850), fp8/fp4 quantized weight export (#962), QAT fake-quant training (#1089), and NPU support (#974).
  • New models — MiniMax H3 (#1075) with Ulysses SP (#1120), LTX-2.3 (#858, #859), GPT-OSS (#854) plus a 120B EP4 LoRA recipe (#921), and an expert-parallel plan for Qwen3.5-MoE causal LM (#1106).
  • LoRA — native PEFT-free LoRA stack with MoE-LoRA and expert parallelism (#758), VLM trainer support (#1018), Qwen3.5-MoE (#1094), EP-sharded streaming load for PEFT models (#915), and faster eager MoE-LoRA routing (#1154, #1155).
  • Parallelism and performance — per-module local parallel state via context manager (#893), FSDP2 torch.compile (#881, #1009), MindSpeed-style async activation offload (#965), shared async Ulysses backward helpers (#1113), FlexAttention with Ulysses (#945), MagiAttention on SM90 with Ulysses (#1021, now an optional extra in #1160), HSDP for expert parallel (#867), optional train-step sync (#938), and reused loss denominators (#941).
  • Muon — Gram Newton-Schulz backends (#953), FSDP2 all-to-all optimization (#973), head-group split (#980) with qualified module names (#1108), and Qwen3.5-MoE Muon on Ascend (#1092).
  • Ascend / NPU — fused npu_rms_norm (#978), AscendC chunk gated delta rule (#924), Triton clamped SwiGLU (#1104), Qwen3-Omni-MoE with fused RoPE (#910), CANN 9.1.0 images (#1128), and GDN Dockerfiles (#1084).
  • More accelerators — AMD ROCm (#890) and Cambricon MLU (#903).
  • Data and trainer — effective-token dynamic batching (#833), map-style datasets in worker-side dynamic batching (#853), dataloader scheduling knobs (#940), model aux metrics reported without folding them into the loss (#1110), channel loss logging (#892), checkpoint early-stop (#926), and a pending async save awaited at train end (#1162).
  • Toolchain — transformers 5.9.0 is the default install (#774), and CUDA 13.0 + PyTorch 2.11 images (#827).

Breaking Changes

  • Per-module local parallel state moves to a context manager (#893)
  • DeepSeek-V4 kernel naming unified (#934)
  • SeedOmni V1 model stack removed (#1082)
  • ChunkMBS removed (#1119)
  • Head-split Muon is selected by qualified module names instead of a DSA indexer flag (#1108)

What's Changed

  • [ckpt, trainer] fix: avoid HBM OOM during DCP save under MoE / VLM training by @TimYangst in #791
  • [model] feat: register ForTokenClassification in model registries by @Luosuu in #795
  • [parallel] fix stride error in npu when dsp+recompute by @yicheng-gong in #793
  • [ckpt, data] refactor: remove _fill_missing_optimizer_states, lazy-import multimodal deps by @Luosuu in #800
  • [ckpt] refactor: deduplicate ExtraParallel dim preprocessing by @Luosuu in #803
  • [ckpt] refactor: centralize async DCP save lifecycle in wait_for_pending_save() by @Luosuu in #802
  • [parallel] fix: keep root's auto-no-reshard so fused-linear kernels can backward by @Luosuu in #806
  • [model, docs] refactor: make patchgen reusable as a library by @TimYangst in #807
  • [ci, docker] chore: bump uv from 0.9.8 to 0.11.16 by @TimYangst in #782
  • [model, ci, docs] feat: bump transformers to 5.9.0 by @TimYangst in #774
  • [config, ci, docs] chore: relax uv pin from ==0.11.16 to a 0.9.8-0.11 range by @TimYangst in #808
  • [model, ci, docs, agent] refactor: ship patchgen as a sibling standalone package by @TimYangst in #809
  • [data, model] feat: Implemented dynamic batch logic optimization by @zhihaofang1017 in #789
  • [ckpt, model] fix: rename per-expert HF weight_map keys for fused MoE HF export by @Coach257 in #799
  • [docs, docker] feat: Update A3 Docker image to 9.0.0 and NPU model support table by @cls1206 in #817
  • [model] fix: derive hybrid-attn layer counts from layer_types for transformers v5 by @TimYangst in #819
  • [docker] feat: update docker with template by @FoolPlayer in #814
  • [docs] chore: drop stale offline MoE merge step from Qwen3 / Qwen3-MoE / Qwen3-Omni recipes by @TimYangst in #813
  • [data, omni, docs] feat: Qwen3-Omni recipe for offline-extracted audio-enabled video by @TimYangst in #812
  • [ops, ci] feat: extend NPU kernel test coverage and add CPU-runnable registry/eager sanity tests by @Kirrito-k423 in #820
  • [data] feat: align StatefulDataLoader snapshot frequency to save_steps by @Fazziekey in #822
  • [docs] fix: Ascend documents fix by @dzy00897185 in #830
  • [model] feat: support sp on Qwen-Image by @FoolPlayer in #815
  • [perf] fix: enable ProfilerWithMem with NPU by @yanghw116 in #831
  • [docker] feat: support cu130+pt211 by @FoolPlayer in #827
  • [model] fix: patched forward docstrings for transformers 5.9 by @Luosuu in #836
  • [misc] feat: pin FA4 4.0.0b16 by @Luosuu in #837
  • [model] fix: document qwen3 moe cache position by @Luosuu in #838
  • [ci] fix: npu aarch64 dependency resolution and uv installation guide by @Crystal-jiang in #832
  • [ci] fix: run wan e2e in bf16 by @Luosuu in #835
  • [data, trainer] feat: support effective-token dynamic batching by @zhangxin81 in #833
  • [trainer] fix: skip NCCL destroy on normal trainer exit by @Luosuu in #849
  • [ops, model] feat: add FlashMLA cuDNN DSA backend by @Luosuu in #850
  • [data, model] fix: coalesce qwen3.5 pad cu-seqlens for GDN by @YLlllllllllll in #845
  • [model] feat: add GPT-OSS VeOmni backend by @Luosuu in #854
  • [model] feat: add LTX-2.3 source code by @JimmyMrr in #858
  • [model, ci] feat: support deepseek v4 by @zhangxin81 in #840
  • [model, ci] fix: GPT-OSS e2e parametrization by @Luosuu in #861
  • [config] fix: auto-resolve ops defaults to NPU-compatible by @Crystal-jiang in #863
  • [ci, lora] test: add LoRA save/load case to NPU unit tests by @xssty123 in #851
  • [data] feat: add data-pipeline dry-run script (tasks/data_sim.py) by @LiuzcEECS in #865
  • [dist] fix: correct FSDP2 grad norm reduction by @Zx55 in #862
  • [ops, dist] feat: support swiglu limit in fused moe by @zhangxin81 in #842
  • [ops] fix: disable quack varlen wgrad tuning to avoid long hang by @Luosuu in #855
  • [trainer] feat: support HSDP for expert parallel by @wuxibin89 in #867
  • [data] fix: loss_mask checking in chatml template by @piamo in #869
  • [data] fix: mask assistant prefix from loss in qwen-series chat template by @Coach257 in #873
  • [docs] fix: Add a list of supported NPU hardware. by @Seren-hao in #885
  • [ci] chore: bump NPU torch/torch-npu to 2.9.0.post2 and CI image to 9.0.0 by @lihanwen7 in #874
  • [lora] fix: support PEFT LoRA on fused MoE experts by @Liuweixiong0118 in #866
  • [ci] chore: update NPU CI image to torch2.9.0-latest tag by @lihanwen7 in #887
  • [docker] feat: add AMD ROCm support (Dockerfile, train.sh, docs) by @amd-fuweiy in #890
  • [ops, perf] refactor: compute MoE scatter-index in O(N) instead of two sorts by @zhangxin81 in #888
  • [dist] feat: per-rank ExtraParallel-slice streaming weight loader + FSDP2 build fixes by @Coach257 in #889
  • [ops, model] feat: add NPU backend for Qwen3.5 gated delta-rule ops by @xssty123 in #879
  • [lora] feat: native PEFT-free LoRA stack with MoE-LoRA + expert-parallel support by @Coach257 in #758
  • [optim, ckpt, ci] fix: drop empty VLM param groups for DCP resume by @YLlllllllllll in #878
  • [ckpt] feat: add dcp_save_to_lowest_rank option for DCP save planner by @williamLyh in #891
  • [perf] fix: add B300 flops support by @KaijingOfficial in #895
  • [trainer] fix: pass labels to model in DPO concatenated_forward by @Winter-Dry in #894
  • [dist, trainer] feat: add FSDP2 torch compile support by @zhangxin81 in #881
  • [config, agent] feat: support GPU dependencies on aarch64 by @Luosuu in #901
  • [model] fix: handle DeepSeek V4 FP8 checkpoint scales by @Luosuu in #899
  • [ci, ckpt] chore: bump NPU torch/torch-npu to 2.10.0 by @lihanwen7 in #904
  • [model] fix: gather input_ids along sequence dim for SP placeholder mask by @cben484 in #905
  • [docs] fix: refresh documentation references by @WenzheWang in #906
  • [model] fix: pass swiglu limit in deepseek v4 fused moe by @zhangxin81 in #868
  • [model, ci] fix: load DeepSeek V4 Flash checkpoints by @Luosuu in #902
  • [trainer] feat: add channel loss logging callback by @YLlllllllllll in #892
  • [model,ops] feat: add DeepSeek V4 TileLang kernels by @Luosuu in #912
  • [model] feat: add ltx2-3 model by @JimmyMrr in #859
  • [BREAKING][dist, trainer] feat: per-module local parallel state via context manager by @Coach257 in #893
  • [agent, config, docs] fix: sync documentation with main by @WenzheWang in #914
  • [dist, trainer, config, docs, ci, task] feat: add packed ChunkMBS support by @WenzheWang in #898
  • [ops] fix: disable Quack GEMM autotuning by @Luosuu in #922
  • [data, model] fix: coalesce pad_to_length FA cu-seqlens on NPU by @YLlllllllllll in #918
  • [ckpt] fix: disable distributed HF save on NPU to avoid intermittent bf16 mismatch under HSDP by @lihanwen7 in #919
  • [model, config, docs, ci] feat: support dense Qwen3.5 ChunkMBS by @WenzheWang in #923
  • [dist, lora] feat: MoE ep_sharded_stream_load for PEFT models + NPU fused MoE-LoRA EP by @Coach257 in #915
  • [config] feat: add DeepSeek V4 training config by @FoolPlayer in #925
  • [model, data, dist] feat: add GPT-OSS 120B EP4 LoRA recipe by @Luosuu in #921
  • [ci] chore: update NPU test images to torch2.10.0-latest by @lihanwen7 in #909
  • [docker] refactor: add ascend 9.0.0 a2.x86 to matrix.yaml and regenerate from template by @lihanwen7 in #913
  • [config, dist, trainer, docs] feat: expose checkpoint early-stop setting by @Luosuu in #926
  • [docker] chore: install openssh server in CUDA 13 image by @FoolPlayer in #937
  • [BREAKING][config, model, ops] refactor: unify DeepSeek V4 kernel naming by @FoolPlayer in #934
  • [model] fix: align DeepSeek V4 runtime numerical semantics by @Luosuu in #928
  • [ckpt, ci, agent] fix: handle integer tensors in HF consolidation by @FoolPlayer in #935
  • [perf] feat: support DeepSeek V4 MFU calculation by @FoolPlayer in #944
  • [ops, model, config, ci, docs] feat: enable Liger kernels for DeepSeek V4 by @FoolPlayer in #946
  • [ops, model, config] feat: support Qwen3-Omni-MoE training on Ascend NPU with fused RoPE by @vvyuervv in #910
  • [model, dist, config, ci] feat: add DeepSeek-V4 Ulysses sequence parallel by @FoolPlayer in #949
  • [dist, trainer, ckpt] fix: skip redundant HF weight loading on DCP resume by @Luosuu in #955
  • [optim, trainer, config, docs, ci] feat: add Gram Newton-Schulz backends for Muon by @FoolPlayer in #953
  • [ops, dist] feat: add FlexAttention with Ulysses sequence parallelism by @Zx55 in #945
  • [trainer] fix: normalize DiT gradient accumulation loss by @651961 in #963
  • [model, parallel, perf] fix: optimize DeepSeek-V4 sparse indexing with SP by @FoolPlayer in #971
  • [optim, ci] feat: optimize Muon FSDP2 all-to-all by @FoolPlayer in #973
  • [ops] feat: add AscendC fused backend (npu_ascendc) for Qwen3.5 chunk gated delta rule by @xssty123 in #924
  • [optim] feat: Support head group split on Muon by @FoolPlayer in #980
  • [ops] feat: replace RMSNorm with ascend fused npu_rms_norm operator by @vvyuervv in #978
  • [agent, docs] feat: add Cursor Cloud (CPU-only) dev environment notes by @Coach257 in #997
  • [logging] fix: Format count_flops for reusability and correctness by @wyettzeng in #1003
  • [perf, ops, model] fix: precompute varlen metadata for AscendC GDN kernels and optimize NPU sync for Qwen3.5 by @Zhang1Sheng in #999
  • [logging, lora] feat: Extend count_flops to allow Lora config for Qwen 2/3/3.5 models by @wyettzeng in #983
  • [model] feat: support NPU Deepseek-V4 by @ChibiQuest in #974
  • [perf] fix: use boost-clock peak FLOPS for Blackwell MFU by @FoolPlayer in #986
  • [ops, dist, perf] fix: remove DeepSeek-V4 SP recompiles and device syncs by @FoolPlayer in #981
  • [model] feat: Deepseek V4 export fp8/fp4 quant weights by @wuxibin89 in #962
  • [trainer, perf] feat: make train step sync optional by @zhangxin81 in #938
  • [data, perf] feat: expose dataloader scheduling knobs by @zhangxin81 in #940
  • [trainer, perf] feat: reuse reduced loss denominators by @zhangxin81 in #941
  • [data] feat: support map-style datasets in worker-side dynamic batching by @LiuzcEECS in #853
  • [dist, trainer, ci] feat: support Qwen3-VL FSDP2 torch compile by @WenzheWang in #1009
  • [ops] fix: preserve fused-linear trunk gradients for non-contiguous inputs by @luzy99 in #1017
  • [ops, perf] refactor: reuse chunk loss denominator by @zhangxin81 in #943
  • [docs] feat: document NPU training for Qwen3.5, Qwen3-Omni and LTX-2.3 by @xssty123 in #1020
  • [docs] feat: add SeedOss training example guide by @Crystal-jiang in #1023
  • [perf] fix: correct Qwen2.5-VL window attention FLOPs by @Stonesjtu in #1024
  • [lora, trainer] feat: Add lora support for VLM trainer by @wyettzeng in #1018
  • [trainer, perf] fix: apply HSDP all-reduce control to custom trainers by @WenzheWang in #1026
  • [model] fix: restore the DeepSeek-V4 NPU patchgen imports by @FoolPlayer in #1013
  • [ops, perf] fix: cut DeepSeek-V4 sparse MLA kernel time by @FoolPlayer in #1012
  • [docs] feat: add Ascend Docker image overview and supported tags by @xssty123 in #1046
  • [misc] chore: widen the uv required-version ceiling to <0.13 by @TimYangst in #1047
  • [ci] feat: add CodeRabbit review configuration by @FoolPlayer in #1049
  • [model] fix: DeepSeek V4 cast q to bfloat16 to use tilelang kernel by @wuxibin89 in #1016
  • [ops, dist] feat: add MagiAttention with SM90 and Ulysses support by @Zx55 in #1021
  • [perf, data] fix: Avoid slow get_vocab for better preprocess performance. by @Shangwei-Li in #1039
  • [model] fix: apply attention mask in flux flash-attention path by @rootkiller6788 in #1051
  • [model] fix: use non-causal attention in flux modeling by @rootkiller6788 in #1050
  • [misc] fix: install flash-qla from its PyPI release instead of a git rev by @TimYangst in #1055
  • [data] fix: make mapping cycles deterministic by @WenzheWang in #1045
  • [trainer] fix: Modify parameters so that logs are not printed repeatedly by @Seren-hao in #1091
  • [docker] feat: add Ascend GDN Dockerfiles by @xssty123 in #1084
  • Replace deprecated v2.ToTensor() with v2.ToImage() + v2.ToDtype(scale=True) by @xyf5432 in #1103
  • [optim, dist, ci] fix: keep the Muon all-to-all path on HSDP meshes by @FoolPlayer in #1011
  • [dist] fix: reduce projection weight & bias grads over batch dim by @rootkiller6788 in #1080
  • [model, config] feat: add MiniMax H3 model support by @vvyuervv in #1075
  • [dist, ci] fix: make the muon_expert_zero_comm guard check the plan by @Coach257 in #1095
  • [lora,model] feat: Add Qwen3.5-MoE LoRA training support by @qcm1 in #1094
  • [BREAKING][omni, model, data] chore: remove the SeedOmni V1 model stack by @Coach257 in #1082
  • [optim, trainer] refactor: read Muon hyperparameters from OptimizerConfig inside build_optimizer by @Coach257 in #1093
  • [dist, parallel] fix: correct DDP gradient sync under Ulysses SP by @Coach257 in #1086
  • [parallel, perf] refactor: use heap for encoder data-balance scheduler by @Stonesjtu in #1118
  • [model, ops, perf] refactor: build DeepSeek-V4 sparse indices without a dense mask, and fuse its RoPE by @FoolPlayer in #1107
  • [BREAKING][dist, trainer, config, docs, ci] chore: remove ChunkMBS by @Luosuu in #1119
  • [dist, parallel, ci] feat: context-parallel infrastructure for DeepSeek-V4 by @FoolPlayer in #1111
  • [dist] feat: add shared async Ulysses backward helpers and op wrappers by @Zx55 in #1113
  • [docker, docs] feat: add CANN 9.1.0 Ascend images by @xssty123 in #1128
  • [ops] feat: add the TileLang teacher-distribution kernel for the DeepSeek-V4 indexer by @FoolPlayer in #1109
  • [trainer, ci] feat: report model aux_metrics without folding them into the loss by @FoolPlayer in #1110
  • [trainer] fix: average DPO metrics across gradient accumulation steps by @FoolPlayer in #1130
  • [model, dist] feat: support Ulysses sequence parallelism in MiniMax H… by @vvyuervv in #1120
  • [docker] chore: drop the Dockerfile template generator by @FoolPlayer in #1133
  • [misc] feat: Add Cambricon MLU support for VeOmni by @sc1915 in #903
  • [dist, trainer, config, docs] feat: add MindSpeed-style async activation offload by @YLlllllllllll in #965
  • [BREAKING][optim, config, docs] feat: qualify head-split Muon module names instead of a DSA indexer flag by @FoolPlayer in #1108
  • [docs] fix: clarify Ascend Docker page titles by @xssty123 in #1141
  • [model, ci] fix: pad Wan sequence before Ulysses SP slice to fix unpatchify mismatch by @OnePunchMonk in #1139
  • [model, dist, ci] feat: context-parallel DeepSeek-V4 attention, compressors and indexer by @FoolPlayer in #1131
  • [ci, agent] feat: verify repo paths and skill refs in agent docs by @FoolPlayer in #1144
  • [ops] fix: make fla_npu import lazy in AscendC flash_gated_delta_rule by @TimYangst in #1150
  • [docs, ci] fix: enforce warning-free documentation builds by @WenzheWang in #1123
  • [model, ops, config, docs] feat: add the DeepSeek-V4 Lightning Indexer KL objective by @FoolPlayer in #1134
  • [ops] feat: add Ascend Triton clamped SwiGLU by @WenzheWang in #1104
  • [optim, ckpt] feat: support Qwen3.5 MoE Muon training on Ascend (#1036) by @Lihui-Gu in #1092
  • [lora, perf] refactor: accumulate independent MoE-LoRA gate and up projections by @Levius-Fubuki in #1154
  • [lora, perf] refactor: avoid one-hot routing in eager MoE LoRA by @Levius-Fubuki in #1155
  • [model] feat: DeepSeek V4 QAT fake quant training by @wuxibin89 in #1089
  • [ckpt] fix: await pending async save at train end by @TimYangst in #1162
  • [model] feat: add Qwen3.5 MoE causal LM expert parallel plan by @yeyeyeping in #1106
  • [ops, ci, docs, agent] chore: make MagiAttention an optional extra by @Coach257 in #1160
  • [model, ci] fix: Wan Ulysses SP sync-path attention correctness by @OnePunchMonk in #1158
  • [dist, parallel] fix: preserve shared gather gradients by @0z5a in #1159

New Contributors

Full Changelog: v0.1.11...v0.1.12

Contributors

TimYangst, Stonesjtu, and 44 other contributors
Assets 2
Loading