v1.2.0

中文版

新特性

1. 支持 GLM-5 共享参数 MTP ，可通过mtp_shared_weights参数启用。
2. 支持 Qwen3.5 FP8 训练和权重导入导出。
3. 支持控制 MTP 分支中 decoder_input 是否停止梯度，即 MTP loss 能否直接通过 decoder_input 回传梯度到 Embedding/ViT，使用mtp_decoder_input_detach参数。
4. 昇腾 NPU 训练兼容 megatron-core 0.15.3。

English Version

New Features

1. Added support for GLM-5 shared-weight MTP, which can be enabled via the mtp_shared_weights argument.
2. Added support for Qwen3.5 FP8 training and FP8 weight import/export.
3. Added support for controlling whether gradients are stopped at decoder_input in the MTP branch, i.e., whether the MTP loss can be back-propagated through decoder_input to Embedding/ViT. This can be configured via the mtp_decoder_input_detach argument.
4. Added compatibility with Megatron-Core 0.15.3 for training on Huawei Ascend NPU.

What's Changed

• [docs] update readme by @Jintao-Huang in #17
• [qwen3.5] compat transformers 5.4.0 by @Jintao-Huang in #18
• [bugfix] fix gptq_bridge by @Jintao-Huang in #19
• Revert qwen3.5 save weight by @Jintao-Huang in #20
• [bugfix] fix multimodal mtp by @Jintao-Huang in #21
• update get_parameter_local_cp by @Jintao-Huang in #22
• [bugfix] Fix the multi-LoRA issue in Twinkle by @Jintao-Huang in #24
• Adapt Mindspeed/Megatron 0.15.3 by @addsubmuldiv in #25
• [bugfix] fix qwen3.5 gpt_bridge lora by @Jintao-Huang in #28
• [bugfix] fix gdn sharded_state_dict lora by @Jintao-Huang in #23
• support Qwen3.5 FP8 by @Jintao-Huang in #30
• [bugfix] fix fp8 by @Jintao-Huang in #32
• [bugfix] fix set_module lora by @Jintao-Huang in #33
• [compat] gdn compat mcore main by @Jintao-Huang in #34
• [bugfix] Fix mtp fp8 by @Jintao-Huang in #35
• support mtp_decoder_input_detach by @Jintao-Huang in #37
• [bugfix] fix gate_up_proj by @Jintao-Huang in #39
• fix mtp_num_layer >= 2 multimodal by @Jintao-Huang in #40
• support mtp_shared_weights by @Jintao-Huang in #41
• compat peft 0.19 by @Jintao-Huang in #42
• [bugfix] fix peft_format qwen3_5_moe by @Jintao-Huang in #43
• fix: Add is_mtp parameter to _set_moe_state avoid type error by @0hujun in #45
• [bugfix] fix grpo qwen3_5_moe full by @Jintao-Huang in #46
• [bugfix] fix safe_ddp_context hang by @Jintao-Huang in #47

New Contributors

• @addsubmuldiv made their first contribution in #25
• @0hujun made their first contribution in #45

Full Changelog: v1.1.0...v1.2.0