Skip to content

v1.6.4

Choose a tag to compare

View all tags
@hjh0119 hjh0119 released this 07 Sep 12:22
· 49 commits to main since this release
v1.6.4
73d6807

English Version

mcore-bridge v1.6.4

🚀 New Model Support

✨ Features

  • Refactor Qwen3.5 GDN to use FLA causal conv1d and cu_seq_lens_q, removing manual padding/unpadding for packed sequences (#176)
  • Optimize padding mask reduction memory usage (#167)
  • Compatibility with Megatron-Core 0.16.0/0.16.1: packed sequence length handling (#175) and Nemotron-H import guard (#171)

🐛 Bug Fixes

  • Fix two FP8 training issues: PLE ngram embedding dequantization on FP8 checkpoint loading, and LoRA A/B excluded from fp8 autocast (#183)
  • Fix LoRA under TP + sequence_parallel: sum grads of the replicated LoRA factor over the TP group (#173); fix SP LoRA regression from #162 (#172)
  • Fix QSA with sequence parallelism (#180) and preserve QSA MRoPE batch dimension (#179)
  • Fix num_samples attribute error (#186)

中文版本

mcore-bridge v1.6.4

🚀 新增模型支持

✨ 新特性

  • 重构 Qwen3.5 GDN，改用 FLA causal conv1d 并直接传入 cu_seq_lens_q，去除 packed 序列下的手动 padding/unpadding（#176）
  • 优化 padding mask reduction 的显存占用（#167）
  • 适配 Megatron-Core 0.16.0/0.16.1：packed 序列长度处理（#175）与 Nemotron-H 导入兼容（#171）

🐛 问题修复

  • 修复两个 FP8 训练问题：FP8 checkpoint 加载时 PLE ngram embedding 未按 scale 反量化；LoRA A/B 被 fp8 autocast 误覆盖（#183）
  • 修复 TP + sequence_parallel 下 LoRA 梯度问题：对 TP 组内复制的 LoRA 因子梯度求和（#173）；修复 #162 引入的 SP LoRA 回归（#172）
  • 修复 QSA 序列并行支持（#180）及 QSA MRoPE batch 维度保持（#179）
  • 修复 num_samples 属性错误（#186）

What's Changed

  • fix: make Nemotron-H import error adaption on Megatron-Core 0.16.1 by @hazelduan in #171
  • fix qwen3.5 lora sp by @hjh0119 in #172
  • [model] optimize padding mask reduction memory by @Zhijun-Xu in #167
  • [model] qwen3.8-flash-next by @hjh0119 in #174
  • Fix packed sequence length handling for Megatron-Core 0.16.0/0.16.1 by @hazelduan in #175
  • fix num_samples attributes error by @tastelikefeet in #186
  • fix: support QSA with sequence parallelism by @taking-lying-flat in #180
  • [bugfix] preserve QSA MRoPE batch dimension by @taking-lying-flat in #179
  • [bugfix] LoRA + TP>1 + sequence_parallel: sum grads of the replicated LoRA factor over the TP group (exported adapter had last-layer linear_proj.lora_B == 0) by @hojuna in #173
  • [bugfix] Fix two FP8-training issues: PLE ngram dequantization and LoRA fp8 autocast guard by @ehooon in #183
  • [model] refactor qwen3.5 GDN to use FLA causal conv1d and cu_seq_lens_q by @ys2025-AI in #176
  • optimize qwen3.8-flash-next by @hjh0119 in #185
  • fix qwen4-exp thd cp by @hjh0119 in #187

New Contributors

Full Changelog: v1.6.2...v1.6.4

Contributors

tastelikefeet, hazelduan, and 6 other contributors
Assets 2
Loading