v1.6.5

mcore-bridge v1.6.5

This minor release adds DeepSeek-V4.1 (SFT & RL) support on the HybridModel backbone, GLM-5.3-Flash / GLM5-Next (incl. CP and MTP), Qwen4-Exp MTP, and a batch of QSA / CP / LoRA / export fixes since v1.6.4.

Highlights

• DeepSeek-V4.1 support for SFT & RL — new deepseek_v41 bridge (CSA2 compressor/indexer, single-pass mHC, trainable Engram, Vision/Aligner, DSpark MTP draft stack), now on the PP-capable HybridModel backbone for every layout (TP/PP/VPP/CP/EP/DP + packing), with Megatron 0.16–0.19 compatibility. Activation recompute (full/selective) is decided by the V4.1 backbone. (#201, #205)
• New models / backbones — GLM-5.3-Flash (glm5_next) support incl. context parallelism and the updated Megatron GLM patch (#194, #198, #200); MTP support for Qwen4-Exp and GLM5-Next (#204).
• MindSpeed / NPU — NPU LoRA projections (native parallel linears, replicated bases, grouped-expert adapters), Qwen3.8-Flash-Next PLE host-gather fallback, and the zigzag CP index device fix. (#209, #214, #196, #231)
• QSA (Qwen sparse attention) hardening — no cross-document scoring for packed inputs, chunked indexer scoring to bound memory, stable top-k, frozen forward-only indexer, and a QSA_SPARSE_KERNEL env switch to disable the sparse kernel. (#211, #199, #206, #203, #202, #190)
• Environment — the transformers upper bound is removed (transformers>=4.33), so newer transformers releases can be used. (#232)

Features

• Support DeepSeek-V4.1 (dpsk-4.1) SFT & RL: deepseek_v41 bridge with CSA2 compressor/indexer, single-pass mHC, Engram modules, Vision/Aligner and the DSpark (MTP) draft stack; HybridModel backbone for all parallelism layouts incl. VPP; Megatron 0.16–0.19 compat guards. (#201)
• DeepSeek-V4.1: defer the activation-recompute decision to the V4.1 backbone, enabling full/selective recompute with Engram. (#205)
• Support the GLM-5.3-Flash model (glm5_next). (#194)
• Support GLM5-Next context parallelism, with an updated Megatron GLM patch. (#198, #200)
• Support MTP for Qwen4-Exp and GLM5-Next. (#204)
• Add the QSA_SPARSE_KERNEL env var to disable the QSA sparse kernel (falls back to full attention under thd/CP>1, bool mask otherwise). (#190)

Bug Fixes

QSA / sparse attention

• Avoid cross-document QSA scoring for packed inputs. (#211)
• Chunk QSA indexer scoring to avoid OOM on long packed sequences. (#199)
• Make indexer top-k stable. (#206)
• Freeze the forward-only QSA indexer. (#203)
• Preserve Qwen checkpoint exports and bound QSA memory; 64-bit offsets for long-sequence sparse attention. (#202)

Context parallelism / RoPE

• Fix the PLE context-parallel gradient. (#230)
• Keep zigzag CP indexes on the activation device (fixes NPU/CPU). (#214)
• Apply DSv4 RoPE from row-aligned frequencies under packing + CP. (#219)
• Fix flash-decode RoPE caching. (#160)

LoRA / PEFT

• Fix LoRA on replicated TELinear bases. (#228)
• Preserve KDA QKV LoRA adapters on export and freeze unsaved MoE expert bias during LoRA training. (#195)
• Exclude the PLE n-gram base state from PEFT saves. (#220)
• Fix NPU MoE LoRA. (#196)
• Mark NPU grouped LoRA expert adapters as expert parameters (per-expert adapters are no longer reduced over the dense groups; EDP still synchronizes replicas of the same expert). (#231)
• Fix MindSpeed NPU LoRA projections and the Qwen3.8-Flash-Next PLE host gather (pinned-host fast path is now CUDA-only). (#209)

Export / checkpoint

• Fix Qwen4-Exp PLE HF export and TP checkpoint metadata. (#192)
• Fix safetensors export OOM. (#193)
• Fix MTP checkpoint saving with TP > 1 (tp_group fallback to the global TP group). (#210)
• Make fp8 scale_inv contiguous before TP/PP collectives (fixes megatron export --fp8_recipe blockwise --fp8_param_gather true with TP > 1). (#197)
• Preserve generated vision-token text embeddings. (#207)

Tooling / environment

• Verify complete Megatron patches before updating installations. (#208)
• Relax the transformers requirement to transformers>=4.33 (upper bound removed). (#232)

What's Changed

• add env to disable qsa kernel by @hjh0119 in #190
• [model] support glm-5.3-flash by @hjh0119 in #194
• Fix NPU MoE LoRA checkpoint replica IDs by @addsubmuldiv in #196
• [bugfix] Fix Qwen4Exp PLE HF export and TP checkpoint metadata by @Zhikaiiii in #192
• Fix safetensors export OOM by @hjh0119 in #193
• [bugfix] make fp8 scale_inv contiguous before TP/PP collectives by @tastelikefeet in #197
• fix: preserve KDA LoRA weights and freeze unsaved expert bias by @taking-lying-flat in #195
• fix: chunk QSA indexer scoring to avoid OOM on long packed sequences by @hjh0119 in #199
• support glm5-next cp by @hjh0119 in #198
• fix glm5_next cp patch by @hjh0119 in #200
• Support dpsk-4.1 sft and rl by @tastelikefeet in #201
• fix(qwen4): freeze forward-only QSA indexer by @leooop-al in #203
• fix(qwen): preserve checkpoint exports and bound QSA memory by @dingzhiqiang in #202
• support qwen4_exp & glm5_next MTP by @hjh0119 in #204
• feat(deepseek_v41): defer activation-recompute decision to the V4.1 b… by @tastelikefeet in #205
• fix: verify complete Megatron patches before updating installations by @FunJim in #208
• [bugfix] fix MTP checkpoint saving with TP > 1 by @WhichPaths in #210
• fix: preserve generated vision token text embeddings by @dingzhiqiang in #207
• make qsa indexer topk stable by @hjh0119 in #206
• [bugfix] avoid recomputing Flash-decode RoPE cache by @taking-lying-flat in #160
• Avoid cross-document QSA scoring for packed inputs by @yuchenwang3 in #211
• Fix MindSpeed NPU LoRA projections and Qwen3.8-Flash-Next PLE host gather by @addsubmuldiv in #209
• fix(qwen4): exclude PLE n-gram base state from PEFT saves by @0KEAHA in #220
• fix: keep zigzag CP indexes on the activation device by @shiaho777 in #214
• fix: apply DSv4 RoPE from row-aligned frequencies by @shiaho777 in #219
• fix ple cp grad by @hjh0119 in #230
• fix replicated lora by @hjh0119 in #228
• [env] relax transformers bound by @hjh0119 in #232
• fix(lora): mark NPU grouped LoRA expert adapters as expert parameters by @li-lizhe in #231

New Contributors

• @Zhikaiiii made their first contribution in #192
• @leooop-al made their first contribution in #203
• @dingzhiqiang made their first contribution in #202
• @FunJim made their first contribution in #208
• @WhichPaths made their first contribution in #210
• @yuchenwang3 made their first contribution in #211
• @0KEAHA made their first contribution in #220
• @shiaho777 made their first contribution in #214
• @li-lizhe made their first contribution in #231

Full Changelog: v1.6.4...v1.6.5