Repository navigation
v1.7.0
MTP Subsystem Rewrite
model/modules/mtp_layer.pyremoved and replaced bymodel/mtp.py+model/mtp_detect.py. (#237)- Instead of branching on a version number, the bridge now probes the installed upstream code (
MtpCapabilities,process_mtp_losssignature sniffing), so one code path covers Megatron-Core 0.18–0.20 and the dev branch. - MTP depth and cross-layer weight sharing are resolved automatically from the checkpoint's
safetensorskeys plus the HF config (resolve_mtp_config, now exported from the package root). - The MTP template is resolved from the full decoder layer list rather than the local rank shard. (#256)
Megatron-Core Compatibility
- Minimum raised to 0.18 (#247) and the upper bound widened to
<0.21, covering the 0.20 release and the latest dev branch. (#252) - DeepSeek-V4 layer-spec patching now handles builders that are
functools.partialobjects (Megatron dev) as well as plainModuleSpec(0.18–0.20), so the compressor / indexer /DSv4HybridSelfAttentionreplacements apply on every release instead of raising on dev. (#260) - The GLM-5.3 dev patch was reworked and substantially reduced; apply it with
tools/apply_megatron_patch.py. (#240)
New Models
nemotron_h_omni— NVIDIA Nemotron-3.5 Super-VL bridge spec. (#235)nemo_speechlm— NVIDIA Nemotron-3.5 Transcribe bridge spec. (#244)
Parallelism & Gradient Correctness
- GDN sequence parallel:
_force_sequence_parallel_offnow clearssequence_parallelon the projections and on their parameters, so the gradient finalizer no longer sums an already-complete copy — this was the reported 4x gamma gradient at TP4. Pinned by a two-rank gloo regression. (#253, #254) - PLE: gradients fixed under sequence parallelism. (#234)
- QSA: packed attention is now built on each document's own block grid instead of the global sequence. (#238)
- LoRA / NPU: row-parallel LoRA input sizing corrected for TENPU (#255); LoRA and expert handling adapted to Megatron-Core 0.18 + TENPU. (#189)
DeepSeek-V4.1
- Requantization on export (#259): exporting a quantized DeepSeek-V4.1 checkpoint now restores the official layout the BF16 training parameters were loaded from — fp8-e4m3 with 32x32-block e8m0 scales for linears, packed MXFP4 with per-32 scales for routed experts — instead of writing everything back as BF16. This is gated on the checkpoint's own
quantization_configdeclaration, not on a user switch, and the numerics mirror the DeepGEMM reference (utils/quantizer.py). Exported artifacts change dtype in this release. - FP8-resident Engram tables (#259):
model/modules/engram_fp8.pykeeps the checkpoint's fp8 bytes and e8m0 scales in place and decodes only the rows a forward actually touches, instead of materializing 2x bf16 row shards; loading becomes a byte copy. Requires the MegatronEPShardedEmbeddingTableAPI (0.20 / dev) and falls back to the BF16 table path when it is absent; a missingengram.embed.weight_scale_invon an fp8 store now raises instead of silently mis-loading. - Language-only checkpoints no longer run visual preprocessing during weight conversion, while the multimodal parent conversion and image-marker order are preserved. (#248)
- Block32 FP8 source projections and Engram WKV weights are dequantized with the existing
Fp8Dequantizerfor BF16 loading (F32/UE8M0 scales, both TP split dimensions, lazy Engram rows); FP8 training parameters are rejected on this path. (#249)
Gemma 4
- Per-layer attention dimensions read correctly for
transformers>=5.15, whereglobal_head_dim/num_global_key_value_headsmoved intoper_layer_config. (#239) - Optional
ffpa-attnkernel forhead_dim=512full-attention layers on packed (THD) inputs, enabled withGEMMA4_FFPA_ATTN=1; avoids materializing ans x sscore matrix on Hopper, where Transformer Engine has no fused backward. Sliding-window layers, non-packed inputs and non-SM90 GPUs keep using TE, with explicit checks for a missing backend, CP, dropout, masks, attention bias and padded packed sequences. (#243)
Testing & Tooling
- New CPU-only regressions: DeepSeek-V4.1 block32 loading and language-only preprocessing, GDN SP projection parameters, Gemma 4 per-layer config and ffpa-attn dispatch/recompute, LoRA row-parallel sizing, PLE SP/TP gradients, QSA tile index and sparse kernels, Nemotron Omni and NemoSpeechLM.
megatron_utilsandpatcherslimmed as version gating moved behind capability probes.
New Contributors
Full Changelog: v1.6.5...v1.7.0