[Feat] Integrate Megatron-LM core_v0.17.1

[XbzOnGit]

Summary

Add Megatron-LM support to DMI through the standalone DMI-Megatron-Integration repository. DMI pins the integration repository as a submodule at commit 7838d16, and that repository pins the matching Megatron-LM-DMI fork at commit 93f45d9, based on Megatron-LM core_v0.17.1.

The integration repository owns the Megatron adapter, hook definitions, metadata propagation, runtime activation, record schemas, ClickHouse materialization and reconstruction, and Megatron-specific validation. DMI core remains framework-neutral.

What changed

• Add third_party/DMI-Megatron-Integration as a reproducibly pinned Git submodule.
• Connect DMI to the Megatron integration package and its matching Megatron-LM fork without copying framework-specific code into DMI core.
• Restore the selective-recomputation gate contract and its documentation and tests: rejected occurrences publish nothing, while accepted zero-row records still publish their descriptor-backed zero-byte tasks.

Compatibility

The gate change affects only rejected generic record-producer occurrences. Existing accepted records, zero-row records, dynamic-size reservation reclamation, legacy inference producers, record schemas, and storage formats remain unchanged.

Existing Hugging Face and vLLM integrations continue to use their established paths. Megatron-specific behavior remains in DMI-Megatron-Integration and Megatron-LM-DMI.

Validation

• Clean environment: Python 3.12.14, PyTorch 2.13.0+cu129, CUDA toolkit 12.9.1, and Transformer Engine 2.14.1.
• Complete Megatron-to-DMI-to-ClickHouse regression: 249 passed, four expected native-Megatron PP=2 CUDA Graph skips, and 0 failed.
• All five Megatron EP reconstruction E2Es passed through Megatron, DMI, ClickHouse write, database query, and reconstruction validation.
• Native record-producer regression: 93 passed.
• Native ring-engine regression: 79 passed.
• Native null-mode regression: 48 passed.
• Record runtime, format, zero-row, and actual-byte reclamation regressions: 17 passed.
• Megatron host replay-selection regressions: 2 passed.
• Megatron TP1 local-CUDA-Graph selective-recomputation DMI-to-ClickHouse E2E: 1 passed.
• Existing Hugging Face and vLLM DMI-to-ClickHouse compatibility matrix: 7/7 cases passed.

Scope

This PR adds the Megatron integration entry point to DMI and includes the core gate correction required by that integration.

[XbzOnGit]

Updated this PR with the Megatron-LM installation and usage documentation:

• Added Megatron-LM to the root README and installation guide alongside the HF and vLLM backends.
• Added docs/megatron.md covering installation order, DMI native rebuild conditions, training launch, hook selection, and persistence modes.
• Updated DMI-Megatron-Integration to 812ea8b, whose installation guide records the required compatibility constraints, tested dependency configuration, pinned Transformer Engine source, and correct DMI build order.
• The integration's core_v0.17.1-dmi and main branches both point to the same integration documentation commit.

This follow-up changes documentation and the integration submodule revision only; it does not change runtime code.

[Copilot]

Pull request overview

This PR integrates Megatron-LM support into DMI by introducing a pinned Megatron integration submodule and aligning DMI’s record “device gate” behavior with the restored selective-recomputation contract (rejected occurrences publish nothing; accepted zero-row occurrences still publish a descriptor-backed zero-byte task).

Changes:

• Add the third_party/DMI-Megatron-Integration Git submodule and document Megatron-LM usage/installation.
• Update record producer kernels so a false device gate returns without publishing any task (no ring counter advancement).
• Update native tests and the integration API documentation to reflect the revised gating contract.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
native/csrc/ring/producer.cu	Changes record producer kernels to skip publication entirely when the device gate rejects.
tests/native/ring/test_producer.cu	Updates gating test to assert “no publication / no ring state change” on gate rejection.
tests/native/ring/test_ring_engine.cu	Removes a record reservation reclaim test that depended on the prior “publish empty task when gated off” behavior.
docs/integration-api-v1.md	Documents the restored contract: gated-off occurrences must be pre-selected on host and be absent from the record stream.
docs/megatron.md	Adds Megatron-LM backend usage guide (installation + activation).
docs/install.md	Updates core installation guide to include the new Megatron integration submodule and checkout/environment guidance.
README.md	Updates project positioning and quick links to include Megatron-LM training support.
.gitmodules	Adds the pinned Megatron integration submodule entry.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Samfisheryu]

Re-tested the reported router-topk capacity-dropping case on one GPU. Training now passes, dropped-route sentinel IDs are preserved, and all dropped-slot weights are zero. No remaining blockers.