[None][chore] update DeepGEMM to 2.6.1 by Barry-Delaney · Pull Request #16673 · NVIDIA/TensorRT-LLM

Barry-Delaney · 2026-07-21T11:36:52Z

Summary

Update the bundled DeepGEMM from 2.5.0 (245dc5d) to 2.6.1 (f8e8fb5).

Adapt the fused mHC TF32 pmap GEMM integration to DeepGEMM's new header and namespace layout.
Update MegaMoE symmetric-buffer allocation to the current DeepGEMM API.
Refresh DeepGEMM attribution metadata and build-input hashes.
Include DeepGEMM #384, which fixes a synchronization/ordering race in the SM100 MQA kernels using FP8 Q/K and FP16 weights.

Validation

Exact TensorRT-LLM base: d75cb89a0e.
Clean SM100 native build and wheel packaging: passed.
Fresh wheel-only provenance validation: passed; TensorRT-LLM, DeepGEMM 2.6.1, and the native DeepGEMM binding were loaded from the candidate wheel.
Targeted B200 pytest cases: 33 passed, 0 failed.
- MegaMoE DeepGEMM: 4/4
- mHC fused backends: 6/6
- FP8/FP4 MQA and fused-cat coverage: 23/23
Pre-commit on all five changed files and git diff --check: passed.
SBOM database validation: passed; all 133 DeepGEMM inputs observed in the clean build were mapped (0 missing).

MegaMoE performance

MegaMoE EP8 microbenchmark on 8x B200, using e384/k6/h7168/i3072, FP8xFP4, no shared expert, and A1/B1/B2/A2 ordering. Each cell contains six repeated outer samples, with each sample averaging ten kernel launches. Lower latency is better.

Workload	Cache	DeepGEMM 2.5.0 (us)	DeepGEMM 2.6.1 (us)	Reduction	ABBA-adjusted reduction
Decode, 32 tokens/rank	Cold	319.224 +/- 0.347	313.926 +/- 0.181	1.659%	1.637%
Decode, 32 tokens/rank	Hot	306.779 +/- 0.628	301.288 +/- 0.543	1.790%	1.846%
Prefill, 8192 tokens/rank	Cold	2,957.500 +/- 5.500	2,701.000 +/- 3.500	8.673%	8.754%
Prefill, 8192 tokens/rank	Hot	2,970.000 +/- 8.500	2,709.000 +/- 7.000	8.788%	8.705%

Values are median +/- MAD. The symmetric buffer decreased from 6.772 GiB to 1.716 GiB (3.95x smaller). Fixed rank-0 output samples were finite, deterministic within each run, and byte-identical across versions for both tested shapes.

Dev Engineer Review

Updated bundled DeepGEMM to 2.6.1 (via 3rdparty/fetch_content.json) and refreshed DeepGEMM attribution/build-input hash mappings (scripts/attribution/data/dependency_metadata.yml, scripts/attribution/data/files_to_dependency.yml), including the DeepGEMM PR #384 fix for an SM100 MQA kernel synchronization/ordering race.
Updated cpp/tensorrt_llm/kernels/mhcKernels/fused_tf32_pmap_gemm.cuh to match DeepGEMM’s new header/namespace layout:
- Removed previously used tma_copy helper/alias usage and switched TMA tile transfers to fully-qualified deep_gemm::tma::copy<...>(...) call sites for the residual and B tiles.
- Adjusted SM90 LDSM helper selection for x_vals/r_vals from deep_gemm::sm90::...::copy to deep_gemm::ptx::...::copy.
- Switched UMMA descriptor construction/advancement to locally named makeBUmmaDescriptor / advanceBUmmaDescriptorLo usage.
Updated MegaMoE DeepGEMM symmetric-buffer allocation API in tensorrt_llm/_torch/modules/fused_moe/mega_moe/mega_moe_deepgemm.py to pass required DeepGEMM allocation configuration (num_shared_experts=0, mma_type="fp8xfp4") and changed the activation argument to keyword form (activation=self.activation).
Reduced MegaMoE symmetric buffer memory footprint (reported 6.772 GiB → 1.716 GiB) consistent with the DeepGEMM/allocator API changes.
Validation: clean SM100 builds, wheel packaging/provenance, targeted B200 tests, pre-commit checks, diff validation, and SBOM mapping; performance measurements show ~1.6–1.8% lower decode latency and ~8.7–8.8% lower prefill latency.
CI note: bot-triggered CI runs failed for commits 23a5473 (pipeline #49028), 7e3c861 (pipeline #49168), and 62cb1aa (pipeline #49395); each requested review/fixes of failing tests and reruns (after NVIDIA-team-triggered attempts).

QA Engineer Review

No test changes.

Barry-Delaney · 2026-07-21T17:10:59Z

/bot run

coderabbitai · 2026-07-21T17:14:14Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

Walkthrough

DeepGEMM is updated to a new revision, attribution mappings are synchronized, fused TF32 pmap kernels use updated DeepGEMM namespaces and descriptor helpers, and MegaMoE supplies expanded SymmBuffer allocation arguments.

Changes

DeepGEMM integration update

Layer / File(s)	Summary
DeepGEMM revision and attribution metadata `3rdparty/fetch_content.json`, `scripts/attribution/data/*`	Updates DeepGEMM commit references and attribution mappings, including additional CUTLASS hash entries.
Fused TF32 pmap kernel integration `cpp/tensorrt_llm/kernels/mhcKernels/fused_tf32_pmap_gemm.cuh`	Updates DeepGEMM includes, TMA and PTX load helper references, and B-matrix descriptor helpers in both kernels.
MegaMoE SymmBuffer allocation contract `tensorrt_llm/_torch/modules/fused_moe/mega_moe/mega_moe_deepgemm.py`	Passes explicit shared-expert count, MMA type, and keyword activation arguments to SymmBuffer allocation.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Suggested reviewers: niukuo, mzweilz, sunnyqgg, rosong11

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise and accurately summarizes the DeepGEMM 2.6.1 update.
Description check	✅ Passed	The PR description is detailed and covers the summary, validation, and performance, though some template sections are omitted.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

tensorrt-cicd · 2026-07-21T17:16:26Z

PR_Github #60741 [ run ] triggered by Bot. Commit: 23a5473 Link to invocation

tensorrt-cicd · 2026-07-21T17:35:43Z

PR_Github #60741 [ run ] completed with state FAILURE. Commit: 23a5473
/LLM/main/L0_MergeRequest_PR pipeline #49028 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Barry-Delaney · 2026-07-22T05:39:25Z

/bot run

tensorrt-cicd · 2026-07-22T05:45:00Z

PR_Github #60900 [ run ] triggered by Bot. Commit: 7e3c861 Link to invocation

tensorrt-cicd · 2026-07-22T06:03:12Z

PR_Github #60900 [ run ] completed with state FAILURE. Commit: 7e3c861
/LLM/main/L0_MergeRequest_PR pipeline #49168 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Barry-Delaney · 2026-07-23T01:54:40Z

/bot run

tensorrt-cicd · 2026-07-23T02:00:32Z

PR_Github #61147 [ run ] triggered by Bot. Commit: 62cb1aa Link to invocation

tensorrt-cicd · 2026-07-23T02:20:40Z

PR_Github #61147 [ run ] completed with state FAILURE. Commit: 62cb1aa
/LLM/main/L0_MergeRequest_PR pipeline #49395 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Barry-Delaney · 2026-07-23T14:58:38Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-23T15:04:36Z

PR_Github #61313 [ run ] triggered by Bot. Commit: 4106f8b Link to invocation

tensorrt-cicd · 2026-07-23T18:26:20Z

PR_Github #61313 [ run ] completed with state FAILURE. Commit: 4106f8b
/LLM/main/L0_MergeRequest_PR pipeline #49542 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Barry-Delaney · 2026-07-23T18:59:12Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-24T01:38:43Z

PR_Github #61426 [ run ] completed with state FAILURE. Commit: 4106f8b
/LLM/main/L0_MergeRequest_PR pipeline #49651 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Barry-Delaney · 2026-07-27T01:09:24Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-27T01:15:27Z

PR_Github #61767 [ run ] triggered by Bot. Commit: 8e999ba Link to invocation

tensorrt-cicd · 2026-07-27T04:49:53Z

PR_Github #61767 [ run ] completed with state FAILURE. Commit: 8e999ba
/LLM/main/L0_MergeRequest_PR pipeline #49968 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Barry-Delaney · 2026-07-27T06:37:35Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-27T06:43:04Z

PR_Github #61838 [ run ] triggered by Bot. Commit: 8e999ba Link to invocation

tensorrt-cicd · 2026-07-27T07:45:41Z

PR_Github #61838 [ run ] completed with state SUCCESS. Commit: 8e999ba
/LLM/main/L0_MergeRequest_PR pipeline #50033 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Barry-Delaney · 2026-07-27T10:44:44Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-27T10:50:03Z

PR_Github #61893 [ run ] triggered by Bot. Commit: 8e999ba Link to invocation

tensorrt-cicd · 2026-07-27T14:39:59Z

PR_Github #61893 [ run ] completed with state FAILURE. Commit: 8e999ba
/LLM/main/L0_MergeRequest_PR pipeline #50084 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Barry-Delaney · 2026-07-27T17:19:26Z

/bot run --disable-fail-fast

Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com>

tensorrt-cicd · 2026-07-27T17:27:43Z

PR_Github #61955 [ run ] triggered by Bot. Commit: ffc23b8 Link to invocation

tensorrt-cicd · 2026-07-27T21:31:55Z

PR_Github #61955 [ run ] completed with state SUCCESS. Commit: ffc23b8
/LLM/main/L0_MergeRequest_PR pipeline #50144 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Barry-Delaney · 2026-07-28T20:52:24Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-28T20:58:53Z

PR_Github #62302 [ run ] triggered by Bot. Commit: ffc23b8 Link to invocation

tensorrt-cicd · 2026-07-29T00:01:14Z

PR_Github #62302 [ run ] completed with state FAILURE. Commit: ffc23b8
/LLM/main/L0_MergeRequest_PR pipeline #50471 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Barry-Delaney · 2026-07-29T06:07:01Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-29T06:13:56Z

PR_Github #62446 [ run ] triggered by Bot. Commit: ffc23b8 Link to invocation

tensorrt-cicd · 2026-07-29T10:10:25Z

PR_Github #62446 [ run ] completed with state SUCCESS. Commit: ffc23b8
/LLM/main/L0_MergeRequest_PR pipeline #50601 completed with status: 'SUCCESS'

CI Report

Link to invocation

sunnyqgg

LGTM

github-actions Bot assigned Barry-Delaney Jul 21, 2026

Barry-Delaney force-pushed the user/jinshik/update-deepgemm-nv-dev branch 2 times, most recently from 7532f36 to 23a5473 Compare July 21, 2026 17:10

Barry-Delaney marked this pull request as ready for review July 21, 2026 17:10

Barry-Delaney requested review from a team as code owners July 21, 2026 17:10

Barry-Delaney requested review from mzweilz, niukuo, rosong11 and sunnyqgg July 21, 2026 17:10

Barry-Delaney force-pushed the user/jinshik/update-deepgemm-nv-dev branch from 23a5473 to 7e3c861 Compare July 22, 2026 05:39

tburt-nv approved these changes Jul 22, 2026

View reviewed changes

Barry-Delaney force-pushed the user/jinshik/update-deepgemm-nv-dev branch from 7e3c861 to 62cb1aa Compare July 23, 2026 01:54

Barry-Delaney force-pushed the user/jinshik/update-deepgemm-nv-dev branch from 512c6e9 to 4106f8b Compare July 23, 2026 14:58

Barry-Delaney force-pushed the user/jinshik/update-deepgemm-nv-dev branch from 4106f8b to 8e999ba Compare July 27, 2026 01:09

lucifer1004 mentioned this pull request Jul 27, 2026

[None][fix] Bump DeepGEMM to nv_dev HEAD for SM120 attention support #16217

Closed

mzweilz approved these changes Jul 27, 2026

View reviewed changes

lucifer1004 mentioned this pull request Jul 27, 2026

[None][feat] Enable DeepSeek-V4 and DSA (DeepSeek-V3.2/GLM) serving on SM120 via FlashInfer sparse-MLA #16224

Open

Barry-Delaney added 2 commits July 28, 2026 01:19

Update DeepGEMM to 2.6.1

f48d6e7

Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com>

Fix mHC UMMA descriptor construction with Clang

ffc23b8

Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com>

Barry-Delaney force-pushed the user/jinshik/update-deepgemm-nv-dev branch from 8e999ba to ffc23b8 Compare July 27, 2026 17:19

sunnyqgg approved these changes Jul 30, 2026

View reviewed changes

rosong11 approved these changes Jul 30, 2026

View reviewed changes

Barry-Delaney merged commit ffba1a6 into NVIDIA:main Jul 30, 2026
7 checks passed

longlee0622 mentioned this pull request Jul 30, 2026

[None][feat] add Kimi K3 SiTU MegaMoE support #17063

Open

Uh oh!

Conversation

Barry-Delaney commented Jul 21, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

MegaMoE performance

Uh oh!

Barry-Delaney commented Jul 21, 2026

Uh oh!

coderabbitai Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Changes

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

Barry-Delaney commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

Barry-Delaney commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

Barry-Delaney commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

Barry-Delaney commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 24, 2026

Uh oh!

Barry-Delaney commented Jul 27, 2026

Uh oh!

tensorrt-cicd commented Jul 27, 2026

Uh oh!

tensorrt-cicd commented Jul 27, 2026

Uh oh!

Barry-Delaney commented Jul 27, 2026

Uh oh!

tensorrt-cicd commented Jul 27, 2026

Uh oh!

tensorrt-cicd commented Jul 27, 2026

Uh oh!

Barry-Delaney commented Jul 27, 2026

Uh oh!

tensorrt-cicd commented Jul 27, 2026

Uh oh!

tensorrt-cicd commented Jul 27, 2026

Uh oh!

Barry-Delaney commented Jul 27, 2026

Uh oh!

tensorrt-cicd commented Jul 27, 2026

Uh oh!

tensorrt-cicd commented Jul 27, 2026

Uh oh!

Barry-Delaney commented Jul 28, 2026

Uh oh!

tensorrt-cicd commented Jul 28, 2026

Uh oh!

tensorrt-cicd commented Jul 29, 2026

Uh oh!

Barry-Delaney commented Jul 29, 2026

Uh oh!

tensorrt-cicd commented Jul 29, 2026

Uh oh!

tensorrt-cicd commented Jul 29, 2026

Uh oh!

sunnyqgg left a comment

Choose a reason for hiding this comment

Barry-Delaney commented Jul 21, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 21, 2026 •

edited

Loading