Skip to content

[https://nvbugs/6095421][fix] Update resolve_moe_backend#14282

Merged
lfr-0531 merged 5 commits into
NVIDIA:mainfrom
heyuhhh:user/yuhangh/fix_6095421
May 20, 2026
Merged

[https://nvbugs/6095421][fix] Update resolve_moe_backend#14282
lfr-0531 merged 5 commits into
NVIDIA:mainfrom
heyuhhh:user/yuhangh/fix_6095421

Conversation

@heyuhhh
Copy link
Copy Markdown
Collaborator

@heyuhhh heyuhhh commented May 19, 2026

Summary by CodeRabbit

  • New Features

    • Improved MOE backend selection to account for quantization configuration, enabling optimal backend resolution when using FP8_BLOCK_SCALES quantization with specific GPU architectures.
  • Tests

    • Activated a previously waived accuracy test for enhanced validation coverage.

Review Change Stack

Description

Follows PR#14182 to complete the fix

Test Coverage

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

tensorrt-cicd and others added 3 commits May 17, 2026 20:37
DeepSeek-V3-Lite-fp8 disagg test hangs on GB300 because resolve_moe_backend()
returns CUTLASS for non-GptOss MoE models, and the CUTLASS FP8_BLOCK_SCALES
path uses a DeepGEMM JIT kernel that only supports Hopper (SM90). On Blackwell
(SM100/103) the worker throws 'fp8 blockscale gemm only support Hopper' during
warmup; the disagg parent then deadlocks waiting for MPI_READY.

Route AUTO moe_backend to DEEPGEMM on SM100/103, which natively supports these
architectures with FP8_BLOCK_SCALES.

Signed-off-by: tensorrt-cicd <90828364+tensorrt-cicd@users.noreply.github.com>
Signed-off-by: tensorrt-cicd <90828364+tensorrt-cicd@users.noreply.github.com>
Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com>
@heyuhhh heyuhhh requested a review from a team as a code owner May 19, 2026 02:32
@heyuhhh
Copy link
Copy Markdown
Collaborator Author

heyuhhh commented May 19, 2026

/bot run --disable-fail-fast

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 19, 2026

📝 Walkthrough

Walkthrough

This PR introduces quantization-aware MOE backend resolution by extending the resolve_moe_backend method with an optional quant_config parameter and refactoring from_pretrained into a two-phase process that loads quantization configuration before finalizing backend selection. A test waiver is removed as no longer necessary.

Changes

Quantization-aware MOE Backend Selection

Layer / File(s) Summary
Enhanced resolve_moe_backend method with quantization awareness
tensorrt_llm/_torch/model_config.py
Method signature accepts optional quant_config parameter. New quantization-aware logic detects FP8_BLOCK_SCALES algorithm on SM 100f GPUs and returns "DEEPGEMM" backend. Import formatting updated for multi-line block.
Two-phase MOE backend resolution in from_pretrained
tensorrt_llm/_torch/model_config.py
from_pretrained refactored to compute MOE backend hint first using architecture only, then load quantization config, then finalize backend resolution with loaded quant_config to enable quantization-aware backend selection.
Test waiver cleanup
tests/integration/test_lists/waives.txt
Removed waiver for Qwen3_30B_A3B MXFP8 latency test, restoring it to active test execution.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Suggested reviewers

  • yuxianq
  • niukuo
  • zhenhuaw-me
  • jieli-matrix
🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check ⚠️ Warning The PR description is minimal and lacks critical technical details. It references PR#14182 but provides no explanation of what is being fixed, why, or the technical changes involved. Expand the description to explain the issue being fixed (MoE backend selection on Blackwell with FP8_BLOCK_SCALES), the solution approach, and reference relevant test coverage. Include why this follows PR#14182.
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The PR title clearly and specifically describes the main change: updating the resolve_moe_backend method to be quantization-aware on specific hardware.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@tensorrt-cicd
Copy link
Copy Markdown
Collaborator

PR_Github #49042 [ run ] triggered by Bot. Commit: b38eb84 Link to invocation

@tensorrt-cicd
Copy link
Copy Markdown
Collaborator

PR_Github #49042 [ run ] completed with state SUCCESS. Commit: b38eb84
/LLM/main/L0_MergeRequest_PR pipeline #38778 completed with status: 'SUCCESS'

CI Report

Link to invocation

@heyuhhh heyuhhh requested a review from lfr-0531 May 19, 2026 12:36
Comment thread tensorrt_llm/_torch/model_config.py Outdated
heyuhhh added 2 commits May 19, 2026 15:25
Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com>
@heyuhhh
Copy link
Copy Markdown
Collaborator Author

heyuhhh commented May 19, 2026

/bot run --disable-fail-fast

@tensorrt-cicd
Copy link
Copy Markdown
Collaborator

PR_Github #49224 [ run ] triggered by Bot. Commit: 25d6fd7 Link to invocation

@tensorrt-cicd
Copy link
Copy Markdown
Collaborator

PR_Github #49224 [ run ] completed with state SUCCESS. Commit: 25d6fd7
/LLM/main/L0_MergeRequest_PR pipeline #38897 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

CI Report

Link to invocation

@lfr-0531 lfr-0531 merged commit fd54508 into NVIDIA:main May 20, 2026
9 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants