[None][fix] Relax W8A16 MoE test tolerance for DTP mode

[xxi-nv]

Summary

• Replace strict torch.testing.assert_close with percent-based check_accuracy in W8A16RefGatedMLPFusedMoE
• In DTP/TTP mode (moe_tp_size > 1), TP AllReduce accumulates bf16 rounding errors on top of INT8 quantization error, causing 0.024% element mismatch that exceeds the element-wise atol
• Use percent=0.96 for TP mode (consistent with UnquantizedRefMLPFusedMoE) and percent=0.99 for single-GPU/EP mode

Root cause

When running CUTLASS W8A16 with DTP parallel mode and DeepSeekV3 routing (top_k == num_experts), each expert's weight matrix is split across 4 ranks. Each rank computes a partial GEMM, then AllReduce sums the partials. The AllReduce introduces bf16 rounding error that compounds with INT8 quantization error. Only 1/4096 elements (0.024%) exceed the strict tolerance.

Affected test cases

All 3 failures share: parallel=DTP, backend=CUTLASS, quant=W8A16, routing=DeepSeekV3, e4_k4_h512_i512, seq=8, bfloat16 — only comm method differs (NVLINK_ONE_SIDED / NVLINK_TWO_SIDED / DEEPEP).

Test plan

• Reproduced failure on GB200 before fix
• Verified all 3 test cases pass after fix (3 passed, 0 failed)

🤖 Generated with Claude Code

Summary by CodeRabbit

Release Notes

• Tests
  • Enhanced accuracy validation for distributed configurations with improved tolerance thresholds to ensure reliable quality checks across different deployment scenarios.

[xxi-nv]

/bot run --disable-fail-fast

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: cddc2950-f863-41f0-8e6c-cf576333ef8d

📥 Commits

Reviewing files that changed from the base of the PR and between 11f6cf7 and 4fa5826.

📒 Files selected for processing (1)

• tests/unittest/_torch/modules/moe/quantize_utils.py

---

📝 Walkthrough

Walkthrough

Updated the check_accuracy method within the test utility to apply conditional tolerance thresholds based on moe_tp_size. The method now computes atol dynamically and routes accuracy checks through utils.check_accuracy with significantly relaxed tolerances (rtol=1e-1, percent=0.96) when moe_tp_size > 1, versus stricter baseline (rtol=1e-7, percent=0.99) otherwise.

Changes

Cohort / File(s)	Summary
Test Assertion Logic tests/unittest/_torch/modules/moe/quantize_utils.py	Updated check_accuracy method to use conditional tolerance thresholds based on moe_tp_size value. Replaced direct torch.testing.assert_close calls with utility-based checks that accommodate higher error accumulation in TP/DTP/TTP setups.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly describes the main change: relaxing test tolerance for W8A16 MoE in DTP mode, which directly matches the core fix in the changeset.
Description check	✅ Passed	The description provides good context including root cause analysis and test results, but lacks explicit sections matching the template structure (missing 'Description' and 'Test Coverage' headers).

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

📝 Coding Plan

• Generate coding plan for human review comments

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Tip

Flake8 can be used to improve the quality of Python code reviews.

Flake8 is a Python linter that wraps PyFlakes, pycodestyle and Ned Batchelder's McCabe script.

To configure Flake8, add a '.flake8' or 'setup.cfg' file to your project root.

See Flake8 Documentation for more details.

[xxi-nv]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #39582 [ run ] triggered by Bot. Commit: 4fa5826 Link to invocation

[tensorrt-cicd]

PR_Github #39582 [ run ] completed with state SUCCESS. Commit: 4fa5826
/LLM/main/L0_MergeRequest_PR pipeline #30794 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[xxi-nv]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #39653 [ run ] triggered by Bot. Commit: 4fa5826 Link to invocation

[tensorrt-cicd]

PR_Github #39653 [ run ] completed with state SUCCESS. Commit: 4fa5826
/LLM/main/L0_MergeRequest_PR pipeline #30857 completed with status: 'SUCCESS'

CI Report

Link to invocation