[https://nvbugs/6418090][fix] Add @torch.inference_mode() decorator to… by trtllm-agent · Pull Request #16000 · NVIDIA/TensorRT-LLM

trtllm-agent · 2026-07-06T20:40:14Z

Summary

Root cause: add_dummy_requests writes prev_num_accepted_tokens/cache_buf_idx (inference tensors) but the method isn't wrapped in torch.inference_mode(), so ADP-dummy/warmup callsites hit "Inplace update to inference tensor outside InferenceMode".
Fix: Add @torch.inference_mode() decorator to PythonMambaCacheManager.add_dummy_requests and remove the two nvbugs/6418090 waivers.
Automated fix generated by repair-bot

Test plan

Verify fix on the same GPU type as the original failure
Check for regressions in related tests

Links

Bug: https://nvbugs/6418090

Summary by CodeRabbit

Bug Fixes
- Improved dummy request handling to run in inference mode, which can reduce unnecessary overhead and improve runtime behavior.
Tests
- Updated integration test waivers to reflect current expected skips across API, disaggregated serving, performance, and platform-specific test coverage.

hnover-nv · 2026-07-06T20:54:19Z

/bot run --extra-stage "DGX_B200-4_GPUs-AutoDeploy-Post-Merge-1, DGX_B200-8_GPUs-AutoDeploy-Post-Merge-1"

tensorrt-cicd · 2026-07-06T21:00:42Z

PR_Github #57823 [ run ] triggered by Bot. Commit: 32ec7dd Link to invocation

tensorrt-cicd · 2026-07-07T00:21:19Z

PR_Github #57823 [ run ] completed with state SUCCESS. Commit: 32ec7dd
/LLM/main/L0_MergeRequest_PR pipeline #46520 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

hnover-nv · 2026-07-07T00:57:13Z

/bot run --extra-stage "DGX_B200-4_GPUs-AutoDeploy-Post-Merge-1, DGX_B200-8_GPUs-AutoDeploy-Post-Merge-1"

tensorrt-cicd · 2026-07-07T01:03:27Z

PR_Github #57870 [ run ] triggered by Bot. Commit: 32ec7dd Link to invocation

tensorrt-cicd · 2026-07-07T03:37:23Z

PR_Github #57870 [ run ] completed with state SUCCESS. Commit: 32ec7dd
/LLM/main/L0_MergeRequest_PR pipeline #46566 completed with status: 'SUCCESS'

CI Report

Link to invocation

coderabbitai · 2026-07-07T08:37:59Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: babf011c-01e3-49a9-b948-529ba05499b9

📥 Commits

Reviewing files that changed from the base of the PR and between 0b0fd54 and d21b6d8.

📒 Files selected for processing (2)

tensorrt_llm/_torch/pyexecutor/mamba_cache_manager.py
tests/integration/test_lists/waives.txt

📝 Walkthrough

Walkthrough

This PR adds a torch.inference_mode() decorator to PythonMambaCacheManager.add_dummy_requests, and updates the integration test waiver list by adding, removing, and replacing SKIP entries across accuracy, disaggregated, examples, GB300, H20, and perf sanity test groups.

Changes

Inference mode decorator

Layer / File(s)	Summary
Inference mode for dummy request allocation `tensorrt_llm/_torch/pyexecutor/mamba_cache_manager.py`	Adds `@torch.inference_mode()` decorator to `PythonMambaCacheManager.add_dummy_requests`.

Test waiver list updates

Layer / File(s)	Summary
Accuracy test waivers `tests/integration/test_lists/waives.txt`	Replaces accuracy/* waivers with new test_llm_api/test_llm_api_autodeploy entries; removes specific DeepSeekV3Lite and Llama3_1_8BInstruct SKIP entries.
Disaggregated, examples, and unit test waivers `tests/integration/test_lists/waives.txt`	Adds a disaggregated test_workers_kv_cache_events waiver; removes a visual_gen SKIP and an A30 cpp unit test SKIP.
GB300 and H20 platform waivers `tests/integration/test_lists/waives.txt`	Replaces GB300 waiver block with new disaggregated serving/LLM API entries; replaces H20 waiver block with GPTOSS and disaggregated DeepSeek scenarios.
Perf sanity upload waivers `tests/integration/test_lists/waives.txt`	Replaces perf_sanity aggr_upload/disagg_upload waivers with new GB200/GB300 model coverage.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

NVIDIA/TensorRT-LLM#15914: Both PRs modify tests/integration/test_lists/waives.txt by adding/adjusting integration test SKIP/waiver entries.
NVIDIA/TensorRT-LLM#16030: Both PRs update H20-related skip entries in tests/integration/test_lists/waives.txt including test_llm_api_*/autodeploy/disaggregated serving cases.
NVIDIA/TensorRT-LLM#15967: Both PRs update perf/test_perf_sanity.py::test_e2e aggr_upload/disagg_upload SKIP waivers.

Suggested reviewers: dongxuy04, jieli-matrix, pcastonguay

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is specific and matches the main change: adding inference mode to the Mamba cache manager fix.
Description check	✅ Passed	The description covers the bug, fix, test plan, and link; it is mostly complete despite not following the template exactly.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

… in torch.inference_mode() The two writes to prev_num_accepted_tokens/cache_buf_idx added by the MTP-replay checkpointing feature run outside inference_mode when add_dummy_requests is invoked from ADP dummy padding / warmup paths, triggering 'Inplace update to inference tensor outside InferenceMode'. Decorate the method to match the sibling _prepare_mamba_cache_blocks pattern and remove the two now-obsolete waivers. Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>

hnover-nv · 2026-07-07T17:25:01Z

/bot run

tensorrt-cicd · 2026-07-07T17:31:43Z

PR_Github #58047 [ run ] triggered by Bot. Commit: 61ef6be Link to invocation

brb-nv

LGTM.

tensorrt-cicd · 2026-07-07T21:57:16Z

PR_Github #58047 [ run ] completed with state SUCCESS. Commit: 61ef6be
/LLM/main/L0_MergeRequest_PR pipeline #46715 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

hnover-nv · 2026-07-07T22:26:14Z

/bot run

tensorrt-cicd · 2026-07-07T22:32:40Z

PR_Github #58090 [ run ] triggered by Bot. Commit: 61ef6be Link to invocation

tensorrt-cicd · 2026-07-07T22:47:29Z

PR_Github #58090 [ run ] completed with state FAILURE. Commit: 61ef6be
/LLM/main/L0_MergeRequest_PR pipeline #46754 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

hnover-nv · 2026-07-07T23:54:45Z

/bot run

tensorrt-cicd · 2026-07-08T00:00:31Z

PR_Github #58096 [ run ] triggered by Bot. Commit: 61ef6be Link to invocation

tensorrt-cicd · 2026-07-08T01:02:58Z

PR_Github #58096 [ run ] completed with state SUCCESS. Commit: 61ef6be
/LLM/main/L0_MergeRequest_PR pipeline #46759 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

hnover-nv · 2026-07-08T03:10:34Z

/bot run

tensorrt-cicd · 2026-07-08T03:17:44Z

PR_Github #58129 [ run ] triggered by Bot. Commit: 911b6bb Link to invocation

tensorrt-cicd · 2026-07-08T05:18:20Z

PR_Github #58129 [ run ] completed with state SUCCESS. Commit: 911b6bb
/LLM/main/L0_MergeRequest_PR pipeline #46789 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

hnover-nv · 2026-07-08T15:13:23Z

/bot run

Signed-off-by: Harris Nover <249353502+hnover-nv@users.noreply.github.com>

Restore the unwaive that merging dropped. Added unwaiving the test from nvbugs/6422315 as I have to rerun CI anyway. Signed-off-by: Harris Nover <249353502+hnover-nv@users.noreply.github.com>

hnover-nv · 2026-07-08T16:23:48Z

/bot run

tensorrt-cicd · 2026-07-08T16:30:24Z

PR_Github #58264 [ run ] triggered by Bot. Commit: 6e8ca2b Link to invocation

tensorrt-cicd · 2026-07-08T16:46:01Z

PR_Github #58264 [ run ] completed with state FAILURE. Commit: 6e8ca2b
/LLM/main/L0_MergeRequest_PR pipeline #46903 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

hnover-nv · 2026-07-08T19:24:30Z

/bot run

tensorrt-cicd · 2026-07-08T19:31:02Z

PR_Github #58292 [ run ] triggered by Bot. Commit: 40420db Link to invocation

tensorrt-cicd · 2026-07-08T19:44:56Z

PR_Github #58292 [ run ] completed with state FAILURE. Commit: 40420db
/LLM/main/L0_MergeRequest_PR pipeline #46927 completed with status: 'ABORTED'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

hnover-nv · 2026-07-08T20:51:03Z

/bot run

tensorrt-cicd · 2026-07-08T20:57:07Z

PR_Github #58299 [ run ] triggered by Bot. Commit: 40420db Link to invocation

hnover-nv · 2026-07-09T16:15:59Z

/bot run

tensorrt-cicd · 2026-07-09T16:22:10Z

PR_Github #58490 [ run ] triggered by Bot. Commit: 40420db Link to invocation

tensorrt-cicd · 2026-07-09T16:22:17Z

PR_Github #58299 [ run ] completed with state ABORTED. Commit: 40420db
/LLM/main/L0_MergeRequest_PR pipeline #46934 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: Eran Geva <19514940+MrGeva@users.noreply.github.com>

MrGeva · 2026-07-09T17:00:05Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-09T17:06:30Z

PR_Github #58499 [ run ] triggered by Bot. Commit: dbb9e13 Link to invocation

tensorrt-cicd · 2026-07-09T17:11:18Z

PR_Github #58490 [ run ] completed with state ABORTED. Commit: 40420db

Link to invocation

trtllm-agent requested review from a team as code owners July 6, 2026 20:40

trtllm-agent requested a review from shaharmor98 July 6, 2026 20:40

trtllm-agent assigned hnover-nv Jul 6, 2026

trtllm-agent requested a review from Wanli-Jiang July 6, 2026 20:40

github-actions Bot assigned trtllm-agent Jul 6, 2026

hnover-nv enabled auto-merge (squash) July 7, 2026 03:53

auto-merge was automatically disabled July 7, 2026 08:34
Head branch was pushed to by a user without write access

trtllm-agent force-pushed the repair-bot-bug6418090 branch from 32ec7dd to d21b6d8 Compare July 7, 2026 08:34

MrGeva approved these changes Jul 7, 2026

View reviewed changes

hnover-nv force-pushed the repair-bot-bug6418090 branch from ec2a8b8 to 61ef6be Compare July 7, 2026 17:21

hnover-nv enabled auto-merge (squash) July 7, 2026 17:24

brb-nv approved these changes Jul 7, 2026

View reviewed changes

juney-nvidia approved these changes Jul 7, 2026

View reviewed changes

Wanli-Jiang approved these changes Jul 8, 2026

View reviewed changes

Merge branch 'main' into repair-bot-bug6418090

911b6bb

hnover-nv added 2 commits July 8, 2026 09:18

Merge branch 'main' into repair-bot-bug6418090

ae9f316

Signed-off-by: Harris Nover <249353502+hnover-nv@users.noreply.github.com>

Remove skipped tests for NemotronSuperV3

6e8ca2b

Restore the unwaive that merging dropped. Added unwaiving the test from nvbugs/6422315 as I have to rerun CI anyway. Signed-off-by: Harris Nover <249353502+hnover-nv@users.noreply.github.com>

Merge branch 'main' into repair-bot-bug6418090

40420db

galagam reviewed Jul 9, 2026

View reviewed changes

Comment thread tests/integration/test_lists/waives.txt Outdated

Update waives.txt

dbb9e13

Signed-off-by: Eran Geva <19514940+MrGeva@users.noreply.github.com>

Uh oh!

Conversation

trtllm-agent commented Jul 6, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Links

Summary by CodeRabbit

Uh oh!

hnover-nv commented Jul 6, 2026

Uh oh!

tensorrt-cicd commented Jul 6, 2026

Uh oh!

tensorrt-cicd commented Jul 7, 2026

Uh oh!

hnover-nv commented Jul 7, 2026

Uh oh!

tensorrt-cicd commented Jul 7, 2026

Uh oh!

tensorrt-cicd commented Jul 7, 2026

Uh oh!

coderabbitai Bot commented Jul 7, 2026

Walkthrough

Changes

Uh oh!

hnover-nv commented Jul 7, 2026

Uh oh!

tensorrt-cicd commented Jul 7, 2026

Uh oh!

brb-nv left a comment

Choose a reason for hiding this comment

Uh oh!

tensorrt-cicd commented Jul 7, 2026

Uh oh!

hnover-nv commented Jul 7, 2026

Uh oh!

tensorrt-cicd commented Jul 7, 2026

Uh oh!

tensorrt-cicd commented Jul 7, 2026

Uh oh!

hnover-nv commented Jul 7, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

hnover-nv commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

hnover-nv commented Jul 8, 2026

Uh oh!

hnover-nv commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

hnover-nv commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

hnover-nv commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

Uh oh!

hnover-nv commented Jul 9, 2026

Uh oh!

tensorrt-cicd commented Jul 9, 2026

Uh oh!

tensorrt-cicd commented Jul 9, 2026

Uh oh!

MrGeva commented Jul 9, 2026

Uh oh!

tensorrt-cicd commented Jul 9, 2026

trtllm-agent commented Jul 6, 2026 •

edited by coderabbitai Bot

Loading