[https://nvbugs/6396728][fix] Add module-level `pytestmark = pytest.mark.threadleak(enabled=False)` to… by trtllm-agent · Pull Request #15839 · NVIDIA/TensorRT-LLM

trtllm-agent · 2026-07-01T14:24:59Z

Summary

Root cause: Ray-driver daemon threads (ray_listen_error_messages, ray_print_logs, PythonGCThread) survive past pytest-threadleak's per-test snapshot because they only tear down inside RayExecutor.shutdown() when the LLM is finally closed, producing "Test terminated unexpectedly" with no stack trace.
Fix: Add module-level pytestmark = pytest.mark.threadleak(enabled=False) to test_llm_update_weights_multi_gpu.py, matching the pattern used by the parent/sibling files.
Automated fix generated by repair-bot

Test plan

Verify fix on the same GPU type as the original failure
Check for regressions in related tests

Links

Bug: https://nvbugs/6396731

Summary by CodeRabbit

Tests
- Adjusted a multi-GPU test suite to avoid a thread-leak check that can fail during normal teardown timing.
- This makes the affected tests more stable and reduces false failures during cleanup.

coderabbitai · 2026-07-01T14:26:37Z

📝 Walkthrough

Walkthrough

This change adds a module-level pytestmark to disable pytest-threadleak enforcement in a Ray orchestrator multi-GPU test file, accompanied by a comment explaining that Ray-backed LLM teardown occurs after threadleak's snapshot timing.

Changes

Test configuration update

Layer / File(s)	Summary
Disable threadleak enforcement `tests/unittest/_torch/ray_orchestrator/multi_gpu/test_llm_update_weights_multi_gpu.py`	Adds `pytestmark = pytest.mark.threadleak(enabled=False)` with a comment documenting that Ray-backed LLM teardown happens after pytest-threadleak's per-test snapshot.

Estimated code review effort: 1 (Trivial) | ~2 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description check	✅ Passed	The description covers the issue, fix, test plan, and bug link, but it omits the explicit PR checklist confirmation.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly reflects the main change: disabling threadleak checks at module scope for the multi-GPU test.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

…ts multi-GPU test test_llm_update_weights_nvfp4[auto-Qwen3/Qwen3-8B] in this multi-GPU file intermittently fails on DGX_B200-4_GPUs-PyTorch-Ray-1 with only "Test terminated unexpectedly" (empty stdout/stderr/stack_trace in CI DB). Locally, on a 2-B200 node with the CI ray pin (ray[default]==2.54.1), the test passes -- no functional bug. The root cause is the same Ray-driver daemon-thread leak (ray_listen_error_messages, ray_print_logs, PythonGCThread) already addressed for the part0 FP8 variants under nvbugs/6372711 and in the parent single_gpu/test_llm_update_weights.py module: Ray-internal daemon threads spawned by ray.init() in the driver are only torn down inside RayExecutor.shutdown(), which runs when the LLM is finally closed -- after pytest-threadleak's per-test snapshot. tests/unittest/pytest.ini's threadleak_exclude regex does not cover these Ray thread names, so the plugin fails the test with an unrecoverable error that surfaces as the observed empty-traceback termination. Apply the same module-level marker pytestmark = pytest.mark.threadleak(enabled=False) that - tests/unittest/_torch/ray_orchestrator/single_gpu/test_llm_update_weights.py (the parent module this file imports from) already uses, and - tests/unittest/_torch/ray_orchestrator/multi_gpu/test_executor.py uses per-test. Scope is intentionally narrow: only the tests in this file opt out of thread-leak checking; no pytest.ini change. Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>

shuyixiong · 2026-07-08T02:47:39Z

/bot run --stage-list "DGX_B200-4_GPUs-PyTorch-Ray-1"

shuyixiong · 2026-07-08T02:53:59Z

/bot run --stage-list "DGX_B200-4_GPUs-PyTorch-Ray-1"

tensorrt-cicd · 2026-07-08T03:12:46Z

PR_Github #58125 [ run ] triggered by Bot. Commit: 98c769a Link to invocation

Signed-off-by: Shuyi Xiong <219646547+shuyixiong@users.noreply.github.com>

shuyixiong · 2026-07-08T03:32:09Z

/bot run --stage-list "DGX_B200-4_GPUs-PyTorch-Ray-1"

tensorrt-cicd · 2026-07-08T03:39:41Z

PR_Github #58135 [ run ] triggered by Bot. Commit: 2be183b Link to invocation

tensorrt-cicd · 2026-07-08T03:43:15Z

PR_Github #58125 [ run ] completed with state ABORTED. Commit: 98c769a

Link to invocation

Superjomn

LGTM

Signed-off-by: Shuyi Xiong <219646547+shuyixiong@users.noreply.github.com>

shuyixiong · 2026-07-08T04:31:39Z

/bot run --stage-list "DGX_B200-4_GPUs-PyTorch-Ray-1"

tensorrt-cicd · 2026-07-08T04:37:15Z

PR_Github #58145 [ run ] triggered by Bot. Commit: 5945eb8 Link to invocation

tensorrt-cicd · 2026-07-08T04:40:41Z

PR_Github #58135 [ run ] completed with state ABORTED. Commit: 2be183b

Link to invocation

tensorrt-cicd · 2026-07-08T05:14:46Z

PR_Github #58145 [ run ] completed with state FAILURE. Commit: 5945eb8
/LLM/main/L0_MergeRequest_PR pipeline #46799 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

shuyixiong · 2026-07-08T06:38:28Z

/bot run --stage-list "DGX_B200-4_GPUs-PyTorch-Ray-1"

tensorrt-cicd · 2026-07-08T06:45:07Z

PR_Github #58183 [ run ] triggered by Bot. Commit: 5945eb8 Link to invocation

tensorrt-cicd · 2026-07-08T07:25:03Z

PR_Github #58183 [ run ] completed with state FAILURE. Commit: 5945eb8
/LLM/main/L0_MergeRequest_PR pipeline #46831 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

shuyixiong · 2026-07-08T07:27:10Z

/bot run --stage-list "DGX_B200-4_GPUs-PyTorch-Ray-1"

tensorrt-cicd · 2026-07-08T07:33:37Z

PR_Github #58189 [ run ] triggered by Bot. Commit: 5945eb8 Link to invocation

tensorrt-cicd · 2026-07-08T08:56:20Z

PR_Github #58189 [ run ] completed with state SUCCESS. Commit: 5945eb8
/LLM/main/L0_MergeRequest_PR pipeline #46834 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

shuyixiong · 2026-07-08T08:58:27Z

/bot skip --comment "Passed the related CI stage "

tensorrt-cicd · 2026-07-08T09:03:40Z

PR_Github #58207 [ skip ] triggered by Bot. Commit: 5945eb8 Link to invocation

tensorrt-cicd · 2026-07-08T09:10:39Z

PR_Github #58207 [ skip ] completed with state SUCCESS. Commit: 5945eb8
Skipping testing for commit 5945eb8

Link to invocation

…ark.threadleak(enabled=False)` to… (NVIDIA#15839) Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com> Signed-off-by: Shuyi Xiong <219646547+shuyixiong@users.noreply.github.com> Co-authored-by: Shuyi Xiong <219646547+shuyixiong@users.noreply.github.com>

trtllm-agent assigned shuyixiong Jul 1, 2026

github-actions Bot assigned trtllm-agent Jul 1, 2026

shuyixiong force-pushed the repair-bot-bug6396731 branch from 8657689 to 98c769a Compare July 8, 2026 02:45

Unwaive test

2be183b

Signed-off-by: Shuyi Xiong <219646547+shuyixiong@users.noreply.github.com>

shuyixiong changed the title ~~[https://nvbugs/6396731][fix] Add module-level pytestmark = pytest.mark.threadleak(enabled=False) to…~~ [https://nvbugs/6396728][fix] Add module-level pytestmark = pytest.mark.threadleak(enabled=False) to… Jul 8, 2026

shuyixiong requested a review from Superjomn July 8, 2026 03:50

Superjomn approved these changes Jul 8, 2026

View reviewed changes

Use mUse stricter resource cleanup

5945eb8

Signed-off-by: Shuyi Xiong <219646547+shuyixiong@users.noreply.github.com>

shuyixiong enabled auto-merge (squash) July 8, 2026 08:58

shuyixiong merged commit e7b2256 into NVIDIA:main Jul 8, 2026
8 checks passed

Uh oh!

Conversation

trtllm-agent commented Jul 1, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Links

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

shuyixiong commented Jul 8, 2026

Uh oh!

shuyixiong commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

shuyixiong commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

Superjomn left a comment

Choose a reason for hiding this comment

Uh oh!

shuyixiong commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

shuyixiong commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

shuyixiong commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

shuyixiong commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

trtllm-agent commented Jul 1, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 1, 2026 •

edited

Loading