[https://nvbugs/6236818][ci] Re-enable Verl post-merge CI stage by Superjomn · Pull Request #15935 · NVIDIA/TensorRT-LLM

Superjomn · 2026-07-04T14:38:06Z

Summary

Re-enables the DGX_B200-4_GPUs-Verl-Post-Merge-1 post-merge CI stage that was previously
disabled due to https://nvbugs/6236818. The bug is now resolved.

Change: Remove the three disabling comment lines in jenkins/L0_Test.groovy (around line 4515)
and restore the active DGX_B200-4_GPUs-Verl-Post-Merge-1 stage entry.

Validation

Local run: 18/18 runnable Verl tests pass on the target hardware.
test_trtllm_abort remains separately waived under https://nvbugs/6272653 (unrelated issue).

CI

The re-enabled stage runs automatically in post-merge CI, or can be triggered manually via:

/bot run --extra-stage "DGX_B200-4_GPUs-Verl-Post-Merge-1"

Summary by CodeRabbit

Tests
- Re-enabled a previously disabled GPU Slurm test stage in the CI pipeline, restoring coverage for that configuration.

Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>

coderabbitai · 2026-07-04T14:40:29Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: a88ed37a-885a-4013-a45c-2546020b5e65

📥 Commits

Reviewing files that changed from the base of the PR and between e4aba85 and 37427ae.

📒 Files selected for processing (1)

jenkins/L0_Test.groovy

📝 Walkthrough

Walkthrough

This change re-enables a previously disabled Slurm test stage, DGX_B200-4_GPUs-Verl-Post-Merge-1, in the x86SlurmTestConfigs list within jenkins/L0_Test.groovy, replacing a commented-out configuration with an active l0_verl test mapping.

Changes

Jenkins Test Configuration

Layer / File(s)	Summary
Re-enable Verl Slurm test stage `jenkins/L0_Test.groovy`	The `DGX_B200-4_GPUs-Verl-Post-Merge-1` entry mapped to `l0_verl` is restored as an active entry in `x86SlurmTestConfigs`, replacing the previously commented-out block.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Compact metadata

Related Issues: None provided
Related PRs: None provided
Suggested labels: None
Suggested reviewers: None

Poem

A rabbit hops through Jenkins' halls,
Uncomments lines where testing calls,
Verl awakes on DGX's might,
Post-merge checks now back in sight,
One small line, restored so bright. 🐰

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title matches the change: it re-enables the Verl post-merge CI stage and includes the relevant NVBugs ticket.
Description check	✅ Passed	The description covers the summary, validation, and CI trigger details, though the checklist section from the template is omitted.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

Superjomn · 2026-07-05T02:03:13Z

/bot run --stage-list "DGX_B200-4_GPUs-Verl-Post-Merge-1"

tensorrt-cicd · 2026-07-05T02:09:20Z

PR_Github #57587 [ run ] triggered by Bot. Commit: 37427ae Link to invocation

tensorrt-cicd · 2026-07-05T03:01:17Z

PR_Github #57587 [ run ] completed with state SUCCESS. Commit: 37427ae
/LLM/main/L0_MergeRequest_PR pipeline #46313 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>

Superjomn · 2026-07-06T07:47:30Z

/bot run

tensorrt-cicd · 2026-07-06T07:53:47Z

PR_Github #57727 [ run ] triggered by Bot. Commit: 77f9140 Link to invocation

Superjomn · 2026-07-06T08:52:00Z

/bot run --stage-list "DGX_B200-4_GPUs-Verl-Post-Merge-1"

tensorrt-cicd · 2026-07-06T08:57:23Z

PR_Github #57740 [ run ] triggered by Bot. Commit: 77f9140 Link to invocation

tensorrt-cicd · 2026-07-06T09:02:02Z

PR_Github #57727 [ run ] completed with state ABORTED. Commit: 77f9140

Link to invocation

tensorrt-cicd · 2026-07-06T09:40:49Z

PR_Github #57740 [ run ] completed with state SUCCESS. Commit: 77f9140
/LLM/main/L0_MergeRequest_PR pipeline #46447 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

…discovery Use nvidia.nvshmem.__path__ instead of __file__ to resolve the pip-installed NVSHMEM package dir. As a namespace package, __file__ is None and os.path.dirname(None) raised TypeError, crashing the shared Verl setup step. Signed-off-by: Chunwei Yan <yanchunwei@outlook.com>

Superjomn · 2026-07-07T01:33:45Z

/bot run --extra-stage "DGX_B200-4_GPUs-Verl-Post-Merge-1"

tensorrt-cicd · 2026-07-07T01:40:03Z

PR_Github #57879 [ run ] triggered by Bot. Commit: 9091923 Link to invocation

tensorrt-cicd · 2026-07-07T05:05:52Z

PR_Github #57879 [ run ] completed with state FAILURE. Commit: 9091923
/LLM/main/L0_MergeRequest_PR pipeline #46575 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Superjomn · 2026-07-07T05:19:10Z

/bot run --disable-fail-fast --extra-stage "DGX_B200-4_GPUs-Verl-Post-Merge-1"

tensorrt-cicd · 2026-07-07T05:24:41Z

PR_Github #57943 [ run ] triggered by Bot. Commit: 9091923 Link to invocation

tensorrt-cicd · 2026-07-07T09:44:47Z

PR_Github #57943 [ run ] completed with state FAILURE. Commit: 9091923
/LLM/main/L0_MergeRequest_PR pipeline #46626 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

…pEP build Drop the explicit nvidia-nvshmem-cu13==3.3.20 install, which shadowed the NVSHMEM version PyTorch was built against and broke torch import during the DeepEP build (libtorch_nvshmem.so: undefined symbol nvshmem_selected_device_transport). Resolve NVSHMEM_DIR to the nvshmem package torch already depends on so DeepEP links against the matching version. Signed-off-by: Chunwei Yan <yanchunwei@outlook.com>

Superjomn · 2026-07-07T12:43:36Z

/bot run --disable-fail-fast --extra-stage "DGX_B200-4_GPUs-Verl-Post-Merge-1"

tensorrt-cicd · 2026-07-07T12:49:12Z

PR_Github #58002 [ run ] triggered by Bot. Commit: e4bed4a Link to invocation

…rl DeepEP build The previous approach resolved NVSHMEM_DIR via `import nvidia.nvshmem`, but that package does not exist in the NGC PyTorch 26.04 base image, so the Verl session-setup fixture aborted and all Verl tests errored at setup. DeepEP's deep_ep_cpp extension hard-links the static device archive (-l:libnvshmem_device.a) and host lib (-l:libnvshmem_host.so); the base image ships neither (only libnvshmem_host.so.3 plus a bitcode device lib). Install the nvidia-nvshmem-cu13 wheel pinned to 3.6.5 -- the exact NVSHMEM version this container's torch was built against -- so libtorch_nvshmem.so's versioned symbols (e.g. nvshmem_selected_device_transport@NVSHMEM) still resolve and `import torch` is not broken by shadowing. Verified locally: import torch succeeds with 3.6.5 on LD_LIBRARY_PATH and DeepEP links sm_100/sm_120/sm_90. Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>

Superjomn · 2026-07-07T13:31:34Z

/bot run --extra-stage "DGX_B200-4_GPUs-Verl-Post-Merge-1"

tensorrt-cicd · 2026-07-07T13:37:27Z

PR_Github #58009 [ run ] triggered by Bot. Commit: 59e165b Link to invocation

tensorrt-cicd · 2026-07-07T13:41:54Z

PR_Github #58002 [ run ] completed with state ABORTED. Commit: e4bed4a

Link to invocation

tensorrt-cicd · 2026-07-07T19:08:00Z

PR_Github #58009 [ run ] completed with state SUCCESS. Commit: 59e165b
/LLM/main/L0_MergeRequest_PR pipeline #46679 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

CI Report

Link to invocation

…IA#15935) Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com> Signed-off-by: Chunwei Yan <yanchunwei@outlook.com>

[https://nvbugs/6236818][ci] Re-enable Verl post-merge CI stage

37427ae

Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>

Superjomn requested review from a team as code owners July 4, 2026 14:38

Superjomn requested review from niukuo and yiqingy0 July 4, 2026 14:38

github-actions Bot assigned Superjomn Jul 4, 2026

[https://nvbugs/6236818][ci] Fix Verl DeepEP NVSHMEM device-lib linking

77f9140

Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>

niukuo approved these changes Jul 8, 2026

View reviewed changes

yiqingy0 approved these changes Jul 8, 2026

View reviewed changes

Superjomn merged commit 126461f into NVIDIA:main Jul 8, 2026
7 checks passed

Superjomn deleted the user/chunweiy/reenable-verl-ci branch July 8, 2026 03:18

Uh oh!

Conversation

Superjomn commented Jul 4, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

CI

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jul 4, 2026

Walkthrough

Changes

Compact metadata

Uh oh!

Superjomn commented Jul 5, 2026

Uh oh!

tensorrt-cicd commented Jul 5, 2026

Uh oh!

tensorrt-cicd commented Jul 5, 2026

Uh oh!

Superjomn commented Jul 6, 2026

Uh oh!

tensorrt-cicd commented Jul 6, 2026

Uh oh!

Superjomn commented Jul 6, 2026

Uh oh!

tensorrt-cicd commented Jul 6, 2026

Uh oh!

tensorrt-cicd commented Jul 6, 2026

Uh oh!

tensorrt-cicd commented Jul 6, 2026

Uh oh!

Superjomn commented Jul 7, 2026

Uh oh!

tensorrt-cicd commented Jul 7, 2026

Uh oh!

tensorrt-cicd commented Jul 7, 2026

Uh oh!

Superjomn commented Jul 7, 2026

Uh oh!

tensorrt-cicd commented Jul 7, 2026

Uh oh!

tensorrt-cicd commented Jul 7, 2026

Uh oh!

Superjomn commented Jul 7, 2026

Uh oh!

tensorrt-cicd commented Jul 7, 2026

Uh oh!

Superjomn commented Jul 7, 2026

Uh oh!

tensorrt-cicd commented Jul 7, 2026

Uh oh!

tensorrt-cicd commented Jul 7, 2026

Uh oh!

tensorrt-cicd commented Jul 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Superjomn commented Jul 4, 2026 •

edited by coderabbitai Bot

Loading