[None][infra] Log infra-retry classify declines instead of silent rethrow by brnguyen2 · Pull Request #17432 · NVIDIA/TensorRT-LLM

brnguyen2 · 2026-08-07T22:30:13Z

Description

When a stage failure or a pod/agent launch failure does not match any pattern in FailureClassifier's catalog, runKubernetesPodWithInfraRetry (both the launch loop and the execution loop) and the SLURM retry loop classify it as a user failure and rethrow with no log output. If an infra event's exception type is missing from the catalog (for example, a pod-scheduling timeout such as KubernetesClientTimeoutException), the failure leaves no trace: there is no [INFRA-RETRY] line to grep for, and from the console it is impossible to distinguish "the classifier saw this and declined to retry" from "the retry wrapper never ran".

This came out of triaging a pre-merge pipeline that was lost to an agent pod that never launched; establishing that the retry wrapper had classified the failure as a user failure (rather than never running) required manually reproducing the classification logic against the raw console. One echo line per decline site makes that determination a grep.

Changes

Adds one echo before each of the three silent UserFailure rethrow sites in jenkins/L0_Test.groovy, stating the stage name, that no infra pattern matched, and the exception text:

the K8s pod/agent launch loop (singleAttempt branch of runKubernetesPodWithInfraRetry)
the K8s execution retry loop
the SLURM retry loop

No behavior change; logging only. A companion internal shared-library change adds the missing catalog pattern for the pod-scheduling timeout case.

Test Coverage

Logging-only change in pipeline Groovy; no test applicable. Verified the echo style and string interpolation match the surrounding [INFRA-RETRY] echoes in the same functions.

Dev Engineer Review

Added [INFRA-RETRY] logging for unmatched Kubernetes pod launch, Kubernetes execution, and SLURM retry classifications.
Logs include the stage name and exception text before the failure is rethrown.
Retry behavior remains unchanged.
Added a waiver for unittest/llmapi/test_llm_pytorch.py -m "part3", pending resolution in PR #17432.
The waiver format and test path are valid. No duplicate entry was identified.
A companion shared-library catalog update adds a pattern for pod-scheduling timeout exceptions.

QA Engineer Review

Modified tests/integration/test_lists/waives.txt.
Added one waiver entry for unittest/llmapi/test_llm_pytorch.py -m "part3".
No test-db or qa files were modified.
CBTS coverage data is unavailable. Verdict: needs follow-up.

File a tracking bug for the CI-waived test(s) and record it with pr-babysitter waive set-bug NVIDIA/TensorRT-LLM#17432 <bug-url> — the waives entry carries a placeholder until then

…hrow When a stage or pod/agent launch failure does not match any pattern in FailureClassifier's catalog, runKubernetesPodWithInfraRetry and the SLURM retry loop classify it as a user failure and rethrow with no log output. A misclassified infra event (for example a pod-scheduling timeout whose exception type is missing from the catalog) then leaves no trace: no [INFRA-RETRY] line to grep for, and no way to tell a deliberate retry-decline from the retry wrapper never running. Add one echo before each of the three silent rethrow sites (K8s launch loop, K8s execution loop, SLURM loop) stating the stage, the decline reason, and the exception text. Signed-off-by: Brian Nguyen <brnguyen@nvidia.com>

coderabbitai · 2026-08-08T12:12:02Z

Walkthrough

SLURM and Kubernetes retry loops now log unmatched failure classifications and exception details before rethrowing non-infrastructure failures. The integration test waiver list now skips the pending part3 PyTorch LLM API test subset.

Changes

Retry failure diagnostics

Layer / File(s)	Summary
Non-infrastructure failure logging `jenkins/L0_Test.groovy`	SLURM and Kubernetes failure paths now log stage, classification, and exception details before rethrowing or terminating without retry. Infrastructure retry behavior remains unchanged.
Integration test waiver `tests/integration/test_lists/waives.txt`	The `part3` subset of `unittest/llmapi/test_llm_pytorch.py` is skipped pending resolution of the tracked issue.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Possibly related PRs

NVIDIA/TensorRT-LLM#17384: Updates the same waiver list with a different test waiver.
NVIDIA/TensorRT-LLM#17385: Updates the same waiver list with a different test waiver.
NVIDIA/TensorRT-LLM#17402: Updates the same waiver list with test waiver entries.

Suggested reviewers: bowenfu, mzweilz

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly identifies the infrastructure retry logging change and follows the repository's required ticket and type format.
Description check	✅ Passed	The description explains the issue, lists the three logging changes, and states test coverage; the repository checklist is not fully completed.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

brnguyen2 · 2026-08-08T12:43:15Z

/bot run

tensorrt-cicd · 2026-08-08T12:49:20Z

PR_Github #64783 [ run ] triggered by Bot. Commit: 5eef699 Link to invocation

tensorrt-cicd · 2026-08-08T13:37:53Z

PR_Github #64783 [ run ] completed with state SUCCESS. Commit: 5eef699
/LLM/main/L0_MergeRequest_PR pipeline #52627 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

brnguyen2 · 2026-08-08T13:50:47Z

/bot run

tensorrt-cicd · 2026-08-08T13:56:38Z

PR_Github #64787 [ run ] triggered by Bot. Commit: 5eef699 Link to invocation

tensorrt-cicd · 2026-08-08T16:21:35Z

PR_Github #64787 [ run ] completed with state SUCCESS. Commit: 5eef699
/LLM/main/L0_MergeRequest_PR pipeline #52631 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

…2[unittest/llmapi/test_llm_pytorch.py -m "part3"] (pre-existing failure, tracking bug pending) Signed-off-by: Brian Nguyen <brnguyen@nvidia.com>

brnguyen2 · 2026-08-09T16:45:56Z

/bot run

tensorrt-cicd · 2026-08-09T16:51:53Z

PR_Github #64862 [ run ] triggered by Bot. Commit: 3b2ff58 Link to invocation

tensorrt-cicd · 2026-08-09T17:26:59Z

PR_Github #64862 [ run ] completed with state SUCCESS. Commit: 3b2ff58
/LLM/main/L0_MergeRequest_PR pipeline #52703 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: Brian Nguyen <brnguyen@nvidia.com>

brnguyen2 · 2026-08-10T02:58:21Z

/bot run

tensorrt-cicd · 2026-08-10T03:06:16Z

PR_Github #64930 [ run ] triggered by Bot. Commit: e283397 Link to invocation

tensorrt-cicd · 2026-08-10T08:14:39Z

PR_Github #64930 [ run ] completed with state FAILURE. Commit: e283397
/LLM/main/L0_MergeRequest_PR pipeline #52771 completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Ask a member of NVIDIA/trt-llm-ci-approvers to add the label, then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

github-actions Bot assigned brnguyen2 Aug 7, 2026

Merge branch 'main' into infra/log-classify-decline

5eef699

brnguyen2 marked this pull request as ready for review August 8, 2026 12:10

brnguyen2 requested a review from a team as a code owner August 8, 2026 12:10

brnguyen2 requested review from dpitman-nvda and mzweilz August 8, 2026 12:10

test waives: skip A100X-PyTorch-1/test_unittests.py::test_unittests_v…

3b2ff58

…2[unittest/llmapi/test_llm_pytorch.py -m "part3"] (pre-existing failure, tracking bug pending) Signed-off-by: Brian Nguyen <brnguyen@nvidia.com>

brnguyen2 requested review from a team as code owners August 9, 2026 16:45

brnguyen2 requested review from BowenFu, QiJune, crazydemo and xinhe-nv August 9, 2026 16:45

BowenFu reviewed Aug 10, 2026

View reviewed changes

Comment thread tests/integration/test_lists/waives.txt Outdated

xinhe-nv approved these changes Aug 10, 2026

View reviewed changes

test waives: revert the part3 llm_pytorch waive (unblock full run)

e283397

Signed-off-by: Brian Nguyen <brnguyen@nvidia.com>

BowenFu approved these changes Aug 10, 2026

View reviewed changes

Uh oh!

Conversation

brnguyen2 commented Aug 7, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes

Test Coverage

Dev Engineer Review

QA Engineer Review

Uh oh!

coderabbitai Bot commented Aug 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

brnguyen2 commented Aug 8, 2026

Uh oh!

tensorrt-cicd commented Aug 8, 2026

Uh oh!

tensorrt-cicd commented Aug 8, 2026

Uh oh!

brnguyen2 commented Aug 8, 2026

Uh oh!

tensorrt-cicd commented Aug 8, 2026

Uh oh!

tensorrt-cicd commented Aug 8, 2026

Uh oh!

brnguyen2 commented Aug 9, 2026

Uh oh!

tensorrt-cicd commented Aug 9, 2026

Uh oh!

tensorrt-cicd commented Aug 9, 2026

Uh oh!

Uh oh!

brnguyen2 commented Aug 10, 2026

Uh oh!

tensorrt-cicd commented Aug 10, 2026

Uh oh!

tensorrt-cicd commented Aug 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

brnguyen2 commented Aug 7, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Aug 8, 2026 •

edited

Loading