[https://nvbugs/6428057][fix] Propagate host stop criteria across PP ranks

[chienchunhung]

Description

PR #15920 (4f1c3cd) added a host-side stop-criteria fast path for greedy TorchSampler requests. On the last pipeline-parallel rank, that path sets SampleStateTorch.use_host_stop_criteria=True and deliberately leaves device finish_reasons unset. The PP sample-state ring only forwarded sample_state.host and result diffs, so receiving ranks kept the default False value and entered the device-finish-reason update path with missing state.

This mismatch caused the PP worker failures reported after post-merge 2826. The failures landed in the same regression window as #15386, but the responsible change is #15920; this PR does not modify the staged post-load or MoE lifecycle.

The four NVBug records map to three unique waived tests:

• NVBug 6428057: DeepSeek V3 Lite BF16 PP4
• NVBug 6428085 and NVBug 6428087: the same DeepSeek V3 Lite NVFP4 CUTEDSL TP2/PP2 failure
• NVBug 6428124: Nemotron-H LoRA PP2

Changes

• Forward use_host_stop_criteria with the PP sample-state ring payload.
• Restore the flag only for sampler states that declare it; other sampler flavors forward False without being mutated.
• Add CPU regression coverage for source, relay, and terminal PP ranks.
• Add coverage for sampler states without use_host_stop_criteria.
• Remove the three unique test waivers associated with the four NVBug records.

Test Coverage

Local:

• TensorRT-LLM pre-commit and commit-time DCO hooks passed for all changed files.
• The focused pytest module could not load locally because the environment lacks the full TensorRT-LLM Python dependency set (transformers).

Targeted CI:

/bot run --disable-fail-fast --extra-stage "DGX_B200-4_GPUs-PyTorch-Post-Merge-1, DGX_B200-4_GPUs-PyTorch-Post-Merge-2, GB300-4_GPUs-PyTorch-Post-Merge-3" --detailed-log

PR_Github #58288 was launched on the initial fix commit dd211ea. These stages execute the exact unwaived DeepSeek TP2/PP2, DeepSeek PP4, and Nemotron-H LoRA PP2 regressions. The follow-up commit adds only the non-Torch compatibility guard and its CPU regression test; no additional GPU run has been requested yet.

Impact

The payload is internal to a homogeneous PP worker group. Torch sampler ranks preserve the host-stop decision, while non-Torch sampler states retain their existing behavior.

Related PRs

• [None][perf] Move greedy stop checks to host #15920 introduced the host stop-criteria fast path.
• [None][fix] Propagate host stop criteria across PP ranks #16096 is an earlier alternative that moves the flag into the host state; it predates the [TRTLLM-13212][refactor] Centralize sampling logic, split backends into isolated modules #15542 sampler package split and currently requires a rebase onto main.
• [https://nvbugs/6428057][fix] In _ring_broadcast_sample_state, forward use_host_stop_criteria alongside… #16130 and [https://nvbugs/6428124][fix] Include use_host_stop_criteria in the PP send/recv payload in… #16134 are repair-bot subsets for individual NVBugs. This PR consolidates the compatibility guard, regression coverage, and all three unique waiver removals.
• [TRTLLM-13248][feat] Wave 3: migrate MoE staged hooks #15386 was initially suspected because it shared the regression window, but it does not touch this sampler state-transfer contract.

[chienchunhung]

/bot run --disable-fail-fast --extra-stage "DGX_B200-4_GPUs-PyTorch-Post-Merge-1, DGX_B200-4_GPUs-PyTorch-Post-Merge-2, GB300-4_GPUs-PyTorch-Post-Merge-3" --detailed-log

[tensorrt-cicd]

PR_Github #58288 [ run ] triggered by Bot. Commit: dd211ea Link to invocation

[chienchunhung]

Closing this draft to avoid duplication and reviewer confusion. PR #16096 will remain the canonical path for root-cause ownership, the fix, waiver cleanup, and verification.

[tensorrt-cicd]

PR_Github #58288 [ run ] completed with state SUCCESS. Commit: dd211ea
/LLM/main/L0_MergeRequest_PR pipeline #46923 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation