[https://nvbugs/6104831][fix] Skip unready futures in checkGenTransferStatus(at_least=1)#13671
Draft
chienchunhung wants to merge 2 commits intoNVIDIA:mainfrom
Draft
[https://nvbugs/6104831][fix] Skip unready futures in checkGenTransferStatus(at_least=1)#13671chienchunhung wants to merge 2 commits intoNVIDIA:mainfrom
chienchunhung wants to merge 2 commits intoNVIDIA:mainfrom
Conversation
…ang in checkGenTransferStatus(at_least=1) Reproduce signature NVIDIA#4 of NVBug 6104831: the disagg generation event loop self-blocks when CacheTransceiver::checkGenTransferStatus() is called with at_least_request_num=1 and the first selected requester future is not yet ready. The PyExecutor disagg loop polls check_gen_transfer_status(1) every iteration to harvest completed receiver-side futures. On stock rc11 the inner for-loop unconditionally calls future.get() on the first entry that lands in the to-complete set, even when its wait_for(0) status is timeout. A single in-flight generation request whose context-side ready signal has not yet arrived therefore blocks the entire decoder event loop, indistinguishable from a wedge. The new test test_check_gen_transfer_status_at_least_one_does_not_block_on_unready_future drives one full ctx/gen handshake to completion to capture a real opaque comm/cache state, then enqueues a second generation request whose context counterpart has not yet been respond_and_send_async-ed. It calls check_gen_transfer_status(1) on a worker thread with a 1s probe timeout and asserts the call returns promptly. Pre-fix the call hangs past the probe timeout (asserted via thread is_alive()); the fix in the matching chained PR makes the call return immediately by probing wait_for(0) and skipping unready futures when blockAll is False. Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com> Made-with: Cursor
…rStatus(at_least=1) Fix signature NVIDIA#4 of NVBug 6104831 by making CacheTransceiver::checkGenTransferStatus() probe each selected requester future with a non-blocking wait_for(0) before calling get(), and skip the entry when it is not yet ready and the caller did not ask to block on all outstanding transfers. The pre-fix loop unconditionally calls future.get() on every entry in toCompleteIdSet. The selection logic above (ready-frequency vector plus insertion-order fallback) does not check future status, so it can pick a request whose receiver-side future is still timeout. When called from the PyExecutor disagg event loop with at_least_request_num=1, an unconditional get() therefore blocks the entire decoder on a single in-flight generation request whose context counterpart has not yet sent its ready signal. From outside the system this is indistinguishable from a permanent wedge. The fix preserves the existing behaviour of the blockAll path (called during teardown) and only changes the at-least-N path used by every PyExecutor iteration: probe wait_for(0); if not ready, advance the iterator and continue. Ready futures are still drained the same way. The chained reproducer test_check_gen_transfer_status_at_least_one_does_not_block_on_unready_future hangs past its 1-second probe timeout pre-fix and returns immediately post-fix. Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com> Made-with: Cursor
chienchunhung
added a commit
to chienchunhung/TensorRT-LLM
that referenced
this pull request
Apr 30, 2026
…13673/NVIDIA#13674 into the investigation report The four sig NVIDIA#4 / NVIDIA#5 / NVIDIA#6 PRs are now open against NVIDIA/TensorRT-LLM: - Sig NVIDIA#4: chained pair NVIDIA#13674 (test) -> NVIDIA#13671 (fix; carries 2 commits including the test, both PRs target main so they can be merged in order) - Sig NVIDIA#5: combined test + fix in NVIDIA#13672 (independent of the #1 chain) - Sig NVIDIA#6: combined test + fix in NVIDIA#13673, chained on top of NVIDIA#13640 (the #1 fix is a prerequisite for the !isReady early-return path) Update the Signature <-> PR Map, the per-signature Status / PRs blocks, and the Next Steps items 2 and 3 to reference these PR numbers and mark the corresponding follow-up items as done. Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com> Made-with: Cursor
chienchunhung
added a commit
to chienchunhung/TensorRT-LLM
that referenced
this pull request
Apr 30, 2026
… PRs landed Apply nine consistency fixes against the post-NVIDIA#13671/NVIDIA#13672/NVIDIA#13673/NVIDIA#13674 state of the investigation: 1. Front-matter Status block: replace the "sig NVIDIA#6 root-caused, validation in flight" wording with the post-run8 picture (all 6 TRT-LLM PRs in review; NVIDIA#7 is an out-of-scope NIXL bug; deadline work is the TRT-LLM-side fallback for NVIDIA#7). 2. Front-matter Branches in this worktree: add the four new sig NVIDIA#4 / NVIDIA#5 / NVIDIA#6 branches. 3. Front-matter Related PRs: add NVIDIA#13674 / NVIDIA#13671 / NVIDIA#13672 / NVIDIA#13673 with chained-on-NVIDIA#13640 callout for NVIDIA#13673. 4. "Configurations that did not reproduce": NVIDIA#5 and NVIDIA#6 now do reproduce in single-process unit tests via the new tests added by NVIDIA#13672 and NVIDIA#13673; only NVIDIA#3 and NVIDIA#7 remain field-only. 5. Phase 6 close: the sig NVIDIA#4 regression test is no longer isolated in local/rc11-disagg-repro - it is now in the chained NVIDIA#13674 / NVIDIA#13671 pair. 6. Signature NVIDIA#6 section: drop the "(suspected)" qualifier and the "(most likely)" hedging on Where-it-lives - both are confirmed by run7 and run8. Rename the section header to describe the actual failure shape (recv buffer index leak via !isReady early return wedging assignBufferIndexForRecv) rather than the early control-path-stall hypothesis. Mirror the rename in the Signature - PR Map row. 7. File / Branch Index "New unit tests": add the new sig NVIDIA#5 (test_cancel_queued_gen_request_fulfills_receiver_future) and sig NVIDIA#6 (test_cancelled_after_ready_does_not_leak_recv_buffer_index, NIXL backend) tests. 8. Signature NVIDIA#3 status hypothesis: add a one-paragraph note that NIXL (signature NVIDIA#7) is now also a candidate cause of the half-initialized state, so a future field hit is not misattributed to a fresh TRT-LLM bug. 9. Phase 5 narrative: add a forward link explaining that the underlying terminal driver of the Phase-5 wedge was already NVIDIA#7 (NIXL), but NVIDIA#4 was the visible TRT-LLM-side symptom because the gen event loop was self-blocking before any of the later layers could surface. Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com> Made-with: Cursor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@coderabbitai summary
Description
Test Coverage
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.