[https://nvbugs/6104831][fix] Fulfill receiver future on disagg gen-side queued cancel#13672
Draft
chienchunhung wants to merge 1 commit intoNVIDIA:mainfrom
Draft
Conversation
…ide queued cancel Fix signature NVIDIA#5 of NVBug 6104831: the receiver-side mirror of the sender-side broken-promise bug fixed in NVIDIA#13640. When a generation request is cancelled while it is still queued in the disagg gen-receiver worker (CacheReceiver::Impl::cancelRequest()), the function erases the (request, promise) pair from mRequestsQueue without first fulfilling the promise. The std::promise destructor then sets a future_error: Broken promise on the future returned by request_and_receive_async(), and the disagg gen polling loop in CacheTransceiver::checkGenTransferStatus() surfaces the cancellation as a generic exception with no actionable diagnostic. The fix mirrors the sender-side cancellation handler in CacheSender::Impl::sendResponse: extract the queued promise out of the entry before erasing, then set_exception() a structured kNETWORK_ERROR via TLLM_REQUEST_EXCEPTION so the consumer sees a real cancellation. Failures in the fulfillment path itself are logged but do not propagate, since cancelRequest() must remain best-effort. The new test test_cancel_queued_gen_request_fulfills_receiver_future holds the receiver worker thread busy with a first orphan generation request whose context counterpart will never respond, then enqueues a second orphan request and cancels it while it is still queued. Pre-fix the test fails because Broken promise appears on stderr; post-fix the cancelled request lands in DISAGG_TRANS_ERROR cleanly and stderr stays clean. Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com> Made-with: Cursor
chienchunhung
added a commit
to chienchunhung/TensorRT-LLM
that referenced
this pull request
Apr 30, 2026
…13673/NVIDIA#13674 into the investigation report The four sig NVIDIA#4 / NVIDIA#5 / NVIDIA#6 PRs are now open against NVIDIA/TensorRT-LLM: - Sig NVIDIA#4: chained pair NVIDIA#13674 (test) -> NVIDIA#13671 (fix; carries 2 commits including the test, both PRs target main so they can be merged in order) - Sig NVIDIA#5: combined test + fix in NVIDIA#13672 (independent of the #1 chain) - Sig NVIDIA#6: combined test + fix in NVIDIA#13673, chained on top of NVIDIA#13640 (the #1 fix is a prerequisite for the !isReady early-return path) Update the Signature <-> PR Map, the per-signature Status / PRs blocks, and the Next Steps items 2 and 3 to reference these PR numbers and mark the corresponding follow-up items as done. Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com> Made-with: Cursor
chienchunhung
added a commit
to chienchunhung/TensorRT-LLM
that referenced
this pull request
Apr 30, 2026
… PRs landed Apply nine consistency fixes against the post-NVIDIA#13671/NVIDIA#13672/NVIDIA#13673/NVIDIA#13674 state of the investigation: 1. Front-matter Status block: replace the "sig NVIDIA#6 root-caused, validation in flight" wording with the post-run8 picture (all 6 TRT-LLM PRs in review; NVIDIA#7 is an out-of-scope NIXL bug; deadline work is the TRT-LLM-side fallback for NVIDIA#7). 2. Front-matter Branches in this worktree: add the four new sig NVIDIA#4 / NVIDIA#5 / NVIDIA#6 branches. 3. Front-matter Related PRs: add NVIDIA#13674 / NVIDIA#13671 / NVIDIA#13672 / NVIDIA#13673 with chained-on-NVIDIA#13640 callout for NVIDIA#13673. 4. "Configurations that did not reproduce": NVIDIA#5 and NVIDIA#6 now do reproduce in single-process unit tests via the new tests added by NVIDIA#13672 and NVIDIA#13673; only NVIDIA#3 and NVIDIA#7 remain field-only. 5. Phase 6 close: the sig NVIDIA#4 regression test is no longer isolated in local/rc11-disagg-repro - it is now in the chained NVIDIA#13674 / NVIDIA#13671 pair. 6. Signature NVIDIA#6 section: drop the "(suspected)" qualifier and the "(most likely)" hedging on Where-it-lives - both are confirmed by run7 and run8. Rename the section header to describe the actual failure shape (recv buffer index leak via !isReady early return wedging assignBufferIndexForRecv) rather than the early control-path-stall hypothesis. Mirror the rename in the Signature - PR Map row. 7. File / Branch Index "New unit tests": add the new sig NVIDIA#5 (test_cancel_queued_gen_request_fulfills_receiver_future) and sig NVIDIA#6 (test_cancelled_after_ready_does_not_leak_recv_buffer_index, NIXL backend) tests. 8. Signature NVIDIA#3 status hypothesis: add a one-paragraph note that NIXL (signature NVIDIA#7) is now also a candidate cause of the half-initialized state, so a future field hit is not misattributed to a fresh TRT-LLM bug. 9. Phase 5 narrative: add a forward link explaining that the underlying terminal driver of the Phase-5 wedge was already NVIDIA#7 (NIXL), but NVIDIA#4 was the visible TRT-LLM-side symptom because the gen event loop was self-blocking before any of the later layers could surface. Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com> Made-with: Cursor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@coderabbitai summary
Description
Test Coverage
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.