Skip to content

[None][fix] Restore CTX transfer progress when scheduling stalls - #17343

Closed
peihu-nv wants to merge 1 commit into
NVIDIA:feat/m3_with_msafrom
peihu-nv:peihengh/m3-disagg-idle-reap-fix-20260805
Closed

[None][fix] Restore CTX transfer progress when scheduling stalls#17343
peihu-nv wants to merge 1 commit into
NVIDIA:feat/m3_with_msafrom
peihu-nv:peihengh/m3-disagg-idle-reap-fix-20260805

Conversation

@peihu-nv

@peihu-nv peihu-nv commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

@coderabbitai summary

Description

PR #17104 removed the disaggregated transfer admission limit and its per-iteration idle-progress collectives. When a context worker subsequently exhausted available KV capacity while sends were still in flight, no forward could run, so the normal nonblocking completion poll became unreachable. The completed sends continued pinning their KV blocks and the worker could stall indefinitely.

Keep unrestricted transfer admission, but restore a targeted blocking completion poll only when no request fits, no generation-init request is ready, and a context transfer is still in flight. The existing transfer-status path performs the required completion consensus and releases the pinned blocks; normal executor iterations retain the collective-free fast path.

Test Coverage

  • Focused TestDisaggTransferIdleProgress suite passes: 11 passed. It covers the zero-fit/in-flight context-transfer condition and verifies that the existing blocking context-status poll is entered without restoring the removed WORLD or TP/CP entry collectives.
  • Changed-file pre-commit hooks pass.
  • Five independent 900-second MiniMax-M3 AgentX profiles pass on GB300 with one TP2 context worker and four TP4 generation workers: 885 warmup requests and 2,348 profiling requests completed with zero serving errors, timeouts, CUDA assertions, or OOMs. The repaired path was exercised 1,153 times.
  • Ten additional matched stress replicas are in progress.

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>
@peihu-nv
peihu-nv force-pushed the peihengh/m3-disagg-idle-reap-fix-20260805 branch from 2cd7a1e to d0595c5 Compare August 6, 2026 17:21
@peihu-nv
peihu-nv marked this pull request as ready for review August 6, 2026 17:26
@peihu-nv
peihu-nv requested review from a team as code owners August 6, 2026 17:26
@peihu-nv

peihu-nv commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #64386 [ run ] triggered by Bot. Commit: d0595c5 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #64386 [ run ] completed with state SUCCESS. Commit: d0595c5
/LLM/main/L0_MergeRequest_PR pipeline #52273 completed with status: 'SUCCESS'

CI Report

Link to invocation

@peihu-nv peihu-nv closed this Aug 7, 2026
@peihu-nv

peihu-nv commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator Author

New PR aligned with main #17428

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants