[https://nvbugspro.nvidia.com/bug/6104831][fix] Disagg transceiver: shared_ptr<LlmRequest>, RAII buffer holders, kv_transfer_timeout_ms enforcement

[yifjiang]

Summary

Fixes NVBug 6104831: disaggregated-serving wedges where prefill/decode workers either crash with std::future_error: Broken promise / bad_optional_access, or silently leak KV-cache blocks and buffer-pool slots until the pool saturates and no new transfers can start. Full investigation notes are in the debug doc.

16 coupled C++/Python commits covering: (a) closing a raw-pointer UAF class on the transceiver's futures maps, (b) enforcing kv_transfer_timeout_ms on both sender and receiver, (c) plumbing perRequestCancel through the data-phase paths so evicted requests actually release their resources, (d) RAII on the buffer-index pool, and (e) hardening the drain worker against non-std::exception escapes.

One-sentence version: shared_ptr ownership + enforced deadline + working cancel path + RAII on pool slots.

Why all 16 commits?

Commit 14 (649d1466b) is the UAF fix proper — it changes mSenderFutures / mRequesterFutures in CacheTransceiver to hold std::shared_ptr<LlmRequest> instead of raw pointers. Confirmed via MALLOC_PERTURB_=85 (observed mRequestId=0x5555555555555555). With that ownership change, the Python-side Path A/B guards that previously deferred cleanup to avoid the UAF are no longer needed; commit 14 removes them.

But fixing the UAF alone doesn't give a working system. These are coupled fixes, not independent — each addresses a distinct failure mode on the same bug family, and removing any one leaves a different no-recovery path open:

Failure mode	Which commits fix it	What happens if you keep only commit 14
UAF / Broken promise / bad_optional_access	14 + 4 (43b518070)	fixed
Stuck transfer never evicted (no deadline)	1–3, 5 (a31977564, a3bce0ce3, a8c27a45b, 3d0a6d183)	executor survives, but stuck senders pin KV blocks forever → pool exhausts → new requests hang
Receiver std::async task blocked in recvReadySignal even after deadline fires	6 (6bb116dac) + 7–10 cancel-flag plumbing (62c5d24ae, 7e8016a2f, e7fcbab01, 2860ac50a)	deadline fires, Python drops its ref, but the worker task is still waiting for a signal that'll never arrive → NIXL/UCX per-request state accumulates → "no recovery" wedge even after load stops
Buffer-pool slot leak on non-happy exits	11–13 RAII holders (39f5452ea, 28f836e19, 6f85ce895)	one early-return in requestSync → index never released → size-1 buffer pool permanently wedged
Unhandled non-std::exception in drain worker	15 (abfcb6d39)	a single std::bad_optional_access (or similar) still kills the thread

Concretely, running a 1P1D disagg loadgen stress reproducer (single prefill worker, single decode worker, concurrent request burst at the saturation point, then idle and probe for recovery):

• Commit 14 alone → crash signatures (Broken promise, bad_optional_access) disappear, but stuck transfers still never get evicted. Pool exhausts, behavior looks superficially similar to the pre-fix wedge (no crash marker, no forward progress). Still no recovery after a 180 s idle probe window.
• 14 + deadline commits (1–5) → deadline fires, but receiver zombies accumulate: each timed-out receive leaves an std::async task spinning inside recvReadySignal, pinning NIXL state. After a saturation window, every new transfer deterministically waits kv_transfer_timeout_ms even if the system is otherwise idle.
• 14 + 1–5 + 6–10 → recovery works in principle, but buffer-pool slots leak on error paths. Slow drift toward wedge under sustained error-path churn.
• All 16 → crash gone, deadline fires, receivers unwind cleanly, buffer slots released; idle-then-probe reliably returns to serving within the probe window.

Order also matters on the removal side: commit 14 only becomes safe after the earlier commits are in place, because the deadline + cancel plumbing they add is what the Path A/B guards were substituting for. Removing the guards without that plumbing in place would regress the orphan-induced KV-block leak the guards existed to mitigate.

Commits (in applied order)

1. a31977564 — Enforce kv_transfer_timeout_ms on prefill sender + guard against dangling pointers
2. a3bce0ce3 — Report timed-out transfers to Python for full cleanup
3. a8c27a45b — Apply kv_transfer_timeout_ms in blockAll mode
4. 43b518070 — Close remaining UAF paths and enforce gen-side deadline
5. 3d0a6d183 — Hoist kv_transfer_timeout_ms check above readiness gate
6. 6bb116dac — Release receiver resources when gen-side deadline evicts
7. 62c5d24ae — sendRequestInfo / receiveReadySignal / data-phase send honor perRequestCancel
8. 7e8016a2f — Sender-side in-flight cancel parity + notifySyncMessage pre-notify check
9. e7fcbab01 — Tie sender cancel-flag lifetime to session + port catch(...) to sender worker
10. 2860ac50a — Bound assignBufferIndex CV wait + plumb cancel + diagnostic [buf] logs
11. 39f5452ea — RAII BufferIndexHolder for requestSync — stop leaking pool slots on non-happy exits
12. 28f836e19 — RAII BufferIndexHolder on sender formatters + reqId-tagged [buf] logs
13. 6f85ce895 — Send-side RAII: atomic release(), cancel parity, FUTURE_WAIT/JOIN diagnostics
14. 649d1466b — Carry std::shared_ptr<LlmRequest> through transceiver — close UAF, remove Path A/B guards, unwedge KV block leak
15. abfcb6d39 — Harden drain worker loop against non-std::exception escapes
16. c9777c4ac — Re-add is_disagg_context_complete_state guard to _handle_responses

Base

Branch is linear on top of v1.3.0rc11 (4e69c14f732a6e6afce4f71616db5b5cd2b10530) — chosen so it can be built + tested against the tensorrt-llm==1.3.0rc11 wheel baked into NVIDIA NGC's dynamo-trtllm images. 16 commits on top of that tag.

[coderabbitai]

📝 Walkthrough

Walkthrough

Two files are modified to enhance KV cache transfer timeout handling and request state management. The C++ cache transceiver adds timeout configuration retrieval, debug logging, and timeout-based cleanup logic for pending transfers. The Python executor adds conditional state handling to defer termination of disaggregated context transmission requests.

Changes

Cohort / File(s)	Summary
KV Cache Transfer Timeout & Logging cpp/tensorrt_llm/batch_manager/cacheTransceiver.cpp	Enhanced timeout handling with retrieval of kvTransferTimeoutMs configuration. Added elapsed-time guards to detect and remove timed-out transfers. Introduced detailed debug logging of sender futures state and completion statistics. Modified exception handling to log warnings without mutating request state, then remove affected entries from pending transfers.
Disaggregated Context Transmission State Handling tensorrt_llm/_torch/pyexecutor/py_executor.py	Added explicit conditional branch in _handle_responses to defer termination of requests in disaggregated context transmission state. Prevents premature termination while allowing existing timeout checks to manage completion or timeout scenarios.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately reflects the main changes: enforcing kv_transfer_timeout_ms and guarding against dangling pointers in the disaggregated transceiver path.
Description check	✅ Passed	The PR description is comprehensive and thoroughly documents the fix, including detailed investigation notes, a 16-commit breakdown with coupling rationale, and test validation.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@cpp/tensorrt_llm/batch_manager/cacheTransceiver.cpp`:
- Around line 499-513: The diagnostic loop in checkContextTransferStatus is
dereferencing potentially-stale raw pointers from mSenderFutures (it reads
req->mRequestId and req->getKvCacheTransferStart()), causing use-after-free;
change the logging so it does not dereference req (only log the pointer address
and index) or else store safe copies of req->mRequestId and the start timestamp
at the time entries are inserted into mSenderFutures so the diagnostics read
those cached values instead of accessing req. Update the loop using the
mSenderFutures entry variables (auto& [req,fut]) to avoid any req->... calls, or
refactor insertion code to capture mRequestId and getKvCacheTransferStart into
the stored struct so getKvCacheTransferStart and mRequestId are never read from
the raw pointer during logging.
- Around line 493-497: The code currently only loads senderFutureTimeoutMs and
kvTransferTimeoutMs when (!blockAll), which causes kvTransferTimeoutMs to be
unset when blockAll is true and allows future.get() to block indefinitely;
change the logic so kvTransferTimeoutMs is always read from
mCacheTransceiverConfig (call getKvTransferTimeoutMs() and assign to
kvTransferTimeoutMs) regardless of blockAll, but keep the conditional skip of
getKvTransferSenderFutureTimeoutMs() / senderFutureTimeoutMs when blockAll is
true; update the block that references mCacheTransceiverConfig,
senderFutureTimeoutMs, kvTransferTimeoutMs, and blockAll to reflect this
separation.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 8073f6b4-20e7-4498-88ea-a36216bad2e6

📥 Commits

Reviewing files that changed from the base of the PR and between 61cef21 and d7844fc.

📒 Files selected for processing (2)

• cpp/tensorrt_llm/batch_manager/cacheTransceiver.cpp
• tensorrt_llm/_torch/pyexecutor/py_executor.py

[coderabbitai]

⚠️ Potential issue | 🟠 Major

Load kvTransferTimeoutMs even when blockAll is true.

blockAll should disable the per-iteration poll timeout, not the overall transfer deadline. With the current guard, any caller that omits atLeastRequestNum still falls through to a blocking future.get() with no kv_transfer_timeout_ms enforcement, so a stuck sender can still hang forever.

Suggested fix

-    if (!blockAll && mCacheTransceiverConfig.has_value())
+    if (mCacheTransceiverConfig.has_value())
     {
-        senderFutureTimeoutMs = mCacheTransceiverConfig->getKvTransferSenderFutureTimeoutMs();
         kvTransferTimeoutMs = mCacheTransceiverConfig->getKvTransferTimeoutMs();
+        if (!blockAll)
+        {
+            senderFutureTimeoutMs = mCacheTransceiverConfig->getKvTransferSenderFutureTimeoutMs();
+        }
     }

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@cpp/tensorrt_llm/batch_manager/cacheTransceiver.cpp` around lines 493 - 497,
The code currently only loads senderFutureTimeoutMs and kvTransferTimeoutMs when
(!blockAll), which causes kvTransferTimeoutMs to be unset when blockAll is true
and allows future.get() to block indefinitely; change the logic so
kvTransferTimeoutMs is always read from mCacheTransceiverConfig (call
getKvTransferTimeoutMs() and assign to kvTransferTimeoutMs) regardless of
blockAll, but keep the conditional skip of getKvTransferSenderFutureTimeoutMs()
/ senderFutureTimeoutMs when blockAll is true; update the block that references
mCacheTransceiverConfig, senderFutureTimeoutMs, kvTransferTimeoutMs, and
blockAll to reflect this separation.

[coderabbitai]

⚠️ Potential issue | 🔴 Critical

Don't dereference req in the dangling-pointer diagnostics.

This loop reads req->mRequestId and req->getKvCacheTransferStart() from mSenderFutures, but that container is exactly where the raw pointer may already be stale. In the failure mode this PR is fixing, the debug path itself becomes the use-after-free.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@cpp/tensorrt_llm/batch_manager/cacheTransceiver.cpp` around lines 499 - 513,
The diagnostic loop in checkContextTransferStatus is dereferencing
potentially-stale raw pointers from mSenderFutures (it reads req->mRequestId and
req->getKvCacheTransferStart()), causing use-after-free; change the logging so
it does not dereference req (only log the pointer address and index) or else
store safe copies of req->mRequestId and the start timestamp at the time entries
are inserted into mSenderFutures so the diagnostics read those cached values
instead of accessing req. Update the loop using the mSenderFutures entry
variables (auto& [req,fut]) to avoid any req->... calls, or refactor insertion
code to capture mRequestId and getKvCacheTransferStart into the stored struct so
getKvCacheTransferStart and mRequestId are never read from the raw pointer
during logging.

[coderabbitai]

⚠️ Potential issue | 🟠 Major

Don't erase timed-out or failed entries without surfacing the request ID back to Python.

After these erase(it) calls, check_context_transfer_status() no longer returns the request ID, but tensorrt_llm/_torch/pyexecutor/py_executor.py:3244-3278 only calls _end_transfer_and_maybe_terminate() for IDs present in finished_requests + error_requests. That means the request can stay pinned in AsyncTransferManager indefinitely unless a later cancel_request() happens to succeed.

Also applies to: 658-662

[Tabrizian]

@yifjiang Is it possible to have a test that demonstrates the issues? Also, can you please provide a more concise summary of the issue and a minimal reproducer that reproduces the problem.

Also, in the PR description is it possible to only reference PRs that are in TRT-LLM main branch (do NOT reference PRs that are closed or not merged).

[Tabrizian]

This link doesn't work: https://github.com/yifjiang/dynamo-gb200-test/blob/main/docs/rc11-crash-and-hang-proof.md

[yifjiang]

@Tabrizian thanks for the review. Addressed in 88987cc:

• Test: added test_context_transfer_times_out_on_stuck_sender in tests/unittest/others/test_kv_cache_transceiver.py. It enqueues a ctx transfer without starting a matching receiver, configures kv_transfer_timeout_ms=500 ms, and asserts that check_context_transfer_status returns the request in error_requests, sets state to DISAGG_TRANS_ERROR, and evicts the entry. This is a direct unit-level reproduction of the stuck-sender path that the C++ change fixes.
• Description: rewritten to be concise — summary, root cause, and a minimal reproducer (Qwen3-0.6B 1P1D, concurrency 64, ISL 8000). Kept only the reference to [TRTLLM-6106][feat] Add support for KVCache transfer from KVCache reuse path #6348 (merged) and dropped all references to unmerged / closed PRs.
• Dead link: removed the reference to the external rc11-crash-and-hang-proof.md — the reproducer is now inlined in the PR description so no external artifact is needed.

[Tabrizian]

Is it possible to include the client command in the reproducing instructions too? Thank you.

[Tabrizian]

The LlmRequest pointer is safe. It is kept alive in AsyncTransferManager until transfer is completed.

[yifjiang]

Reviewing my earlier reply — I was sloppy. Let me split the answer by code path because your statement holds on the line you anchored on but not universally.

For this specific line (elif request.is_disagg_context_transmission_state: pass inside the force_terminate_for_partial_reuse branch of _handle_responses): you are correct. The pre-fix code would call _terminate_request → _do_terminate_request → resource_manager.free_resources(request). None of that touches async_transfer_manager; AsyncTransferManager._requests_in_transfer still holds the Python reference. So the LlmRequest itself is not destroyed. No classic LlmRequest-UAF on this path.

What the guard still prevents is some form of corruption that empirically fires under load (free(): invalid next size through LRUEvictionPolicy::claimBlock, heap metadata detected as invalid at a later unrelated free()). I previously claimed this was specifically "KV-cache-block corruption" — I retract that claim. I don't have ASan or similar verification for the actual mechanism; I was constructing a plausible story. Could be something else entirely. What I can say is that eliminating this termination reliably eliminates the crash under the overload workload.

For Path B (_check_disagg_ctx_cache_transfer_status, the py_kv_transfer_timed_out retry-cancel loop): the invariant "AsyncTransferManager keeps it alive" is not applicable. Path B calls _end_transfer_and_maybe_terminate, which calls AsyncTransferManager.end_transfer, which is precisely the code that pops _requests_in_transfer — i.e. that path itself is the "transfer is completed" signal that drops AsyncTransferManager's hold. A "Request X not found in transfer manager" warning observed once in an instrumented run is the direct fingerprint of this premature pop. Surviving Python references after Path B (active_requests membership, locals) are not load-bearing. So on Path B, LlmRequest lifetime after the pop is not guaranteed.

For Path A gen side (_handle_responses → py_kv_transfer_timed_out → cancel_request + _handle_errors, gen request in DISAGG_GENERATION_TRANS_IN_PROGRESS): gen requests never enter AsyncTransferManager._requests_in_transfer at all (only ctx requests go through async_transfer_manager.start_transfer). The only Python references are active_requests and locals. _handle_errors explicitly removes from active_requests then calls _terminate_request. No AsyncTransferManager holder exists to save us here — this guard is the only thing standing between _handle_errors and a classic UAF on mRequesterFutures.

I've pushed 568454fbd that updates the inline comments to reflect this split (and admits the force_terminate_for_partial_reuse crash mechanism is uncorroborated) rather than the blanket "dangling pointer / UAF" framing that was there before.

[Tabrizian]

This is not true, store_blocks_for_reuse has a pin argument to make sure the blocks are not deallocated even after the request is terminated.

[yifjiang]

You're right — I hadn't accounted for the store_blocks_for_reuse(request, True) pin set at AsyncTransferManager.start_transfer (py_executor.py:207). I've confirmed:

• start_transfer calls kv_cache_manager.store_blocks_for_reuse(request, True) → blocks are pinned.
• _terminate_request → free_resources(request) calls remove_sequence(..., pin_on_release=False). Per your note, that does not drop the pin set by store_blocks_for_reuse, so the blocks remain live through a Python-side termination.
• unpin_blocks_by_id is invoked only from AsyncTransferManager.end_transfer (py_executor.py:241), i.e. the transfer-completion signal.

So for the force_terminate_for_partial_reuse path (the line you anchored on), both safety nets apply: AsyncTransferManager holds the LlmRequest Python ref, and the pin keeps the KV blocks live. My earlier "KV-block corruption from free_resources" story is ruled out by the pin. I withdraw that framing.

That leaves me more confused, not less, about why the crash actually fires on that path. Empirically, with enable_partial_reuse_for_disagg=true on rc11 (no guard), 37× storeContextBlocks: Can not find sequence warnings and free(): invalid next size (fast) through LRUEvictionPolicy::claimBlock → __libc_free reproduce in ~45–65 s at conc 48, ISL 8000. With the is_disagg_context_transmission_state guard active, that crash disappears and the storeContextBlocks count drops to 1. Something about running _terminate_request on an in-transmission request is breaking the KV cache manager's invariants in a way that later surfaces during claimBlock — but with both LlmRequest and pinned blocks accounted for, I can't offer a mechanism. My best current guess is that the storeContextBlocks: Can not find sequence warnings are a symptom of remove_sequence running with a sequence the partial-reuse path expected to later storeContextBlocks for, leaving a dangling block-id / sequence-id mapping that claimBlock later trips on — but that's still speculation.

Do you have a theory on what's actually being corrupted? If not, I'd like to run this under MALLOC_CHECK_=3 or ASan to get a writer stack on the corruption site rather than leaving it as "empirical correlation, mechanism TBD." The fix is straightforward; I just don't want to ship a comment that keeps hand-waving about the cause.

For the other two paths the pin argument doesn't change the per-path verification I posted: _check_disagg_ctx_cache_transfer_status Path B explicitly calls end_transfer which unpins the blocks (py_executor.py:241) and pops _requests_in_transfer; _handle_responses Path A gen side has no AsyncTransferManager/pin path at all (receiver-side transfers go through request_and_receive_async, not async_transfer_manager.start_transfer). So those guards are still preventing real, nameable hazards. I'll update the code comment at 3791 to reflect your correction on the pin.

[yifjiang]

Follow-up — got the forensic mechanism. Ran the same rc11 pre-fix image under MALLOC_PERTURB_=85 / MALLOC_CHECK_=3 with the same workload (conc 48, ISL 8000, 120 s) and caught the UAF directly:

[TensorRT-LLM][ERROR] Error occurred during context transfer for
  request 6148914691236517205: std::future_error: Broken promise

6148914691236517205 == 0x5555555555555555 — the glibc poison fill written into freed memory. That log line comes from checkContextTransferStatus's catch block doing LOG_ERROR("... %ld ...", it->first->mRequestId). Request IDs are monotonic 64-bit integers, so 0x55..55 is not a legal ID; the mRequestId field must have been read from a freed slot. Direct mechanistic proof the LlmRequest was freed out from under mSenderFutures.

Walking through why, given your pin correction:

• AsyncTransferManager.start_transfer pins via store_blocks_for_reuse(request, True) — blocks stay live through _terminate_request (your correction, confirmed).
• _terminate_request alone does not free the LlmRequest — AsyncTransferManager's strong ref survives. So the force_terminate_for_partial_reuse path in isolation does not produce the UAF; it only produces the storeContextBlocks: Can not find sequence warnings (observed 37x-102x) via remove_sequence.
• The free happens on Path B in _check_disagg_ctx_cache_transfer_status. Path B calls _end_transfer_and_maybe_terminate, which calls AsyncTransferManager.end_transfer, which pops _requests_in_transfer — the drop you were pointing at as "until transfer is completed" is baked into this path's cancel sequence. After the pop, the iteration-local request is the only remaining strong ref; when the loop moves on, the pybind shared_ptr hits zero and the C++ LlmRequest is destroyed. The mSenderFutures entry's raw pointer is now dangling.
• Under MALLOC_PERTURB_, the freed region is filled with 0x55; the catch-block read captures that pattern. Without MALLOC_PERTURB_ (original rc11 run), the same region gets reused by a later allocation; a subsequent C++ write (setState(kDISAGG_TRANS_ERROR) at 0x06 into what is now another object's mid-word) corrupts heap metadata, and the next unrelated free() in LRUEvictionPolicy::claimBlock aborts with invalid next size.

So the corrected mechanism for this PR's fix is:

• Path B guard (in _check_disagg_ctx_cache_transfer_status): forensically confirmed to prevent a classic LlmRequest-UAF. This is the path that drops the AsyncTransferManager ref.
• Path A gen-side guard (in _handle_responses): classic UAF by construction — gen requests never enter AsyncTransferManager, so _handle_errors is a direct free. No forensic run targeted gen-side separately in this investigation, but the argument is airtight from the code.
• force_terminate_for_partial_reuse guard (the line you anchored on): neither the LlmRequest nor the pinned blocks are freed by this path alone. Skipping it still matters because in the real workload the same request is later picked up by Path B once py_kv_transfer_timed_out fires, and Path B is where the confirmed free happens. Whether this path alone can crash independent of Path B is not directly isolated.

Caveats on the heap-debug run:

• The MALLOC_CHECK_=3 abort on invalid next size did not trigger in the 12-min window — a pure read of a poisoned slot doesn't flag the check. The poison read is the proof, not an abort.
• An ASan rebuild would pin the first write through the dangling pointer with its stack. That's a source rebuild, so out of scope for this image-only investigation but a clean follow-up if more forensics are wanted.

Pushed 5779e5c53df42104b063a342f98a832407fec82d updating the inline comments to reflect this — Path B guard comment now cites the 0x55..55 evidence; Path A guard splits ctx vs gen accurately; force_terminate_for_partial_reuse guard comment keeps your pin correction and frames the composite-with-Path-B story without overclaiming. Also walking the walk-back back in the C++ catch-block comments.

[Tabrizian]

We already have a timeout in the py_executor level, this test doesn't trigger the py_executor level changes so that could be why that timeout is not triggered.

[yifjiang]

Correct — this test exercises the C++ check_context_transfer_status API directly, bypassing py_executor. The py_executor-level _check_kv_transfer_timeout → cancel_request → _handle_errors path is intentionally not in the loop here; the point of the test is to verify the C++ kv_transfer_timeout_ms enforcement in isolation.

Why the C++-layer enforcement still matters even though py_executor already has a timeout: the py_executor-level path (_check_kv_transfer_timeout fires py_kv_transfer_timed_out → either _handle_responses Path A or _check_disagg_ctx_cache_transfer_status Path B) calls cancel_request + _handle_errors / _end_transfer_and_maybe_terminate on requests whose C++ transfer is still in flight. Under load, that reliably produces free(): invalid next size on ctx side, and creates a classic UAF window on gen side (gen requests have no AsyncTransferManager holder). See the split analysis on the py_executor.py thread above for per-path specifics.

The Python guards in this PR skip that py-executor cleanup for in-transmission requests and defer to the C++ deadline, which evicts the entry from mSenderFutures / mRequesterFutures before any Python-side free_resources can run. That's the enforcement path the test is directly exercising.

I can add a heavier py_executor-level integration test too if you'd like — it would need a full LLM + 1P1D fixture. Let me know if that's worth it vs. the C++-API unit test.

[yifjiang]

@Tabrizian client command added to the reproducer section of the PR description — full loadgen script (asyncio, concurrency 64, ISL ~8000, OSL ~200, 180s load + 5 recovery probes) plus a single-shot curl sanity check. It talks to the OpenAI-compatible /v1/chat/completions endpoint on the frontend.

I also updated the "what actually breaks" paragraph to reflect the mechanism you pointed out in py_executor.py:3762 — the LlmRequest stays alive via AsyncTransferManager; the corruption is at the KV-cache-block level (free_resources returns blocks to LRUEvictionPolicy's free queue while the NIXL sender is still writing GPU bytes into them). Full reply on that thread too.

[yifjiang]

Adding empirical backing for the two inline threads above. Numbers below, mechanism analysis further down. Update (2026-04-17): forensic confirmation via MALLOC_PERTURB_ at the bottom of this comment.

Setup — reproducible from a fresh TRT-LLM build (no prebuilt image assumed) with a small observability patch that instruments counters on the two Python paths in question. Deploy 1P1D disaggregated serving (one prefill + one decode worker) with:

• Model: Qwen/Qwen3-0.6B
• Backend: NIXL over TCP (UCX_TLS=tcp,cuda_copy,self)
• cache_transceiver_config.kv_transfer_timeout_ms = 60000
• cache_transceiver_config.kv_transfer_sender_future_timeout_ms = 10000
• Both workers: free_gpu_memory_fraction: 0.3
• Prefill: disable_overlap_scheduler: true
• Decode: disable_overlap_scheduler: false

Workload — asyncio client hitting the frontend's /v1/chat/completions: concurrency 48, ISL ~Normal(8000, 2000) floored at 100, OSL ~Normal(200, 50) floored at 50, temperature=0, non-streaming, 120 s sustained load, then 5 recovery probes.

Run A — baseline (no PR #13056 fix), observability only. Counts how often each Python path operates on an in-transmission request:

Event	Prefill	Decode
_check_disagg_ctx_cache_transfer_status Path B — cancel_request returned True on a ctx request in DISAGG_CONTEXT_TRANS_IN_PROGRESS, then _end_transfer_and_maybe_terminate ran	23	0
_handle_responses Path A — py_kv_transfer_timed_out would have triggered cancel_request + _handle_errors on a gen request in DISAGG_GENERATION_TRANS_IN_PROGRESS	—	12
CacheSender::cancelRequest returning True on an in-transmission ctx request	23 / 23	—
"Request X not found in transfer manager" warning	1	0

The 23/23 on CacheSender::cancelRequest shows this isn't a corner case — every ctx request whose Python-level timeout fires while C++ is still driving its transfer gets cancel_request == True (the request is still in mReadyResponses), and the pre-fix code proceeds into _handle_errors / _end_transfer_and_maybe_terminate. The single "not found" warning is the direct fingerprint that Path B did pop _requests_in_transfer on a request that C++ still considered in flight.

Run B — same setup and workload, with the Python guards in this PR applied (both Path A and Path B skip termination when the request is in a transmission state):

Event	Prefill	Decode
Termination skipped by the new guard (per firing, not unique requests)	4322	12
Heap crash (free(): invalid next size)	none	none
C++ kv_transfer_timeout_ms eviction fires (ctx side)	63	0
C++ kv_transfer_timeout_ms eviction fires (gen side)	—	0
Recovery probes pass after sustained load	—	0 / 5

So:

• Prefill is healed by the Python guard + ctx-side C++ deadline. 63 stuck entries evicted cleanly, KV blocks unpinned via _end_transfer_and_maybe_terminate, no crash, no hang.
• Decode still hangs even with the Python guard — the 12 would-be terminations are correctly skipped, but without a checkGenTransferStatus deadline, stuck receivers accumulate in mRequesterFutures, the gen KV pool exhausts, and the server doesn't recover. That's why daf70d403 adds the gen-side C++ deadline and b9450529b hoists the check above the toCompleteIdSet readiness gate (so entries that are never "ready" still get their deadline checked).

---

Mechanism — per-path

• Path B guard (_check_disagg_ctx_cache_transfer_status retry-cancel): this path calls _end_transfer_and_maybe_terminate → AsyncTransferManager.end_transfer, which is literally the code that pops _requests_in_transfer. The "AsyncTransferManager keeps it alive" invariant does not apply here — this path is the completion signal that drops the hold. After the pop, surviving Python references (active_requests, loop locals) are transient; the pybind shared_ptr refcount drops to zero, the C++ LlmRequest is destroyed, and mSenderFutures's raw pointer is stale. Forensic confirmation in the follow-up run below.

• Path A gen-side guard (_handle_responses → py_kv_transfer_timed_out for gen): gen requests never enter AsyncTransferManager. _handle_errors explicitly removes the request from active_requests and calls _terminate_request. No holder exists to save the LlmRequest — this guard is the only thing preventing a classic UAF on mRequesterFutures on the gen side.

• force_terminate_for_partial_reuse guard (the line @Tabrizian anchored on): _terminate_request does not touch async_transfer_manager, and store_blocks_for_reuse(request, True) pins the KV blocks (thanks to @Tabrizian for that correction). So neither the LlmRequest nor its pinned blocks are freed by this path alone. The observable impact is the 37×–102× storeContextBlocks: Can not find sequence warnings from remove_sequence. In the real workload the same request is later picked up by Path B once py_kv_transfer_timed_out fires, and Path B is where the forensically confirmed free occurs. Whether this path could crash independent of Path B via a KVCacheManager-state inconsistency has not been isolated.

---

Update (2026-04-17): forensic confirmation of the Path B UAF via MALLOC_PERTURB_

Re-ran the same pre-fix image, same workload, with heap-debug env on both workers:

MALLOC_CHECK_: "3"
MALLOC_PERTURB_: "85"        # fills freed memory with 0x55
GLIBC_TUNABLES: "glibc.malloc.check=3"
LIBC_FATAL_STDERR_: "1"

After ~60 s of load, the prefill worker logged:

[TensorRT-LLM][ERROR] Error occurred during context transfer
  for request 6148914691236517205: std::future_error: Broken promise

6148914691236517205 == 0x5555555555555555 — the poison fill pattern glibc writes into freed memory under MALLOC_PERTURB_=85. The log line comes from checkContextTransferStatus's catch block reading it->first->mRequestId off mSenderFutures. Request IDs are monotonic 64-bit integers from the executor; 0x55…55 is not a legal ID. The only way for mRequestId to read as 0x55…55 is if the memory the field was stored in has been freed and poisoned.

This is direct, mechanical proof that the LlmRequest was freed out from under mSenderFutures.

Sequence:

1. Python timeout fires for a request in DISAGG_CONTEXT_TRANS_IN_PROGRESS. Path B runs: cancel_request → True, _end_transfer_and_maybe_terminate → AsyncTransferManager.end_transfer pops _requests_in_transfer (drops the AsyncTransferManager ref), then _terminate_request → free_resources → remove_sequence (explaining the 102 storeContextBlocks: Can not find sequence warnings in the same run).
2. The iteration-local request in Path B is the only surviving strong ref. When the loop moves on, pybind shared_ptr refcount hits zero; the C++ LlmRequest is destroyed.
3. Glibc either reuses the slot or fills it with 0x55.
4. Either via broken-promise resolution or the next tick iterating mSenderFutures, the C++ code dereferences the stale raw pointer. The catch-block log above is the smoking gun.

Caveats:

• MALLOC_CHECK_=3 did not SIGABRT during the heap-debug window. A pure read of poisoned bytes doesn't flag invalid next size; it only fires on metadata damage at a later unrelated free. The poison read is the proof, not the abort.
• The subsequent request->setState(kDISAGG_TRANS_ERROR) writes 0x06 into the freed region's mState offset. Whether that write trips the check depends on which glibc bin the slot was assigned to — timing-sensitive. To force a deterministic abort at the write site, the next step would be an ASan rebuild, which is out of scope for this image-only investigation.
• Decode worker stayed entirely quiet (0 of every signature). The crash mechanism in this image is exclusively prefill/sender side — consistent with mSenderFutures being the container holding the freed LlmRequest.

Pushed 5779e5c53 updating the inline code comments with this forensic evidence (Path B guard comment cites the 0x55..55 read; Path A guard splits ctx vs gen accurately; force_terminate_for_partial_reuse guard acknowledges @Tabrizian's pin correction and frames the composite-with-Path-B story without overclaiming).