[None][feat] BREAKING Support saving last N turns in per-conversation policy

[jiaganc]

Dev Engineer Review

• Added nested BlockReuseConfig under KvCacheConfig.
• Added max_num_turns, with a default value of 1.
• Updated cache managers, Mamba handling, manifests, examples, and tests to use the nested configuration.
• Multi-turn per_conversation reuse now retains the configured number of completed turns.
• This is a breaking API change. Replace kv_cache_config.block_reuse_policy with kv_cache_config.block_reuse_config.policy.
• Pre-commit checks and MARS compilation passed.
• Disaggregated AGA benchmarks started successfully for supported reuse modes.

QA Engineer Review

• Updated tests/unittest/_torch/executor/test_kv_cache_manager_v2.py.
  • Added test_per_conversation_policy_retains_configured_number_of_turns.
  • Updated the manager fixture to accept max_num_turns.
  • Added the max_num_turns fixture.
• Updated configuration usage in:
  • tests/unittest/disaggregated/test_cache_transceiver_single_process.py
  • tests/unittest/disaggregated/test_kv_transfer.py
  • tests/unittest/kv_cache_manager_v2_tests/test_kv_cache_stats_behavior.py
  • tests/unittest/llmapi/test_llm_args.py
  • tests/unittest/_torch/executor/test_mamba_cache_manager.py
• Tests cover nested configuration defaults, serialization, validation, warning behavior, Mamba snapshot handling, and multi-turn cache eviction.
• No integration test-list coverage is provided for the modified test functions. Verdict: needs follow-up.

Description

Conversation-scoped KV-cache reuse currently retains only one completed turn, which prevents reuse when a conversation rewinds to an older turn.

This change:

• introduces a nested BlockReuseConfig under KvCacheConfig;
• moves block_reuse_policy into that config;
• adds max_num_turns, a positive integer that defaults to 1;
• retains the configured number of completed conversation turns for per_conversation reuse.

The default remains one retained turn, so runtime behavior is unchanged unless max_num_turns is configured. The configuration shape changes from:

kv_cache_config:
  block_reuse_policy: per_conversation

to:

kv_cache_config:
  block_reuse_config:
    block_reuse_policy: per_conversation
    max_num_turns: 5

Test Coverage

• Pre-commit checks passed.
• MARS compilation completed successfully in Slurm job 277527.
• Disaggregated AGA benchmarks started successfully for all_reusable, per_request, per_conversation with one turn, and per_conversation with five turns. Each run processed more than 7,000 main-conversation requests before cancellation.

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[jiaganc]

/bot run

[tensorrt-cicd]

PR_Github #62933 [ run ] triggered by Bot. Commit: 59b867a Link to invocation

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

Walkthrough

The change adds nested BlockReuseConfig settings, bounded conversation-turn retention, updated cache-manager handling, and migrated examples, tests, telemetry, and exports.

Changes

Block reuse configuration and conversation retention

Layer / File(s)	Summary
BlockReuseConfig API contract tensorrt_llm/llmapi/llm_args.py, tensorrt_llm/llmapi/__init__.py, tensorrt_llm/usage/llm_args_golden_manifest.json, tests/unittest/llmapi/test_llm_args.py, docs/source/developer-guide/telemetry.md	Adds nested policy and max_num_turns settings. Updates exports, validation, serialization, warnings, manifest paths, and telemetry documentation.
Runtime conversation retention tensorrt_llm/_torch/pyexecutor/kv_cache_manager_v2.py, tensorrt_llm/_torch/pyexecutor/mamba_cache_manager.py, tests/unittest/_torch/executor/*	Reads nested block reuse settings. Retains bounded conversation drop handles and validates eviction after the configured turn limit.
Configuration migration across integrations examples/disaggregated/slurm/cache_transceiver_test/run_cache_transceiver_test.py, tests/unittest/disaggregated/*, tests/unittest/kv_cache_manager_v2_tests/test_kv_cache_stats_behavior.py	Replaces string-based block_reuse_policy fields with per-instance BlockReuseConfig values.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Request
  participant KVCacheManagerV2
  participant ConversationManager
  participant DropHandle
  Request->>KVCacheManagerV2: Start conversation request
  KVCacheManagerV2->>ConversationManager: Pass max_num_turns
  ConversationManager->>DropHandle: Append planned drop handle
  ConversationManager->>DropHandle: Drop oldest handle after the limit
  KVCacheManagerV2-->>Request: Complete cache reuse flow

Loading

Possibly related PRs

• NVIDIA/TensorRT-LLM#16060: Updates related KVCacheManagerV2 integration.
• NVIDIA/TensorRT-LLM#16714: Modifies another KVCacheManagerV2 cache behavior layer.
• NVIDIA/TensorRT-LLM#16957: Modifies KvCacheConfig, BlockReuseConfig, and KV-cache block reuse behavior.

Suggested reviewers: nvpohanh, simengliu-nv, chuangz0

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 10.53% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly identifies the breaking feature: retaining the last N turns for per-conversation KV-cache reuse.
Description check	✅ Passed	The description explains the motivation, API changes, default behavior, test coverage, and required checklist sections.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tensorrt_llm/llmapi/llm_args.py`:
- Around line 3621-3649: Update stale block_reuse_policy references: in
test_mamba_cache_manager.py, pass the policy through KvCacheConfig’s
block_reuse_config using BlockReuseConfig(block_reuse_policy=...) instead of
passing the field directly; in telemetry.md, access it via
kv_cache_config.block_reuse_config.block_reuse_policy.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e7fd2494-2650-4401-a6c8-cc06c53c6e64

📥 Commits

Reviewing files that changed from the base of the PR and between 138eb43 and 59b867a.

📒 Files selected for processing (11)

• examples/disaggregated/slurm/cache_transceiver_test/run_cache_transceiver_test.py
• tensorrt_llm/_torch/pyexecutor/kv_cache_manager_v2.py
• tensorrt_llm/_torch/pyexecutor/mamba_cache_manager.py
• tensorrt_llm/llmapi/__init__.py
• tensorrt_llm/llmapi/llm_args.py
• tensorrt_llm/usage/llm_args_golden_manifest.json
• tests/unittest/_torch/executor/test_kv_cache_manager_v2.py
• tests/unittest/disaggregated/test_cache_transceiver_single_process.py
• tests/unittest/disaggregated/test_kv_transfer.py
• tests/unittest/kv_cache_manager_v2_tests/test_kv_cache_stats_behavior.py
• tests/unittest/llmapi/test_llm_args.py

[jiaganc]

/bot run

[coderabbitai]

🧹 Nitpick comments (1)

tests/unittest/_torch/executor/test_mamba_cache_manager.py (1)

1651-1684: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Expose max_num_turns in the test helper.

The generic KVCacheManagerV2 test covers max_num_turns=2, but no MambaHybridCacheManagerV2 test covers multi-turn retention or eviction. Add the parameter, pass it to BlockReuseConfig, and add hybrid coverage, including the five-turn case.

Test coverage verdict: insufficient.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/unittest/_torch/executor/test_mamba_cache_manager.py` around lines 1651
- 1684, Update _build_v2_hybrid_with_mamba_layer to accept a max_num_turns
parameter and pass it into BlockReuseConfig. Add MambaHybridCacheManagerV2 tests
covering multi-turn retention and eviction, including the five-turn case,
matching the existing KVCacheManagerV2 max_num_turns=2 coverage.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@tests/unittest/_torch/executor/test_mamba_cache_manager.py`:
- Around line 1651-1684: Update _build_v2_hybrid_with_mamba_layer to accept a
max_num_turns parameter and pass it into BlockReuseConfig. Add
MambaHybridCacheManagerV2 tests covering multi-turn retention and eviction,
including the five-turn case, matching the existing KVCacheManagerV2
max_num_turns=2 coverage.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: ba22a143-156c-4b0c-84da-c197c2383f09

📥 Commits

Reviewing files that changed from the base of the PR and between 59b867a and 8f8ad1a.

📒 Files selected for processing (2)

• docs/source/developer-guide/telemetry.md
• tests/unittest/_torch/executor/test_mamba_cache_manager.py

[tensorrt-cicd]

PR_Github #62940 [ run ] triggered by Bot. Commit: 8f8ad1a Link to invocation

[tensorrt-cicd]

PR_Github #62933 [ run ] completed with state ABORTED. Commit: 59b867a

Link to invocation

[jiaganc]

/bot run

[tensorrt-cicd]

PR_Github #63282 [ run ] triggered by Bot. Commit: 8f8ad1a Link to invocation

[tensorrt-cicd]

PR_Github #63282 [ run ] completed with state DISABLED
Pipeline is freezed and top-1 instance is under maintenance. For urgent request, contact Yiteng Niu

Link to invocation

[qiaoxj07]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #63319 [ run ] triggered by Bot. Commit: 8f8ad1a Link to invocation

[tensorrt-cicd]

PR_Github #63319 [ run ] completed with state FAILURE. Commit: 8f8ad1a
/LLM/main/L0_MergeRequest_PR pipeline #51313 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[qiaoxj07]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #63358 [ run ] triggered by Bot. Commit: 8f8ad1a Link to invocation

[tensorrt-cicd]

PR_Github #63358 [ run ] completed with state SUCCESS. Commit: 8f8ad1a
/LLM/main/L0_MergeRequest_PR pipeline #51345 completed with status: 'SUCCESS'

CI Report

Link to invocation

[yuanjingx87]

Approved on oss compliance perspective.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/unittest/_torch/executor/test_mamba_cache_manager.py`:
- Around line 2041-2089: Add a probe for request_b after its initial processing
and before request_c in
test_v2_hybrid_retains_configured_number_of_conversation_turns. Capture and
assert its prepopulated_prompt_len and Mamba state index match the expected
reused second-turn state, then free the probe resources before continuing the
existing eviction checks.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 7b34f034-b3e5-4aaa-8bb1-61da5f91761c

📥 Commits

Reviewing files that changed from the base of the PR and between 8f8ad1a and bc57ee4.

📒 Files selected for processing (9)

• docs/source/developer-guide/telemetry.md
• tensorrt_llm/_torch/pyexecutor/kv_cache_manager_v2.py
• tensorrt_llm/_torch/pyexecutor/mamba_cache_manager.py
• tensorrt_llm/llmapi/llm_args.py
• tensorrt_llm/usage/llm_args_golden_manifest.json
• tests/unittest/_torch/executor/test_kv_cache_manager_v2.py
• tests/unittest/_torch/executor/test_mamba_cache_manager.py
• tests/unittest/kv_cache_manager_v2_tests/test_kv_cache_stats_behavior.py
• tests/unittest/llmapi/test_llm_args.py

🚧 Files skipped from review as they are similar to previous changes (7)

• docs/source/developer-guide/telemetry.md
• tests/unittest/kv_cache_manager_v2_tests/test_kv_cache_stats_behavior.py
• tensorrt_llm/_torch/pyexecutor/mamba_cache_manager.py
• tests/unittest/_torch/executor/test_kv_cache_manager_v2.py
• tests/unittest/llmapi/test_llm_args.py
• tensorrt_llm/usage/llm_args_golden_manifest.json
• tensorrt_llm/_torch/pyexecutor/kv_cache_manager_v2.py

[jiaganc]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #63643 [ run ] triggered by Bot. Commit: e1576f5 Link to invocation

[jiaganc]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #63684 [ run ] triggered by Bot. Commit: 9bc296c Link to invocation

[tensorrt-cicd]

PR_Github #63643 [ run ] completed with state ABORTED. Commit: e1576f5

Link to invocation

[tensorrt-cicd]

PR_Github #63684 [ run ] completed with state FAILURE. Commit: 9bc296c
/LLM/main/L0_MergeRequest_PR pipeline #51636 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation