[None][fix] BREAKING Block reuse policy rename and add more tests

[jiaganc]

Dev Engineer Review

• Renamed BlockReuseConfig.block_reuse_policy to policy across API validation, KV-cache managers, Mamba consumers, telemetry, documentation, tests, and the golden manifest.
• Preserved valid policy values, defaults, and behavior.
• Updated max_num_turns manifest coverage.
• Added retained-turn, snapshot-reuse, and eviction coverage for hybrid Mamba V2.
• No test-list files changed.
• Pre-commit, DCO, whitespace, rebase, and range-diff checks passed.
• Reported test result: 740 tests passed with 69 expected skips.

QA Engineer Review

Modified test code includes:

• KV-cache manager V2 configuration tests.
• KV-cache statistics configuration setup.
• LLM argument validation and warning tests.
• Hybrid Mamba V2 helpers and the per-conversation retained-turn regression test.

The modified tests are unit tests and are not listed in tests/integration/test_lists/ for CI or manual QA coverage.

Verdict: sufficient.

Description

Follow-up to #16883. That PR documented the nested configuration path as
kv_cache_config.block_reuse_config.policy, but the merged implementation retained the
redundant block_reuse_policy field name inside BlockReuseConfig.

This change:

• renames BlockReuseConfig.block_reuse_policy to BlockReuseConfig.policy;
• updates KV-cache and Mamba consumers, validation, warnings, documentation, and telemetry metadata;
• provides a targeted migration error for the old field name;
• regenerates llm_args_golden_manifest.json from the corrected schema; and
• adds retained-turn coverage for KV cache manager V2 and hybrid Mamba V2, including exact recurrent-state snapshot reuse and eviction.

The user-facing configuration is now the shape declared by #16883:

kv_cache_config:
  block_reuse_config:
    policy: per_conversation
    max_num_turns: 5

This remains an intentional breaking change: configurations that use
kv_cache_config.block_reuse_config.block_reuse_policy are rejected rather than accepted as a
deprecated alias. Validation now reports that the field was renamed to
kv_cache_config.block_reuse_config.policy instead of returning only the generic Pydantic
extra-field error.

Test Coverage

• Pre-commit and DCO checks passed for the follow-up commits.
• B200 changed-file validation on the pre-rebase head: 740 passed, 69 expected skips, zero failures or errors.
• The formerly failing hybrid Mamba retained-turn test passed independently on B200.
• Rebased onto upstream/main at 91fb4433c5; git range-diff confirmed all six replayed patches are unchanged.
• git diff --check upstream/main...HEAD passed.

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[jiaganc]

/bot run

[coderabbitai]

Walkthrough

The PR renames BlockReuseConfig.block_reuse_policy to policy. Runtime cache managers, manifests, telemetry, documentation, and tests now use the renamed field. Validation rejects configurations that still use the legacy key.

Changes

Block reuse policy field rename

Layer / File(s)	Summary
Configuration contract and metadata tensorrt_llm/llmapi/llm_args.py, tensorrt_llm/usage/llm_args_golden_manifest.json, docs/source/developer-guide/telemetry.md, docs/source/features/kvcache.md, tests/unittest/llmapi/test_llm_args.py	BlockReuseConfig renames block_reuse_policy to policy. Validation rejects the legacy key. Manifests, telemetry, documentation, and API tests use the renamed field.
Runtime policy wiring tensorrt_llm/_torch/pyexecutor/kv_cache_manager_v2.py, tensorrt_llm/_torch/pyexecutor/mamba_cache_manager.py	Cache managers read and update block-reuse policy through policy.
Cache-manager validation tests/unittest/_torch/executor/test_kv_cache_manager_v2.py, tests/unittest/_torch/executor/test_mamba_cache_manager.py, tests/unittest/kv_cache_manager_v2_tests/test_kv_cache_stats_behavior.py	Tests construct configurations with policy. Mamba tests cover conversation snapshot reuse and turn-limit eviction.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

• NVIDIA/TensorRT-LLM#17308: Both PRs modify KVCacheManagerV2 block-reuse handling.

Suggested reviewers: bowenfu, qijune, tongyuantongyu

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 12.50% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly identifies the breaking block reuse policy rename and related test additions.
Description check	✅ Passed	The description explains the change, rationale, breaking API impact, test coverage, and checklist status.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[tensorrt-cicd]

PR_Github #63923 [ run ] triggered by Bot. Commit: 33e2798 Link to invocation

[brnguyen2]

Approving — the comments below are optional touch-ups, not blockers.

Rename verified complete — I grepped for block_reuse_config.block_reuse_policy across .py/.md/.json/.yaml; nothing stale remains, and the manifest reorder is just alphabetical-by-path from regeneration.

One process point worth a sentence in the PR description: BlockReuseConfig inherits StrictBaseModel (extra = "forbid"), so any user or checked-in YAML still saying block_reuse_policy: now fails with a bare pydantic "extra inputs are not permitted" — no hint that the field was renamed. That's defensible for a status="prototype" field with the api-breaking label, and the immediately adjacent mamba_state_cache_interval deprecated-alias precedent argues the other way, so it's a deliberate call either way — just make it explicit rather than implicit.

Separately: docs/source/features/kvcache.md discusses the per_conversation policy (line 91) but never shows its config path, so after this PR the only doc naming the field is the generated telemetry table. The YAML snippet from the PR description would be a useful two-line addition there.

[brnguyen2]

This assertion can pass for the wrong reason: with max_tokens=512 / tokens_per_block=32 the pool is 16 blocks, so request_a's state could be gone from ordinary capacity pressure rather than from the max_num_turns=2 retention limit. Add a paired survivor check right after — re-probe request_b's prefix (new request, fresh conversation id) and assert it still resolves to request_b_state_index. That pins the behavior to "turn A aged out, turn B retained", which is what the test name claims.

[jiaganc]

Fixed in bf1225d. After confirming turn A is no longer reusable, the test now probes turn B with a fresh request/conversation ID and verifies both the reused prompt length and the original Mamba state index. This distinguishes max_num_turns eviction from ordinary capacity pressure.

[brnguyen2]

Worth a comment on why the probes use distinct conversation ids (conv-2/conv-3/conv-4) while the turns share conv-1 — presumably so a probe doesn't itself count as a retained turn and shift the eviction accounting. That's the non-obvious part of this test and the next reader will otherwise assume it's a copy-paste slip.

[jiaganc]

Fixed in bf1225d. Added a comment explaining that each probe uses a fresh conversation ID so it queries the shared prefix cache without participating in or changing conv-1 retained-turn accounting.

[brnguyen2]

Since StrictBaseModel forbids extras, existing block_reuse_policy: YAML now dies with an unhelpful pydantic message. If you don't want a full deprecated alias (as mamba_state_cache_interval has), a model validator on BlockReuseConfig that catches the old key and raises "renamed to policy" costs ~4 lines and turns a confusing failure into a self-explanatory one. Your call given the prototype status — but the migration path should be a deliberate decision, not a side effect.

[jiaganc]

Fixed in bf1225d while keeping the prototype rename intentionally breaking. BlockReuseConfig now has a before validator that detects the old key and reports the full old and new configuration paths. Added a nested KvCacheConfig validation test for the migration message, and documented the new YAML path in kvcache.md.

[QiJune]

LGTM

[tensorrt-cicd]

PR_Github #63923 [ run ] completed with state SUCCESS. Commit: 33e2798
/LLM/main/L0_MergeRequest_PR pipeline #51861 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[coderabbitai]

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

[jiaganc]

/bot run

[tensorrt-cicd]

PR_Github #64026 [ run ] triggered by Bot. Commit: bf1225d Link to invocation

[tensorrt-cicd]

PR_Github #64026 [ run ] completed with state SUCCESS. Commit: bf1225d
/LLM/main/L0_MergeRequest_PR pipeline #51955 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[jiaganc]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #64086 [ run ] triggered by Bot. Commit: bf1225d Link to invocation