[None][fix] Add KV cache V2 recompute pause path

[yizhang-nv]

@coderabbitai summary

Description

This PR separates destructive recompute pause from ordinary KV cache V2 suspension in the Python scheduler/executor path. It adds a scheduler output field for recompute-paused requests with a default empty value so V1 scheduler construction remains unchanged, gates suspended-victim promotion on host-tier availability, and has PyExecutor terminate/replay recompute-paused requests separately from ordinary suspended requests.

Test Coverage

• python3 -m pytest -s tests/unittest/_torch/executor/test_kv_cache_v2_scheduler.py tests/unittest/_torch/executor/test_scheduler_serializable_output.py
• python3 -m pytest -s tests/unittest/pyexecutor/test_iter_stats_populate.py tests/unittest/_torch/executor/test_py_scheduler.py -k "paused or full_pipeline_output_structure"
• python3 -m pytest -s tests/unittest/_torch/executor/test_py_scheduler.py -k full_pipeline_output_structure
• git diff --check
• pre-commit hooks on commit

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[yizhang-nv]

/bot run

[tensorrt-cicd]

PR_Github #51334 [ run ] triggered by Bot. Commit: e9ba97c Link to invocation

[nvpohanh]

@lowsfer could you review this? thanks

[tensorrt-cicd]

PR_Github #51334 [ run ] completed with state FAILURE. Commit: e9ba97c
/LLM/main/L0_MergeRequest_PR pipeline #40746 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[yizhang-nv]

/bot run

[tensorrt-cicd]

PR_Github #51337 [ run ] triggered by Bot. Commit: 1e8ba4a Link to invocation

[tensorrt-cicd]

PR_Github #51337 [ run ] completed with state FAILURE. Commit: 1e8ba4a
/LLM/main/L0_MergeRequest_PR pipeline #40749 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[yizhang-nv]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #51492 [ run ] triggered by Bot. Commit: 1e8ba4a Link to invocation

[tensorrt-cicd]

PR_Github #51492 [ run ] completed with state SUCCESS. Commit: 1e8ba4a
/LLM/main/L0_MergeRequest_PR pipeline #40898 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[yizhang-nv]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #51796 [ run ] triggered by Bot. Commit: 1e8ba4a Link to invocation

[tensorrt-cicd]

PR_Github #51796 [ run ] completed with state SUCCESS. Commit: 1e8ba4a
/LLM/main/L0_MergeRequest_PR pipeline #41160 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[yizhang-nv]

closing in favor of #15252