Skip to content

Add extended prompt cache TTL setup option - #6515

Open
Sanskarzz wants to merge 1 commit into
stacklok:mainfrom
Sanskarzz:prompt-cache
Open

Add extended prompt cache TTL setup option#6515
Sanskarzz wants to merge 1 commit into
stacklok:mainfrom
Sanskarzz:prompt-cache

Conversation

@Sanskarzz

Copy link
Copy Markdown
Contributor

Summary

  • Claude Code users need a ToolHive-managed way to opt into the one-hour prompt-cache lifetime. Add a persisted --extended-ttl-cache option to thv llm setup and thv llm config set.
  • Apply the main, subagent, and legacy Claude Code cache settings, report unsupported clients and conflicting five-minute overrides, and remove managed settings when the option is explicitly disabled or ToolHive setup is torn down.

Fixes #6453

Type of change

  • Bug fix
  • New feature
  • Refactoring (no behavior change)
  • Dependency update
  • Documentation
  • Other (describe):

Test plan

  • Unit tests (task test)
  • E2E tests (task test-e2e)
  • Linting (task lint-fix)
  • Manual testing (describe below)
  • Build (task build)
  • CLI docs generation (task docs)

API Compatibility

  • This PR does not break the v1beta1 API, OR the api-break-allowed label is applied and the migration guidance is described above.

Changes

Area Change
LLM CLI and config Add and persist the extended TTL option, including explicit disable support.
Client setup Configure supported Claude Code cache settings and detect higher-precedence overrides.
Tests Cover persistence, setup reporting, conflict handling, cleanup, and the CLI workflow.
CLI docs Document the new setup and config flags.

Does this introduce a user-facing change?

Yes. Users can enable a persisted one-hour prompt-cache lifetime with --extended-ttl-cache; Claude Code is currently supported, and the setting can be cleared with --extended-ttl-cache=false.

Implementation plan

Approved implementation plan
  • Add a tri-state CLI option so omission preserves the saved value and explicit false clears it.
  • Persist the preference and apply declarative Claude Code settings with conflict and compatibility reporting.
  • Cover configuration, setup, cleanup, and CLI behavior, then regenerate the command documentation.

Special notes for reviewers

Claude Code is currently the only client with extended prompt-cache TTL support. Other detected clients are reported as unsupported without failing setup.

task test is currently blocked by a gotestfmt v2.5.0 parser panic (Empty package name encountered); task build succeeds.

@codecov

codecov Bot commented Sep 4, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 88.88889% with 19 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.32%. Comparing base (da85cdb) to head (4b47ee5).
⚠️ Report is 6 commits behind head on main.

Files with missing lines Patch % Lines
pkg/client/llm_gateway.go 85.47% 17 Missing ⚠️
pkg/llm/setup.go 96.07% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #6515      +/-   ##
==========================================
+ Coverage   78.25%   78.32%   +0.07%     
==========================================
  Files         770      770              
  Lines       75389    75632     +243     
==========================================
+ Hits        58996    59240     +244     
+ Misses      16388    16387       -1     
  Partials        5        5              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add --extended-ttl-cache to thv llm setup to opt clients into the one-hour prompt cache lifetime

1 participant