Repository navigation
Replies: 3 comments
|
For reference, I implemented this direction in my fork: Alb11747#6. It's not proposed as an upstream PR. It's a Claude instance setting, off by default. A compaction counts as cold only when both of these hold:
That compaction's process opens with Testing so far:
Real-thread results (Oct 11, Opus 5.5): all threads were idle well past an hour, through a gateway that leaves markerless requests alone. Per-request usage:
Still open:
|
Possible related finding: weekly and five-hour limits may weight cache reads differentlyA separate observational analysis of Claude subscription usage suggests that cache reads carry much less weight on the weekly meter than on the five-hour meter. A possible explanation is that reads are free or nearly free weekly, while still counting against the five-hour limit. The available evidence does not establish that weekly reads are exactly free. This is a tentative finding that needs more research, not a documented provider rule. The analysis compared request token buckets with subsequent usage-meter readings within stable reset windows, treating weekly and five-hour usage separately. It covered approximately five days, with five partial weekly histories and 30 usable five-hour histories. Exploratory Opus estimates: the table reports percentage points of the respective usage meter per million tokens, with nominal 95% intervals clustered by reset window. These are observational fitted rates, not provider prices or guaranteed allowances.
For scale, the read estimates correspond to roughly 49 million cache-read tokens per one percentage point weekly, versus 2.0 million per one percentage point on the five-hour meter. Those compare fractions of two different allowances; they are not a token-price ratio. Why this still needs research: another, simpler fit put the read weight at 0.0138 weekly (95% reset-window bootstrap interval 0–0.038) and 0.0527 for five hours (0.037–0.065), relative to an assumed, rather than measured, ordinary-input weight of 1. Its write/output weights and some model factors were also assumed. That broader weekly interval includes zero. Changing aggregation intervals also moved the weekly estimate substantially. The detailed bucket fit above generalized poorly for five-hour usage: held-out-account error was 10.16 percentage points, versus 4.49 for the simpler fit. All intervals are conditional on the modeling assumptions and omit uncertainty from missing traffic, outside use, and an incorrect model. Coarse percentage readings and workload confounding, including a dominant cache-read-heavy workload, further limit confidence. Ordinary-input and five-minute cache-write rates were not reliably identified, so these numbers cannot establish a one-hour-write premium over ordinary input or shorter-lived caching. For this proposal, the distinction matters: cold compaction avoids cache writes, while the finding above concerns cache reads. It does not establish how much subscription quota is saved by replacing a cold cache write with ordinary input. API-equivalent savings and measured subscription-quota savings remain separate claims. The pattern is consistent with preserving caching for warm history, but it does not settle the cold-compaction tradeoff. Useful follow-up would be controlled warm-cache reads with minimal output on an otherwise idle account, plus paired cold-compaction tests with caching enabled and disabled. Each should span several meter ticks, include before/after readings after usage has settled, and be repeated across independent reset windows. |
|
This matches what I measured running Claude Code on the hour TTL. In one session of 199 steps, nine resumes after an idle gap and one model switch made up about 76% of all tokens written to the cache. I also tried the opposite fix, an empty ping every 50 minutes to keep the cache warm. Priced over 102 runs it cost $105.83 and avoided $105.56 of rewrites, because about half the time the user came back inside the hour anyway. Skipping the write on a cold compaction is the better lever. Two suggestions. First, derive the idle threshold from the TTL in force rather than fixing it at 70 minutes. With a TTL of five minutes any longer gap is cold, and a write on the hour TTL costs 2x base input against 1.25x on the short one, so the waste per cold compaction differs. Second, a paired run would settle the counterfactual. Fork one idle thread, compact one copy with the flag and one without, and compare the billed write tokens. The measurements sit in a reference architecture I maintain, under the pattern on position and price. Drafted with Claude Code and reviewed by me before posting. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Status: implemented in my fork
This is implemented in my fork, as an opt-in Claude instance setting, Skip cache writes on cold compaction (off by default). It applies when the thread has been idle for at least 70 minutes and its Claude process has already been released. The compaction starts with prompt caching disabled; the next productive turn reopens Claude with normal caching. Existing live processes, including those with background work, are left alone.
The implementation also sends
X-CPA-Prompt-Cache-Mode: explicitfor CLIProxyAPI builds that honor that header, to prevent the proxy from adding cache markers back. Other gateways must likewise preserve markerless requests. Real-thread tests through a compatible gateway recorded zero cache-write tokens with the setting on, while a separate default compaction wrote its history to the 1h cache (see the follow-up below). Dollar savings remain counterfactual API-price calculations: the runs used different threads, not paired copies of identical history. Pro/Max usage-limit savings are unknown.Implementation commit:
6837cc9. Source: setting, Claude adapter, and fork notes.Problem
When a large Claude thread is resumed after its prompt cache has expired, the Compact path (
/compactbefore send, or "Resume with less context") can cache-write the whole old history while summarizing it. The next turn then caches the new, smaller context. The old history is about to be dropped, so in this case most of that write buys nothing: one measured cold compaction spent about half its cost writing history that was discarded.T3 can address this on its own side. In the usual cold case it already starts a fresh Claude CLI process for that compaction turn, and it already passes an
envper query. The proposal is narrow: for an explicitly cold compaction only, open that one compaction process with prompt caching disabled, then reopen normally for the next productive turn. Warm compaction and normal turns would keep caching.This is distinct from the warm-window timing ideas (#18011, #14233, draft #16855) and from compact-on-send defaults (#17467, #17170). Those decide when compaction happens; this is about how much the cold compaction itself costs. I'd like a read on whether this direction and scope would be acceptable upstream; the opt-in implementation is already in my fork.
Evidence
Measured cold compaction (Opus 5.5, Claude Code via T3, Oct 9, about 112 min idle, so past the 1h cache):
/compactsummarizerThe compaction boundary reported ~448k tokens before and ~10.9k after. The summarizer request was matched to the compaction boundary by timing (completion within 15 ms, duration ~42.6 s vs ~42.8 s), the same session, and the following turn's usage. No request ID is persisted in the transcript, so this is a strong timing/sequence match, not an explicit ID join.
At published API rates (Opus 5.5: input $4, 1h write $8, cache read $0.20, output $20 per MTok), that compaction costs about $3.01. With the same token counts:
11.6k warm prefix: **$1.55** (saves ~$1.46, ~49%).These are arithmetic estimates, not a paired live experiment. Removing cache markers could change the request in other ways. The replacement-context write on the next turn (36k) is unaffected.
For scale, if a cold compaction would otherwise write 800,000 tokens into the 1h cache, that input costs $6.40 at the rates above, versus $3.20 as ordinary uncached input: a potential $3.20 saving on that compaction, before any difference from an already-warm prefix. Summary output costs extra either way. This assumes all 800k tokens would otherwise receive a fresh 1h cache write; it is an extrapolation, not a measured saving or a claim about every 800k-context thread.
Runtime behavior (Claude Code 2.1.296, Agent SDK 0.3.276, read from the shipped minified runtime):
skipCacheWrite: true. That only moves the tail cache breakpoint back one message. Earlier history keeps its ordinary ephemeralcache_controlmarker. It is not a read-only-cache mode, so with an expired cache the old prefix is written again./compactstill wrote ~2.2k tokens on top of ~26–27k cached reads. That's cheap when warm; the problem is cold history.enablePromptCachingitself. The request builder falls back to the model-level check, which returns false whenDISABLE_PROMPT_CACHING(orDISABLE_PROMPT_CACHING_<MODEL>) is set in the process env. So a CLI process started with that env sends the compaction with no cache markers. Confirmed at the CLI's outbound request in the follow-up test below.CLAUDE_CODE_COLD_COMPACTexists, but CLAUDE_CODE_COLD_COMPACT is unreachable in the local CLI: the auto-window gate returns before it, and an explicit window routes through the reactive path, which never reads it (2.1.270–2.1.272) anthropics/claude-code#95694 reports it is unreachable on the tested local paths, so it isn't a dependable switch.Follow-up test: scoped env flag (synthetic Haiku 5.5, Claude Code 2.1.296, SDK 0.3.276, 4 requests, warm seed). Sequence: seed normally, close, resume
/compactwithDISABLE_PROMPT_CACHING=1, close, resume normally for two productive turns. Usage is per request, as returned by the API through the gateway:/compactwith flagcache_controlmarker at the CLI's outbound request. The flag doesn't make the summary uncacheable: the ~2.8k-char summary and the rest of the replacement context were byte-identical across both later requests, were written on the next turn, and were read on the one after. Prompt caching stores input prefixes, so the summary enters the cache when the next request sends it as input, not as output of the summarizer call.cache_controlto markerless requests from callers they don't recognize as native Claude Code. A source reading of the one I use shows that behavior, but I haven't confirmed that's what happened here. So this test does not show an uncached compaction end to end, and it measures no savings. The seed was warm, so it isn't a cold paired test either.Later real-thread verification (Oct 11; raw records rechecked Oct 15): the implementation follow-up contains four real Opus 5.5 runs through a gateway honoring the opt-out. With the setting on, the compaction requests reported 141,605, 328,592 and 100,838 ordinary input tokens, each with 0 cache reads and 0 cache writes. The setting-off control reported 121,581 cache-write tokens, all 1h, plus 2 ordinary input, 0 reads and 3,355 output. Gateway usage logs corroborate those exact figures; compaction boundaries and subsequent usage corroborate the request sequence where available. In the first two setting-on threads, the next productive turn wrote the replacement context and the following turn read it, confirming normal caching resumed.
At the listed API rates, the control is $1.039756; the same tokens fully uncached would be $0.553432, a calculated $0.486324 (46.8%) saving. This is real token-accounting evidence for the mechanism, not a paired dollar-saving measurement or a subscription-quota result. The 800k example above remains an extrapolation. The current Anthropic pricing table was rechecked: Opus 5.5 standard input is $4/MTok and 1h cache writes are $8/MTok.
Implementation in my fork
Original upstream inspection at
273835b:ProviderSessionManager.ts:DEFAULT_IDLE_TIMEOUT_MS = 30 min. Idle Claude sessions are released well before the 70-min resume offer (CLAUDE_RESUME_COMPACTION_MINUTESinContextWindowMeter.logic.ts). So a cold compaction normally already opens a new CLI process withresume.ClaudeAdapterV2.ts:compactThreadisstartTurnwith text/compact.makeClaudeQueryOptionsalready forwardsenv. The live process is reused only whenqueryPolicyKey/selectionKeymatch; otherwise it is replaced, and replacement is refused while background work runs.The fork implements this as follows (off by default):
DISABLE_PROMPT_CACHING=1and a distinct query-policy key. It also addsX-CPA-Prompt-Cache-Mode: explicitto the custom headers for compatible CLIProxyAPI builds.The env flag only controls what the CLI sends. Savings also require every gateway or proxy on the path to forward a markerless request without adding
cache_control. With a proxy that injects defaults, the compaction is cached as before.This is not a request to disable caching globally. That would make warm compaction and every normal turn more expensive.
Tradeoff: forks and branches
Transcript-based forks keep working. T3's fork uses the standalone SDK
forkSession(sessionId, { upToMessageId }), which copies the transcript. In a synthetic Haiku 5.5 test, forking an already compacted session at its last precompaction assistant message restored the full old history and all earlier thinking blocks (27,758 cached read, 0 write). The native streamingresumeSessionAtpath behaves differently and is not what T3 uses. I tested the SDK call, not the UI fork button.What is lost is only the warm-cache benefit for a branch made from the precompaction history within the cache TTL, on the same account. Per 100k history tokens on Opus 5.5 with a 1h TTL:
Break-even is about a 51% chance of such a branch (about 21% with a 5m TTL). Branches made after expiry would not have benefited from the write anyway.
Not verified
Environment
Claude Code 2.1.295/2.1.296,
@anthropic-ai/claude-agent-sdk0.3.276, T3main@273835b, Windows 11.Related: #13988 / #18011, #14233, #16855, #17467, #17170, #16829 (PR watch waking cold threads), #17423 (revert across compaction, separate from fork), anthropics/claude-code#95694, anthropics/claude-code#66115, anthropics/claude-code#99624, anthropics/claude-code#70459.
All reactions