fix(anthropic-oauth): clamp fixed thinking budgets under max_tokens - #51
Open
rezaho wants to merge 2 commits into
Open
fix(anthropic-oauth): clamp fixed thinking budgets under max_tokens#51rezaho wants to merge 2 commits into
rezaho wants to merge 2 commits into
Conversation
…rows A prompt-cache entry is only worth writing if the NEXT request can read it, which requires its hashed prefix to be bytes that request still contains. When a caller appends per-request content after the durable conversation — a clock, a budget figure, anything derived from "now" — the unconditional last-message breakpoint lands on exactly the row the next request cannot reproduce, so every request writes a fresh entry and reads none. `CACHE_EXEMPT_KEY` is a neutral per-item key on the caller's own message dict (the `defer_loading` shape) marking a row as per-request. The marker then lands on the last durable row, which the next request contains by construction since durable rows only ever grow by append. Counted from the end and stopping at the first unmarked row: the exemption is about position, so a marked row with durable rows after it is not a tail. An all-exempt list writes no marker at all. Measured on Bedrock/Opus 5, single-step turns with tools present: marker on the volatile row: turn 1 write=8211 -> turn 2 write=8226 read=0 marker on the last durable row: turn 1 write=8425 -> turn 2 write=15 read=8425 With no exempt row the payload is byte-identical to before, so no existing caller changes behaviour. Twinned on the OAuth leg, which is not a subclass and has its own payload builder. The key never reaches the wire — both builders rebuild rows with only the wire-legal fields.
Parity with AnthropicAdapter._thinking_payload: budget_tokens >= max_tokens is an illegal payload (live 400) for every non-adaptive model, and a thinking flag without a usable budget put budget_tokens=None/0 on the wire. Clamp with the twin's headroom rules, drop thinking when nothing legal fits, warn either way. The shape that surfaced it: a background model built at max_tokens=4096 with the default 8192 thinking budget 400s on every call once its model id resolves to a non-adaptive catalog model.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The OAuth builder sent a configured thinking budget verbatim, so any budget >= max_tokens - 1 was rejected by the API (the api-key twin has clamped this at anthropic.py:325-337 since the beginning; the OAuth path drifted).
Now: the budget is clamped to max_tokens minus 1024 headroom, with a warning when clamping occurs. If the clamped value falls below the 1024 API minimum, thinking is disabled for the call instead of erroring. A non-int or non-positive budget also disables thinking with a warning rather than sending a malformed payload.
Three tests in tests/models/test_oauth_claude5_payload_shape.py cover clamp, can't-fit-disables, and enable-without-budget.