fix(anthropic): surface streaming stop_reason and message_delta input_tokens - #2236
Open
shlim33 wants to merge 3 commits into
Open
fix(anthropic): surface streaming stop_reason and message_delta input_tokens#2236shlim33 wants to merge 3 commits into
shlim33 wants to merge 3 commits into
Conversation
The streaming API parses message_delta.stop_reason but drops it, so consumers cannot distinguish a max_tokens truncation from a normal stop. Carry it through to StreamingCompletionResponse alongside usage.
…ider sends it Anthropic reports input usage on message_start, but Anthropic-compatible gateways (e.g. OpenRouter) send input_tokens: 0 there and the real count on message_delta. Reading message_start alone silently yields a zero prompt count against those providers, which breaks any consumer that sizes its context from input tokens. Fall back to message_start when the delta does not carry one, so behaviour against Anthropic proper is unchanged.
…data Covers both dropped fields in one recorded interaction: message_start reports input_tokens: 0 while message_delta carries stop_reason "max_tokens" and input_tokens: 9.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2235
The Anthropic streaming provider parses two pieces of metadata off the wire and
then drops them before the consumer can see them. This surfaces both, without
changing behaviour against Anthropic proper.
1. Surface
stop_reasonon the streaming responsemessage_delta.delta.stop_reasonwas read only to decide when to break out ofthe SSE loop; the value was discarded.
StreamingCompletionResponsecarriedusageand nothing else, so amax_tokenstruncation was indistinguishablefrom a normal
end_turn.The value is captured in the same block that already builds
final_usage, andfilled in at the
FinalResponseconstruction site. The two test-fixtureconstruction sites get
None.This matters for consumers that must refuse to act on a truncated response — a
tool call cut mid-arguments has to be declined, and
stop_reasonis the onlysignal that says so.
2. Prefer
message_delta.usage.input_tokenswhen the provider sends itinput_tokenswas captured atmessage_startonly, andmessage_delta'susage.input_tokenswas ignored even thoughPartialUsagealready models thefield. Anthropic reports input usage on
message_start, so this worked againstthe real API — but Anthropic-compatible gateways do not all behave that way.
OpenRouter's Anthropic Messages endpoint sends:
which yielded a silent
Usage { input_tokens: 0 }— worse than a missing value,because a consumer sizing its context window from input tokens sees zero and
never acts.
Behaviour against Anthropic proper is unchanged:
PartialUsage.input_tokensis
#[serde(default)] Option<usize>, and Anthropic'smessage_deltapayloaddoes not carry the field, so the expression always falls through to the
message_startvalue there.Tests
streaming_metadata_from_message_delta— a cassette-backed regression coveringboth changes in one recorded interaction:
message_startreportsinput_tokens: 0whilemessage_deltacarriesstop_reason: "max_tokens"andinput_tokens: 9.I verified the test is load-bearing rather than incidentally green: reverting
the
input_tokenschange alone turns it red withleft: Some(0), right: Some(9),and reverting the
stop_reasonchange removes the field the test reads.cargo test -p rig --test anthropic→ 109 passed / 0 failed.cargo clippy -p rig-core --all-targets→ clean.Notes
(
CompatibleFinishReason { ToolCalls, Other }ispub(crate)andlengthislost at
providers/openai/completion/streaming.rs). I left it out to keep thisPR focused on the provider where a typed
stop_reasonalready exists on theresponse object. Happy to open a separate issue for it if that is wanted.
cause — the final response not carrying what
message_deltareported. If youwould rather review them separately I can split the PR.