Skip to content

fix(anthropic): surface streaming stop_reason and message_delta input_tokens - #2236

Open
shlim33 wants to merge 3 commits into
0xPlaygrounds:mainfrom
shlim33:xyren/anthropic-stop-reason
Open

fix(anthropic): surface streaming stop_reason and message_delta input_tokens#2236
shlim33 wants to merge 3 commits into
0xPlaygrounds:mainfrom
shlim33:xyren/anthropic-stop-reason

Conversation

@shlim33

@shlim33 shlim33 commented Jul 31, 2026

Copy link
Copy Markdown

Fixes #2235

The Anthropic streaming provider parses two pieces of metadata off the wire and
then drops them before the consumer can see them. This surfaces both, without
changing behaviour against Anthropic proper.

1. Surface stop_reason on the streaming response

message_delta.delta.stop_reason was read only to decide when to break out of
the SSE loop; the value was discarded. StreamingCompletionResponse carried
usage and nothing else, so a max_tokens truncation was indistinguishable
from a normal end_turn.

pub struct StreamingCompletionResponse {
    pub usage: PartialUsage,
    #[serde(default)]
    pub stop_reason: Option<String>,   // added
}

The value is captured in the same block that already builds final_usage, and
filled in at the FinalResponse construction site. The two test-fixture
construction sites get None.

This matters for consumers that must refuse to act on a truncated response — a
tool call cut mid-arguments has to be declined, and stop_reason is the only
signal that says so.

2. Prefer message_delta.usage.input_tokens when the provider sends it

input_tokens was captured at message_start only, and message_delta's
usage.input_tokens was ignored even though PartialUsage already models the
field. Anthropic reports input usage on message_start, so this worked against
the real API — but Anthropic-compatible gateways do not all behave that way.
OpenRouter's Anthropic Messages endpoint sends:

message_start  {"input_tokens": 0, "output_tokens": 0, ...}
message_delta  {"input_tokens": 9, "output_tokens": 4, ...}

which yielded a silent Usage { input_tokens: 0 } — worse than a missing value,
because a consumer sizing its context window from input tokens sees zero and
never acts.

input_tokens: usage
    .input_tokens
    .filter(|v| *v > 0)
    .or_else(|| usize::try_from(input_tokens).ok()),

Behaviour against Anthropic proper is unchanged: PartialUsage.input_tokens
is #[serde(default)] Option<usize>, and Anthropic's message_delta payload
does not carry the field, so the expression always falls through to the
message_start value there.

Tests

streaming_metadata_from_message_delta — a cassette-backed regression covering
both changes in one recorded interaction: message_start reports
input_tokens: 0 while message_delta carries stop_reason: "max_tokens" and
input_tokens: 9.

I verified the test is load-bearing rather than incidentally green: reverting
the input_tokens change alone turns it red with left: Some(0), right: Some(9),
and reverting the stop_reason change removes the field the test reads.

cargo test -p rig --test anthropic → 109 passed / 0 failed.
cargo clippy -p rig-core --all-targets → clean.

Notes

  • The OpenAI-compatible side has the same gap for finish reasons
    (CompatibleFinishReason { ToolCalls, Other } is pub(crate) and length is
    lost at providers/openai/completion/streaming.rs). I left it out to keep this
    PR focused on the provider where a typed stop_reason already exists on the
    response object. Happy to open a separate issue for it if that is wanted.
  • The two changes are in adjacent lines of the same function and share a root
    cause — the final response not carrying what message_delta reported. If you
    would rather review them separately I can split the PR.

shlim33 added 3 commits July 31, 2026 14:14
The streaming API parses message_delta.stop_reason but drops it, so
consumers cannot distinguish a max_tokens truncation from a normal stop.
Carry it through to StreamingCompletionResponse alongside usage.
…ider sends it

Anthropic reports input usage on message_start, but Anthropic-compatible
gateways (e.g. OpenRouter) send input_tokens: 0 there and the real count on
message_delta. Reading message_start alone silently yields a zero prompt
count against those providers, which breaks any consumer that sizes its
context from input tokens. Fall back to message_start when the delta does
not carry one, so behaviour against Anthropic proper is unchanged.
…data

Covers both dropped fields in one recorded interaction: message_start
reports input_tokens: 0 while message_delta carries stop_reason
"max_tokens" and input_tokens: 9.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(anthropic): streaming drops stop_reason and message_delta input_tokens

1 participant