Skip to content

fix: accumulate ChatUsage across Anthropic streaming events - #2467

Open
waterWang wants to merge 1 commit into
agentscope-ai:mainfrom
waterWang:fix/anthropic-streaming-usage
Open

fix: accumulate ChatUsage across Anthropic streaming events#2467
waterWang wants to merge 1 commit into
agentscope-ai:mainfrom
waterWang:fix/anthropic-streaming-usage

Conversation

@waterWang

Copy link
Copy Markdown

Fixes #2094

Problem

When AnthropicChatModel is used with streaming enabled, every ChatResponse has usage == null. The Anthropic streaming protocol sends token usage in message_start (input_tokens) and message_delta (output_tokens) events, but these events have empty content blocks and are filtered out by the .filter(response -> !response.getContent().isEmpty()) in parseStreamEvents().

Fix

  1. parseStreamEvent(): Parse inputTokens from the message_start event's message.usage.input_tokens and create a ChatUsage object.

  2. parseStreamEvents(): Use a mutable accumulator array to merge usage across events:

    • message_start → sets inputTokens
    • message_delta → sets outputTokens
    • content_block_delta → attaches the accumulated usage when content is emitted

This ensures that every content-bearing event carries the complete token usage (input + output tokens), while the filter still correctly removes empty events.

@gemini-code-assist

Copy link
Copy Markdown

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@oss-maintainer oss-maintainer left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review — PR #2467

fix: accumulate ChatUsage across Anthropic streaming events

Fixes #2094

Summary

Fixes ChatUsage being null in streaming responses from AnthropicChatModel by accumulating token usage from message_start (input tokens) and message_delta (output tokens) events.

Analysis

The approach is correct:

  1. parseStreamEvent() already extracts usage from individual events
  2. parseStreamEvents() now uses a mutable ChatUsage[] accumulator to merge usage across events
  3. Content-bearing events get the accumulated usage attached before emission

The merge logic handles the case where message_start provides inputTokens first, then message_delta adds outputTokens, and subsequent content events carry the full accumulated usage.

Observations

  • ✅ Correctly identifies that message_start and message_delta have empty content blocks and are filtered out
  • ✅ Accumulator pattern is appropriate for the stateful nature of streaming usage
  • ✅ Fixes a real user-facing issue (no token usage tracking for Anthropic streaming)
  • CI failed — Spotless formatting in AnthropicResponseParser.java. Run mvn spotless:apply to fix.
  • ⚠️ Minor: the ChatUsage[] array as mutable accumulator works but could be replaced with AtomicReference<ChatUsage> for clarity in concurrent contexts (though flatMap here is sequential per-event, so the array is fine).

Required Action

@waterWang Please run mvn spotless:apply in the agentscope-extensions-model-anthropic module and push the fix.

Verdict

Request Changes — logic is correct, but Spotless formatting must be fixed before merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]:Anthropic streaming responses drop ChatUsage

2 participants