feat(provider): add anthropic chat format for streaming#27
Merged
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
✅ Files skipped from review due to trivial changes (1)
📝 WalkthroughWalkthroughImplemented a stateful hub→Anthropic streaming bridge that incrementally converts OpenAI chat SSE chunks into Anthropic stream events, tightens hub validation, propagates aggregated usage including prompt cache tokens, and changed non‑stream native usage extraction to be format‑driven via Changes
Sequence Diagram(s)sequenceDiagram
participant Client
participant Gateway
participant Hub as OpenAI Hub (SSE)
participant Bridge as AnthropicBridgeState
participant ClientStream as Anthropic Client Stream
Client->>Gateway: messages(stream: true)
Gateway->>Hub: POST /v1/chat/completions (SSE)
Hub-->>Gateway: chat.completion.chunk SSE events
loop per chunk
Gateway->>Bridge: process OpenAI chunk
alt first relevant chunk
Bridge->>ClientStream: MessageStart (usage incl. cache fields)
end
alt text delta
Bridge->>ClientStream: ContentBlockStart (if needed)
Bridge->>ClientStream: ContentBlockDelta (TextDelta)
end
alt tool/input delta
Bridge->>ClientStream: ContentBlockStart (ToolUse)
Bridge->>ClientStream: ContentBlockDelta (InputJsonDelta)
end
alt block boundary
Bridge->>ClientStream: ContentBlockStop
end
end
Gateway->>Bridge: stream_end_events()
Bridge->>ClientStream: finalize ContentBlockStop (if open)
Bridge->>ClientStream: MessageDelta (aggregated usage)
Bridge->>ClientStream: MessageStop
ClientStream-->>Client: Anthropic stream events
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 2 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
Pull request overview
This PR enables hub-based streaming for the AnthropicMessagesFormat by bridging OpenAI-style chat.completion.chunk SSE chunks into Anthropic streaming events, and updates tests/docs to reflect the new streaming behavior.
Changes:
- Implement hub streaming bridge logic for
AnthropicMessagesFormatusing a state machine that emits Anthropic stream event lifecycles (message/content blocks) and stop reasons. - Update gateway integration test to validate end-to-end hub SSE streaming → Anthropic stream events and final usage delivery.
- Update internal documentation to remove the previous limitation note about Anthropic hub streaming being unsupported.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| src/gateway/gateway.rs | Reworks the messages streaming test to exercise hub SSE streaming and assert Anthropic stream events + usage. |
| src/gateway/formats/anthropic_messages.rs | Adds a bridge state + state machine to convert hub ChatCompletionChunk streams into AnthropicStreamEvents (text + tool_use). |
| docs/internals/llm-gateway.md | Removes outdated statement that AnthropicMessagesFormat rejects non-native hub streaming. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
src/gateway/formats/anthropic_messages.rs (1)
833-840: Avoid duplicating finish-reason mapping logic.
openai_finish_reason_to_anthropic_streammirrorsopenai_finish_reason_to_anthropic; centralizing this mapping will prevent drift.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/gateway/formats/anthropic_messages.rs` around lines 833 - 840, The mapping logic in openai_finish_reason_to_anthropic_stream duplicates openai_finish_reason_to_anthropic; refactor by centralizing the mapping into a single function (e.g., keep openai_finish_reason_to_anthropic as the canonical mapper) and have openai_finish_reason_to_anthropic_stream call that function and convert/output the result as needed, updating any callers to use the centralized mapper to avoid drift between openai_finish_reason_to_anthropic and openai_finish_reason_to_anthropic_stream.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@src/gateway/formats/anthropic_messages.rs`:
- Around line 27-37: Add a rustdoc comment (///) above the public struct
AnthropicBridgeState that briefly explains its role tracking streaming
conversion state for Anthropic messages—e.g., that it records whether a message
has started, the current block index/type/open state, stop reason, token counts,
and tool->block mappings during incremental parsing/streaming. Keep the comment
short (one or two sentences) and mention that the struct is used internally to
manage streaming message assembly and token accounting.
- Around line 728-775: On the first tool delta path (inside the loop over
choice.delta.tool_calls) validate the tool_call.type before emitting a ToolUse
block: check tool_call.type.as_deref() is present and equals the expected
tool-call kind (reject/return GatewayError::Bridge for unsupported/missing
types), then only proceed to call close_current_block, insert into
state.tool_block_map, set current block state and push the
AnthropicStreamEvent::ContentBlockStart with AnthropicContentBlock::ToolUse;
update the code around where tool_id, function and tool_name are read (the
tool_call branch that calls close_current_block and events.push) to perform this
type check first and fail fast for non-tool types.
---
Nitpick comments:
In `@src/gateway/formats/anthropic_messages.rs`:
- Around line 833-840: The mapping logic in
openai_finish_reason_to_anthropic_stream duplicates
openai_finish_reason_to_anthropic; refactor by centralizing the mapping into a
single function (e.g., keep openai_finish_reason_to_anthropic as the canonical
mapper) and have openai_finish_reason_to_anthropic_stream call that function and
convert/output the result as needed, updating any callers to use the centralized
mapper to avoid drift between openai_finish_reason_to_anthropic and
openai_finish_reason_to_anthropic_stream.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: abe234f8-5e62-4812-8097-c0671f003f05
📒 Files selected for processing (3)
docs/internals/llm-gateway.mdsrc/gateway/formats/anthropic_messages.rssrc/gateway/gateway.rs
💤 Files with no reviewable changes (1)
- docs/internals/llm-gateway.md
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@src/gateway/formats/anthropic_messages.rs`:
- Around line 69-73: The current check only rejects top-level req.cache_control
but still allows per-message cache_control which later gets dropped during
Anthropic→OpenAI conversion; update the validation (where req is inspected) to
also iterate req.messages and return Err(GatewayError::Bridge(...)) if any
message.cache_control.is_some() for non-system roles (i.e., reject per-block
cache_control on user/assistant messages), referencing req.messages,
message.cache_control and the BridgeContext conversion path so the request fails
fast instead of silently dropping caching directives.
- Around line 652-690: The delta handler recomputes input/total tokens from
sparse DeltaUsage, which zeroes omitted cached-token fields and undercounts when
merged; update anthropic_delta_usage_to_common_usage (or adjust
update_native_usage_from_event) so it doesn't rebuild totals from missing
fields: only sum input_tokens/cache_* when those Option values are Some (use
Option::zip/and_then to produce input_tokens and total_tokens only if all
components present), or compute input/total by consulting the existing state
(state.usage) when cache_creation_input_tokens/cache_read_input_tokens are None;
ensure message deltas only supply the fields they contain and avoid replacing
totals with zeros before calling state.usage.merge.
In `@src/gateway/providers/anthropic/transform.rs`:
- Around line 605-622: The function apply_anthropic_delta_usage_to_stream_state
currently zeroes omitted cache counters by using
usage.cache_creation_input_tokens.unwrap_or(0) and unwrap_or(0) on cache_read
when computing state.input_tokens; instead preserve previously-seen cache
counters from state when DeltaUsage omits them. Update the input_tokens branch
in apply_anthropic_delta_usage_to_stream_state to add usage.input_tokens plus
state.cache_creation_input_tokens.unwrap_or(0) and
state.cache_read_input_tokens.unwrap_or(0) (not usage.unwrap_or(0)), and
continue to only overwrite cache_creation_input_tokens and
cache_read_input_tokens when usage provides Some(...) so existing counters are
retained otherwise.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 009f91fe-04d5-495e-b05e-ad7d481293dd
📒 Files selected for processing (4)
src/gateway/formats/anthropic_messages.rssrc/gateway/gateway.rssrc/gateway/providers/anthropic/transform.rssrc/gateway/types/anthropic.rs
🚧 Files skipped from review as they are similar to previous changes (2)
- src/gateway/gateway.rs
- src/gateway/types/anthropic.rs
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
As 13th part of the major provider refactoring, add the Anthropic Messages API chat format for streaming.
Summary by CodeRabbit
New Features
Bug Fixes
Documentation