[Bug]: Anthropic streaming emits wrong stop_reason after mixed text + tool-call turns

[thiscantbeserious]

[Bug]: Anthropic streaming translation emits end_turn after mixed text + tool-call streams from vLLM

Prerequisites

• I have searched existing issues and discussions to avoid duplicates.
• I am using the latest version or have tested against main/nightly.

Description

When Bifrost translates an OpenAI-compatible streaming response from vLLM into Anthropic Messages streaming, mixed text + tool-call turns are terminated with the wrong Anthropic stop_reason.

The problematic upstream shape is:

1. vLLM streams one or more choices[0].delta.content chunks.
2. In the same assistant turn, vLLM switches to choices[0].delta.tool_calls.
3. vLLM ends the OpenAI stream with choices[0].finish_reason: "tool_calls" and then data: [DONE].

Bifrost correctly creates separate Anthropic content blocks for text and tool use, but the final Anthropic message_delta is emitted as:

{"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null}}

It should be:

{"type":"message_delta","delta":{"stop_reason":"tool_use","stop_sequence":null}}

This matters for Claude Code and other Anthropic-compatible tool runners because tool_use is the signal that the client should execute the emitted tool_use block and continue the conversation. With end_turn, the stream is syntactically closed but semantically says the assistant is done, despite containing a tool call.

Official references:

• Anthropic Messages streaming event flow is message_start, content blocks (content_block_start / content_block_delta / content_block_stop), message_delta, then message_stop: https://platform.claude.com/docs/en/build-with-claude/streaming
• Anthropic stop-reason docs state that in streaming mode stop_reason is null in message_start, provided in message_delta, and not provided in other events: https://platform.claude.com/docs/en/build-with-claude/handling-stop-reasons
• Anthropic tool-use workflows branch on response.stop_reason == "tool_use" to execute tools and continue: https://platform.claude.com/docs/en/build-with-claude/handling-stop-reasons
• Claude Code gateway docs require Anthropic Messages-compatible gateways to expose /v1/messages and forward anthropic-beta / anthropic-version: https://code.claude.com/docs/en/llm-gateway

Steps to reproduce

Dependencies

• Bifrost:
  • affected release observed in production: v1.5.2
  • reproduced on latest main: 884c06157 v1.5.3 (#3531)
• Claude Code: 2.1.145 for the original client symptom
• vLLM:
  • real deployment used for validation: vllm/vllm-openai:latest, reported as v0.21.0
  • model: rdtand/Qwen3.5-122B-A10B-PrismaQuant-4.75bit-vllm
• Mock repro:
  • Python 3 stdlib only
  • no Flask dependency required

1. Start a deterministic vLLM-compatible mock

Save this as mock_vllm.py:

#!/usr/bin/env python3
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json
import sys
import time


class Handler(BaseHTTPRequestHandler):
    protocol_version = "HTTP/1.1"

    def log_message(self, fmt, *args):
        sys.stderr.write("%s - - [%s] %s\n" % (self.client_address[0], self.log_date_time_string(), fmt % args))

    def do_GET(self):
        if self.path == "/health":
            self.send_response(200)
            self.send_header("Content-Length", "2")
            self.end_headers()
            self.wfile.write(b"ok")
            return
        if self.path == "/v1/models":
            body = json.dumps({
                "object": "list",
                "data": [
                    {
                        "id": "rdtand/Qwen3.5-122B-A10B-PrismaQuant-4.75bit-vllm",
                        "object": "model",
                    }
                ],
            }).encode()
            self.send_response(200)
            self.send_header("Content-Type", "application/json")
            self.send_header("Content-Length", str(len(body)))
            self.end_headers()
            self.wfile.write(body)
            return
        self.send_error(404)

    def do_POST(self):
        length = int(self.headers.get("content-length", "0"))
        if length:
            self.rfile.read(length)
        if self.path != "/v1/chat/completions":
            self.send_error(404)
            return

        self.send_response(200)
        self.send_header("Content-Type", "text/event-stream")
        self.send_header("Cache-Control", "no-cache")
        self.send_header("Connection", "close")
        self.end_headers()

        chunks = [
            {
                "choices": [
                    {
                        "delta": {"content": "\n\nNow let me fetch the source branch:"},
                        "index": 0,
                        "finish_reason": None,
                    }
                ]
            },
            {
                "choices": [
                    {
                        "delta": {
                            "tool_calls": [
                                {
                                    "index": 0,
                                    "id": "chatcmpl-tool-123",
                                    "type": "function",
                                    "function": {
                                        "name": "Bash",
                                        "arguments": "{\"command\":\"git fetch origin feat-IGPS-15746-controller-health-endpoint\"}",
                                    },
                                }
                            ]
                        },
                        "index": 0,
                        "finish_reason": None,
                    }
                ]
            },
            {"choices": [{"delta": {}, "index": 0, "finish_reason": "tool_calls"}]},
        ]

        for chunk in chunks:
            self.wfile.write(("data: " + json.dumps(chunk, separators=(",", ":")) + "\n\n").encode())
            self.wfile.flush()
            time.sleep(0.1)
        self.wfile.write(b"data: [DONE]\n\n")
        self.wfile.flush()


if __name__ == "__main__":
    port = int(sys.argv[1]) if len(sys.argv) > 1 else 8001
    ThreadingHTTPServer(("127.0.0.1", port), Handler).serve_forever()

Run:

python3 mock_vllm.py 8001

2. Configure Bifrost with vLLM provider pointed at the mock

Create a clean app directory for the repro:

mkdir -p /tmp/bifrost-sse-repro

Save this file as /tmp/bifrost-sse-repro/config.json:

{
  "$schema": "https://www.getbifrost.ai/schema",
  "providers": {
    "vllm": {
      "keys": [
        {
          "name": "mock-vllm",
          "value": "test-key",
          "weight": 1,
          "models": ["*"],
          "vllm_key_config": {
            "url": "http://127.0.0.1:8001",
            "model_name": "rdtand/Qwen3.5-122B-A10B-PrismaQuant-4.75bit-vllm"
          }
        }
      ],
      "network_config": {
        "default_request_timeout_in_seconds": 30,
        "stream_idle_timeout_in_seconds": 5
      }
    }
  }
}

Build and run Bifrost HTTP transport from the repository root:

go build -o /tmp/bifrost-http ./transports/bifrost-http
/tmp/bifrost-http \
  -app-dir /tmp/bifrost-sse-repro \
  -host 127.0.0.1 \
  -port 18080 \
  -log-level debug \
  -log-style pretty

3. Trigger Anthropic Messages streaming through Bifrost

curl -sS -N --max-time 3 http://127.0.0.1:18080/anthropic/v1/messages \
  -H 'Content-Type: application/json' \
  -H 'anthropic-version: 2023-06-01' \
  -H 'User-Agent: claude-cli/2.1.145' \
  -H 'x-api-key: test-key' \
  -d '{
    "model": "vllm/rdtand/Qwen3.5-122B-A10B-PrismaQuant-4.75bit-vllm",
    "messages": [
      {"role": "user", "content": "Fetch branch"}
    ],
    "max_tokens": 256,
    "stream": true
  }'

4. Unit-level reproduction

This focused regression test fails before the fix:

func TestMixedTextThenToolCallsStreamMapsStopReasonToToolUse(t *testing.T) {
    ctx := schemas.NewBifrostContext(t.Context(), time.Time{})
    state := schemas.AcquireChatToResponsesStreamState()
    defer schemas.ReleaseChatToResponsesStreamState(state)

    content := "\n\nNow let me fetch the source branch:"
    toolCallID := "chatcmpl-tool-123"
    toolName := "Bash"
    toolType := "function"
    arguments := `{"command":"git fetch origin feat-IGPS-15746-controller-health-endpoint"}`
    finishReason := string(schemas.BifrostFinishReasonToolCalls)

    chatChunks := []*schemas.BifrostChatResponse{
        {
            Choices: []schemas.BifrostResponseChoice{
                {
                    Index: 0,
                    ChatStreamResponseChoice: &schemas.ChatStreamResponseChoice{
                        Delta: &schemas.ChatStreamResponseChoiceDelta{Content: &content},
                    },
                },
            },
        },
        {
            Choices: []schemas.BifrostResponseChoice{
                {
                    Index: 0,
                    ChatStreamResponseChoice: &schemas.ChatStreamResponseChoice{
                        Delta: &schemas.ChatStreamResponseChoiceDelta{
                            ToolCalls: []schemas.ChatAssistantMessageToolCall{
                                {
                                    Index: 0,
                                    ID:    &toolCallID,
                                    Type:  &toolType,
                                    Function: schemas.ChatAssistantMessageToolCallFunction{
                                        Name:      &toolName,
                                        Arguments: arguments,
                                    },
                                },
                            },
                        },
                    },
                },
            },
        },
        {
            Choices: []schemas.BifrostResponseChoice{
                {
                    Index:        0,
                    FinishReason: &finishReason,
                    ChatStreamResponseChoice: &schemas.ChatStreamResponseChoice{
                        Delta: &schemas.ChatStreamResponseChoiceDelta{},
                    },
                },
            },
        },
    }

    var events []*AnthropicStreamEvent
    for _, chunk := range chatChunks {
        for _, responseEvent := range chunk.ToBifrostResponsesStreamResponse(state) {
            events = append(events, ToAnthropicResponsesStreamResponse(ctx, responseEvent)...)
        }
    }

    if *events[6].Delta.StopReason != AnthropicStopReasonToolUse {
        t.Fatalf("message_delta stop_reason = %q, want %q", *events[6].Delta.StopReason, AnthropicStopReasonToolUse)
    }
}

Failure before fix:

--- FAIL: TestMixedTextThenToolCallsStreamMapsStopReasonToToolUse
    responses_stream_mixed_test.go:98: message_delta stop_reason = "end_turn", want "tool_use"
FAIL github.com/maximhq/bifrost/core/providers/anthropic

Expected behavior

Bifrost should translate a mixed OpenAI-compatible stream ending with:

{"choices":[{"delta":{},"index":0,"finish_reason":"tool_calls"}]}

into Anthropic Messages streaming that ends with:

event: content_block_stop
data: {"type":"content_block_stop","index":1}

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"tool_use","stop_sequence":null}}

event: message_stop
data: {"type":"message_stop"}

Actual behavior

Before the fix, Bifrost emits a tool_use block but terminates the message as if no tool was requested:

event: content_block_stop
data: {"type":"content_block_stop","index":1}

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":""}}

event: message_stop
data: {"type":"message_stop"}

In Claude Code, the original production symptom was an indefinite Waiting... state after rendering the tool block. I was able to reproduce the protocol-level defect deterministically with the mock and with a unit test. A plain curl request closes cleanly, so the deadlock symptom appears to be a client/tool-runner consequence of the wrong terminal stop reason rather than vLLM failing to terminate the upstream HTTP stream.

Affected area(s)

• Core (Go)
• Transports (HTTP)

Version

• Observed in production with Bifrost v1.5.2
• Reproduced against latest main at 884c06157 v1.5.3 (#3531)

Environment

- Client: Claude Code 2.1.145
- Bifrost transport: bifrost-http
- Provider: vllm
- vLLM image/version: vllm/vllm-openai:latest, reported as v0.21.0
- vLLM model: rdtand/Qwen3.5-122B-A10B-PrismaQuant-4.75bit-vllm
- Repro OS: macOS workstation
- Repro mock dependency: Python 3 stdlib only

Relevant logs/output

Mock-backed current-main output before fix:

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"\n\nNow let me fetch the source branch:","stop_sequence":null}}

event: content_block_stop
data: {"type":"content_block_stop","index":0}

event: content_block_start
data: {"type":"content_block_start","index":1,"content_block":{"type":"tool_use","id":"chatcmpl-tool-123","name":"Bash","input":{}}}

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":"{\"command\":\"git fetch origin feat-IGPS-15746-controller-health-endpoint\"}","stop_sequence":null}}

event: content_block_stop
data: {"type":"content_block_stop","index":1}

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":""}}

event: message_stop
data: {"type":"message_stop"}

Real-vLLM output after applying the proposed fix:

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"\n\nNow let me fetch the source branch.\n\n","stop_sequence":null}}

event: content_block_stop
data: {"type":"content_block_stop","index":0}

event: content_block_start
data: {"type":"content_block_start","index":1,"content_block":{"type":"tool_use","id":"<tool-call-id>","name":"Bash","input":{}}}

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":"{\"command\": \"git fetch origin feat-IGPS-15746-controller-health-endpoint\"}","stop_sequence":null}}

event: content_block_stop
data: {"type":"content_block_stop","index":1}

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"tool_use","stop_sequence":null},"usage":{"input_tokens":314,"output_tokens":95}}

event: message_stop
data: {"type":"message_stop"}

Bifrost log row for the successful real-vLLM repro after the fix:

{
  "object_type": "responses_stream",
  "provider": "vllm",
  "model": "rdtand/Qwen3.5-122B-A10B-PrismaQuant-4.75bit-vllm",
  "status": "success",
  "latency": 1224.0,
  "stream": 1,
  "token_usage": "{\"prompt_tokens\":314,\"completion_tokens\":95,\"total_tokens\":409}",
  "responses_output": "[{\"type\":\"message\",\"role\":\"assistant\",\"content\":[{\"type\":\"output_text\",\"text\":\"\\n\\nNow let me fetch the source branch.\\n\\n\"}]},{\"type\":\"function_call\",\"status\":\"in_progress\",\"name\":\"Bash\",\"arguments\":\"{\\\"command\\\": \\\"git fetch origin feat-IGPS-15746-controller-health-endpoint\\\"}\"}]",
  "error_details": ""
}

Regression?

Unknown. The production symptom was observed on v1.5.2; the protocol-level issue was still present on latest main before the fix.

Severity

High (major functionality broken)

Rationale: Anthropic-compatible clients can receive a tool block but no tool_use stop reason, which breaks the tool execution contract for Claude Code-style agent workflows.

Root cause

The chat-to-responses stream conversion maps OpenAI chat finish reasons to Responses terminal status, but the terminal BifrostResponsesResponse did not preserve the chat FinishReason.

The Anthropic serializer already knows how to map Bifrost finish reason tool_calls to Anthropic stop reason tool_use, but it only does so when bifrostResp.Response.StopReason is populated. Because the stream terminal response had no StopReason, the Anthropic serializer fell back to end_turn.

Relevant path:

• vLLM ResponsesStream falls back to OpenAI-compatible ChatCompletionStream.
• OpenAI chat SSE is converted to Bifrost Responses stream events.
• core/schemas/mux.go creates the terminal BifrostResponsesResponse without StopReason.
• core/providers/anthropic/responses.go defaults completed responses to end_turn when Response.StopReason == nil.

Fix proposal

1. In core/schemas/mux.go, preserve known OpenAI/Bifrost chat finish reasons on the terminal Responses event:

   • stop -> stop
   • tool_calls -> tool_calls
   • length -> length
   • unknown or empty finish reasons should preserve current behavior and leave StopReason unset.

2. Also populate StopReason in non-stream chat-to-responses conversion for consistency.

3. Keep the status mapping separate:

   • stop and tool_calls map to Responses status:"completed"
   • length maps to Responses status:"incomplete" with incomplete_details.reason:"max_output_tokens"

4. Add regression coverage:

   • schema-level test that a terminal stream event for finish_reason:"tool_calls" includes Response.StopReason == "tool_calls"
   • Anthropic-level test that mixed text + tool-call stream emits:
     • content_block_start text
     • content_block_delta text
     • content_block_stop text
     • content_block_start tool_use
     • content_block_delta input_json_delta
     • content_block_stop tool_use
     • message_delta.stop_reason == "tool_use"
     • message_stop

Patch shape:

func responsesStopReasonFromChatFinishReason(finishReason *string) *string {
    if finishReason == nil || *finishReason == "" {
        return nil
    }
    if _, _, mapped := responsesStatusFromChatFinishReason(*finishReason); !mapped {
        return nil
    }
    return Ptr(*finishReason)
}

Then use it when constructing the terminal stream response:

response := &BifrostResponsesResponse{
    ID:                state.MessageID,
    CreatedAt:         state.CreatedAt,
    Usage:             usage,
    Status:            &responseStatus,
    StopReason:        responsesStopReasonFromChatFinishReason(choice.FinishReason),
    IncompleteDetails: terminalIncompleteDetails,
}

And populate StopReason in non-stream conversion only after the finish reason is recognized by responsesStatusFromChatFinishReason.

Validation performed

Focused tests pass after the fix:

go test ./schemas -run 'TestToBifrostResponses(StreamResponse|Response)_' -count=1
go test ./providers/anthropic -run 'TestMixedTextThenToolCallsStreamMapsStopReasonToToolUse|TestToAnthropicResponsesResponse_PreservesStopReason|TestToAnthropicResponsesStreamResponse_CompletedWithCompactionStopReason' -count=1

Real vLLM wire-protocol E2E also passes after the fix:

• upstream vLLM model discovery works
• mixed text + Bash tool-call stream is emitted by vLLM
• Bifrost returns message_delta.stop_reason:"tool_use"
• Bifrost returns message_stop
• curl exits normally

Note: a full go test ./providers/anthropic ./schemas -count=1 currently hits unrelated existing provider/tool-version validation failures in the Anthropic package. The focused regression tests for this issue pass.

[coderabbitai]

🔗 Related PRs

#710 - refactor: standardize finish reason and single response handling across providers [merged]
#3335 - feat: enrich realtime routing, logging, cost, and session tracking [merged]
#3495 - fix: set idle stream timeouts in streaming requests [merged]
#3522 - handle ctx cancel before handling read errors in streaming [merged]
#3554 - feat: service tier mappings for gemini and anthropic [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[thiscantbeserious]

Verified end to end against a real vLLM deployment using Claude Opus 4.7 (1M context).

Reproduction steps from the issue work exactly as described:

1. On stock main at 884c061, the focused unit test fails with:
   message_delta stop_reason = "end_turn", want "tool_use"
2. With the proposed fix applied, the same test passes.
3. Wire protocol against a real vLLM serving rdtand/Qwen3.5-122B-A10B-PrismaQuant-4.75bit-vllm, with an Anthropic Messages streaming request that defines a Bash tool:
   • Buggy build returns "delta":{"stop_reason":"end_turn","stop_sequence":""}
   • Patched build returns "delta":{"stop_reason":"tool_use","stop_sequence":null}
4. Event ordering matches the expected block in the issue: text content_block_start and content_block_stop, then tool_use content_block_start and content_block_stop, then message_delta, then message_stop.

Credits for the original investigation and fix:

• Initial root cause analysis: Gemini.
• Issue write up, fix, and unit tests: GPT 5.5 Max at high reasoning.
• Independent end to end verification on real vLLM: Claude Opus 4.7 (1M context).

@akshaydeo this happened as I was prepping for the Enterprise test after the demo. We will move to test and triage a few streaming bugs also related to SGLang that I also discovered with this journey on the OSS first to ensure we have a smooth demo.

I believe this is in your interest to fix this as this will currently result in tool calls that endlessly freeze using Claude Code - until the timeout is hit.

The real vllm docker compose this was tested agains looks like this with a single RTX 6000 Pro:

  ai-vllm-core:
    image: vllm/vllm-openai:latest
    container_name: ai-vllm-core
    command: >
      --model rdtand/Qwen3.5-122B-A10B-PrismaQuant-4.75bit-vllm
      --host 0.0.0.0
      --port 8000
      --max-model-len 163840
      --gpu-memory-utilization 0.95
      --enable-prefix-caching
      --trust-remote-code
      --max-num-seqs 16
      --enable-auto-tool-choice
      --reasoning-parser qwen3 
      --tool-call-parser qwen3_xml
    environment:
      - HF_TOKEN=${HF_TOKEN:-}
    volumes:
      - /mnt/user/llm/huggingface:/root/.cache/huggingface
    ipc: host
    ports:
     - 8000:8000
    ulimits:
      nofile:
        soft: 65535
        hard: 65535
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    restart: unless-stopped