Model Provider Adapters & Streaming

Model Provider Adapters & Streaming

This page covers the wire-level half of the chat stack: two provider adapters that translate the app's internal conversation into a provider request and translate the provider's streamed response back into a provider-agnostic ChatEvent stream, plus the single SSE frame reader both adapters consume. The chat runtime described in Chat Runtime, Conversation & Cost never talks HTTP itself; it drives one of these adapters as an async generator and reacts to events. Everything here is Electron-free and dependency-free — global fetch only — and each adapter accepts an injectable fetchFn, which is the seam tests use.

How a streamed turn runs

The adapters are lazy async generators. Nothing happens at construction; the HTTP request is issued on the first next() from the runtime, and each subsequent pull pushes the generator forward through the response body. Because the generator yields while reading, the runtime sees delta/thinking/toolCall/usage events in provider order rather than a buffered result. All state (tool accumulators, token counters, SSE buffer) is local to one generator invocation — no per-request state survives between turns; the runtime owns the conversation history and feeds it back in on the next call.

Both adapters normalize to the same event contract, inferred from how they are yielded:

kind	Payload	Meaning
delta	text	Assistant-visible text chunk
thinking	text	Reasoning/thinking chunk (provider-specific source)
toolCall	call: { id, name, argsJson, status, isError? }	A complete (or definitively broken) tool call
usage	inputTokens, outputTokens, model	Token accounting for the turn
done	reason: 'end_turn' | 'stopped'	Terminal event; stopped only on abort

done is always terminal and never carries the provider's stop reason — stop_reason / finish_reason are read from neither stream, so end_turn is reported even when the model stopped for max_tokens or to request a tool. Consumers infer continuation from yielded toolCalls instead.

sequenceDiagram
  participant RT as Chat runtime
  participant AD as streamAnthropic / streamOpenAI
  participant SSE as readSse (sse.ts)
  participant P as Provider endpoint

  RT->>AD: iterate the async generator
  AD->>P: POST history + tools, stream: true, signal
  P-->>AD: 200 text/event-stream body
  loop until message_stop / [DONE]
    AD->>SSE: pull frames from ReadableStream
    SSE-->>AD: SseMessage { event?, data }
    AD-->>RT: yield ChatEvent (delta | thinking | toolCall | usage)
  end
  AD-->>RT: yield done(end_turn)
  Note over AD,RT: AbortSignal fires → fetch/readSse reject AbortError → yield done(stopped)

Loading

Key nodes: the adapter builds the request body (provider-specific), then hands the raw res.body to readSse, which is the only code that understands SSE framing. The adapter only interprets the JSON inside each data: line. Abort is handled at both ends — the signal is passed to fetch and to readSse, and both adapters catch AbortError and convert it into done/stopped rather than throwing. Non-abort network failures and non-2xx responses are rethrown as errors / ChatError.

Request and response mapping at a glance

Concern	Anthropic (anthropic.ts)	OpenAI-compatible (openai.ts)
Endpoint	baseUrl defaults to https://api.anthropic.com; appends /v1/messages	baseUrl required; appends /chat/completions, inserting /v1 when absent
Auth	x-api-key + anthropic-version: 2023-06-01	Authorization: Bearer
System prompt	Hoisted out of messages into the top-level system field	Stays in-band as a role: 'system' message
Tool result	tool_result block inside a user turn; consecutive results merge into one user turn	role: 'tool' message with tool_call_id
Assistant tool replay	text (if any) + tool_use content blocks	tool_calls[] array; content forced to '' (OpenAI forbids null)
Tool schema	{ name, description, input_schema }	{ type: 'function', function: { name, description, parameters } }
Thinking stream	delta.type === 'thinking_delta'	delta.reasoning_content
Tool-call accumulation	One block at a time; input_json_delta fragments concatenated	Map keyed by tool_call.index; id/name/arguments concatenated per index
Usage	message_start seeds it, message_delta updates it, message_stop emits it	Only if the provider sends a usage-only chunk (no choices)
End sentinel	message_stop	data: [DONE]
Max tokens	maxTokens option, default 4096	Hardcoded 4096; no override
note messages	Dropped before serialization	Dropped before serialization

File responsibilities and how they collaborate

The three files form a strict layering: sse.ts knows bytes → frames; the adapters know frames → provider JSON → ChatEvent; the runtime knows ChatEvent → conversation state. The adapters share no code with each other beyond readSse and the types in ./types / ./tools, so a change to one provider cannot regress the other.

src/core/chat/sse.ts — the shared frame parser

readSse(body, signal) decodes the response stream with a streaming TextDecoder, buffers until newlines, and dispatches each block at a blank line. Its tolerances are load-bearing for real providers: CRLF line endings are normalized (trailing \r stripped), a trailing block without a final blank line is flushed at stream end, multiple data: lines in one block are joined with \n, leading whitespace after data: is trimmed, : keepalive comments and id:/retry: fields are ignored, and event: names are captured.

Two boundaries matter when modifying it: unparseable content is not its problem — it yields SseMessage values whose data is opaque to it; and abort surfaces as a DOMException('Aborted', 'AbortError'), either from the explicit signal.aborted check before each read or from the underlying fetch body rejecting. The finally block cancels the reader and swallows cancel errors. Note that neither current adapter reads message.event; only message.data is consumed, so event names are parsed but unused — a hook for future providers that frame semantics by event name.

src/core/chat/anthropic.ts — Anthropic Messages adapter

toAnthropicBody(messages, system) is the pure request-shaper, exported for direct unit testing. It enforces the API's structural invariants:

• A leading system message is promoted to the top-level system string only when no explicit system option was passed and it is the first non-dropped message; a system message appearing after any other message is silently dropped, and an explicit system option takes precedence over the head message.
• note messages never go over the wire.
• tool messages become tool_result blocks inside a user turn. Consecutive tool results merge into the previous user turn when that turn already has array content, because the API requires strict user/assistant alternation and each tool_result must pair with its initiating tool_use.
• An assistant message that requested tools replays what the model asked for: its text (if any) plus tool_use blocks, with argsJson parsed into input and falling back to {} on malformed JSON.

anthropicEndpoint(baseUrl) strips trailing slashes and appends /v1/messages. Unlike the OpenAI builder it does not deduplicate an existing /v1, so baseUrl must be a bare origin.

streamAnthropic(opts) posts the body (stream: true, max_tokens defaulting to 4096, tools mapped as above) and drives a small state machine over the SSE data: message_start seeds inputTokens/outputTokens; content_block_start with tool_use opens a single toolAcc accumulator; content_block_delta dispatches to delta, thinking, or accumulates partial_json; content_block_stop closes the accumulator and yields a toolCall whose status is done if the accumulated JSON parses and error/isError otherwise; message_delta updates token counts; message_stop emits usage if any tokens were seen. A fallback after the loop emits usage if the stream ended without a message_stop usage emission. Lines that fail JSON.parse are skipped rather than fatal.

src/core/chat/openai.ts — OpenAI-compatible adapter

openAiEndpoint(baseUrl) is provider-tolerant: it strips trailing slashes and appends /v1/chat/completions, but skips the /v1 insertion when the URL already contains /v1/ or ends in /v1. This is what lets one adapter serve the bare origins the apiProviders configuration stores, covering OpenAI, Groq, OpenRouter, LM Studio, llama.cpp, and Ollama's OpenAI shim.

toOpenAiMessages(messages) drops note messages, converts tool messages to role: 'tool' with the originating tool_call_id, and replays assistant tool_calls as { id, type: 'function', function: { name, arguments: argsJson } }, forcing content to '' when the assistant text is empty so the request is not rejected.

streamOpenAI(opts) is chunk-oriented rather than block-oriented, which drives its state: accs: Map<number, ToolAcc> keyed by tool_call.index, and a yielded set recording indices already emitted. Content deltas yield immediately; reasoning_content maps to thinking. Tool calls concatenate id/name/arguments fragments per index and are yielded eagerly the moment id, name, and args are all present and the args parse as complete JSON — this is what lets the runtime start executing a tool while later chunks are still arriving. Once [DONE] is seen the loop breaks, but read carefully: the post-loop flush still runs, emitting any accumulator that was never eagerly yielded (incomplete or empty args) with whatever status toolCallFrom assigns — broken JSON becomes an errored call rather than a silent drop. Usage is only read from chunks that carry usage and have no choices, so providers that never send such a chunk produce no usage event at all (no accumulation fallback, unlike Anthropic).

Boundaries, failure modes, and extension points

• Abort vs. error. AbortError from either fetch or readSse becomes done/stopped; every other failure propagates. Non-2xx responses throw ChatError tagged with the provider name, HTTP status, and the first 200 chars of the body; a missing response body throws ChatError with no status.
• Malformed data is survivable. Bad JSON in an SSE frame is skipped by both adapters; bad tool-argument JSON still produces a toolCall marked error, preserving id/name pairing for the runtime.
• Thinking blocks are transport-neutral. The adapters only distinguish delta from thinking; how thinking is displayed or whether it is persisted is a runtime/UI concern.
• fetchFn injection in both option bags is the primary test seam and the way to route through a proxy or mock without touching adapter logic.
• New OpenAI-compatible providers are configuration-only — base URL, model, key. A provider with a differently shaped API needs a new adapter that reuses readSse and yields the same ChatEvent kinds; adding a new kind requires touching the chat type contracts in ./types and every consumer, since the Anthropic adapter's switch silently ignores unknown event types.
• Asymmetries to know before editing: Anthropic exposes maxTokens, OpenAI does not; Anthropic's endpoint builder does not deduplicate /v1, OpenAI's does; Anthropic reconstructs usage from three events, OpenAI only from a dedicated chunk; Anthropic requires alternation-merge logic for tool results that the OpenAI shape does not.

Limits of this page

Only anthropic.ts, openai.ts, and sse.ts were inspected. The exact declarations of ChatEvent, ChatMessage, ChatToolCall, ChatError (from ./types) and ChatToolDef (from ./tools) are described here only as far as these adapters exercise them; see Chat Tool Calling & Project Tools for the type contracts, and Chat Runtime, Conversation & Cost for the consumer loop, cancellation orchestration, and cost accounting built on top of these events.

Sources: src/core/chat/anthropic.ts, src/core/chat/openai.ts, src/core/chat/sse.ts