RFC: Virtualize the LLM provider interface behind a normalized Adapter in ChatBase

[dsapandora]

RFC: Virtualize the LLM provider interface behind a normalized Adapter in ChatBase

Status: Proposal · Author: Ariel Vernaza · Area: packages/ai (LLM chat layer)

1. Objective — what we want to solve

ChatBase currently speaks each provider's native wire shape directly. That coupling produced the recent production incident (#1658 / #1659) and blocks correct multi-turn reasoning. Concretely:

• Provider shapes leak into the base. The base branches on isinstance(content, list), on Anthropic thinking/text/signature blocks, on OpenAI Responses events, on reasoning_content, on betas, etc. Any new provider quirk becomes a base-class change.
• The crash. With extended thinking on, Anthropic returns a list of typed blocks. The non-streaming / expectJson (agent) path returned it verbatim → 'list' object has no attribute 'strip', or leaked raw thinking blocks. (Root cause: the base handles provider content shapes at all.)
• Reasoning state is thrown away. Providers require prior-turn reasoning to be replayed verbatim (Anthropic signature / redacted_thinking; OpenAI encrypted_content reasoning items). Today history is flattened to a text block and reasoning is discarded, so multi-turn reasoning is degraded/incorrect.
• Per-model knowledge rots. build_anthropic_thinking_kwargs blanket-rejects all Haiku (so Haiku 4.5 silently gets none) and sends type:"enabled" to adaptive-only models (Sonnet 5 / Fable → 400).

Goal: virtualize the OpenAI/Anthropic (and every other) provider behind a single normalized interface, and make ChatBase consume only that interface. Providers become interchangeable; the base never touches a provider-native shape again.

Goals

• One normalized streaming contract for every provider.
• The base is provider-agnostic: it sees only text / thinking / done events.
• Reasoning state round-trips correctly across turns (opaque, verbatim).
• Adding/upgrading a provider = one adapter, zero base changes.

2. How it works today

Entry: LLMBase._question (llm_base.py:40) calls chat.chat(question, on_chunk, on_finish, on_reasoning_chunk). Question.getPrompt() (question.py:524) flattens Question.history into a ### Conversation History: text block (question.py:598-603).

Dispatch: everything funnels through the monolithic chat_string (chat.py:516-680), a 4-way branch, each branch provider-aware:

flowchart TD
  A["LLMBase.ask / _question"] --> B["Question.getPrompt()<br/>flattens history → single string"]
  B --> C["ChatBase.chat_string<br/>(monolithic 4-way dispatch)"]
  C -->|"reasoning + Responses"| D["OpenAI Responses API<br/>responses.create(stream=True)"]
  C -->|"native registry"| E["Anthropic Messages<br/>messages.create(stream=True)"]
  C -->|"native registry"| F["openai_compat<br/>chat.completions(stream=True)"]
  C -->|"else / fallback"| G["LangChain .stream() / .invoke()"]
  D --> K["reasoning → on_reasoning_chunk<br/>(SSE 'thinking' lane, then discarded)"]
  E --> K
  F --> K
  D --> H{"content shape?"}
  E --> H
  F --> H
  G --> H
  H -->|"str (OpenAI-style)"| I["answer text ✅"]
  H -->|"list of typed blocks (Anthropic)"| J["returned verbatim →<br/>💥 crash / raw-block leak on agents"]

Loading

Provider-leak points in the base today: chat.py (Responses responses.create + response.* events; isinstance(content, list); thinking/text/signature/reasoning block handling; raw openai.OpenAI client build; _make_think_tag_splitter for Ollama <think>), and llm_native_stream.py (betas, adaptive/enabled thinking, _NATIVE_CREATE_KEYS, reasoning_content, messages.create/beta.messages.create). Thinking vs text is separated by two callbacks (on_chunk / on_reasoning_chunk). Reasoning is emitted to the UI and dropped — only the visible text is returned.

3. How it will work — normalized Adapter

Introduce a provider-agnostic streaming interface. Every provider implements it; ChatBase consumes only it.

@dataclass
class Event:
    type: str                 # "thinking" | "text" | "done"
    text: str = ""
    items: list = field(default_factory=list)

class Adapter(Protocol):
    history: list             # provider-native message/input list, owned by the adapter
    def stream(self, user_text: str) -> Iterator[Event]: ...

Contract:

• Yields Event("thinking"|"text", text=...) for display, in order.
• Yields Event("done", items=[...]) exactly once, last.
• items is provider-native and OPAQUE: appended to the adapter's history verbatim. Never inspected, edited, reordered, or reserialized — both providers reject or silently misbehave on tampered reasoning state.

flowchart TD
  A["LLMBase.ask / _question"] --> B["ChatBase (thin)<br/>drives the Adapter"]
  B --> C["Adapter.stream(user_text)<br/>→ Iterator[Event]"]
  C --> D["AnthropicAdapter<br/>messages.stream()"]
  C --> E["OpenAIAdapter<br/>responses.stream()"]
  C --> F["LangChainAdapter<br/>(the other ~18 providers)"]
  D --> G["Event('thinking') → SSE 'thinking' lane"]
  E --> G
  F --> G
  D --> H["Event('text') → answer string (always str)"]
  E --> H
  F --> H
  D --> I["Event('done', items=OPAQUE)<br/>→ append to history verbatim"]
  E --> I
  F --> I
  I --> J[("Structured conversation state<br/>reasoning signatures / encrypted items preserved")]

Loading

ChatBase becomes thin: create the right adapter, drive adapter.stream(...), route text→answer, thinking→the existing SSE lane (llm_base.py:59), and on done append items to the adapter's history. The non-streaming / expectJson path simply drains the stream keeping text events → always a plain string. The crash disappears structurally: the base never sees a provider content shape.

The three adapters

• AnthropicAdapter — client.messages.stream(...); thinking_delta→thinking, text_delta→text; get_final_message() gives the opaque assembled content (intact signature, redacted_thinking). The per-model thinking mode (adaptive vs extended+budget vs none, Haiku 4.5 correctness) lives here, replacing the stale build_anthropic_thinking_kwargs.
• OpenAIAdapter — client.responses.stream(..., store=False); reasoning_summary_text.delta/reasoning_text.delta→thinking, output_text.delta→text; get_final_response().output items (with encrypted_content) are the opaque items.
• LangChainAdapter — wraps the existing ChatAnthropic/ChatOpenAI/etc. so the other ~18 providers emit the same Event stream. Its done.items is just the plain-text turn. This normalizes everything without rewriting 20 drivers to raw SDKs.

4. What changes, concretely (single implementation)

We land this as one cohesive change, not phased:

1. New packages/ai/src/ai/common/llm_adapter.py — Event, Adapter Protocol, AnthropicAdapter, OpenAIAdapter, LangChainAdapter.
2. ChatBase — replace the chat_string monolith (chat.py:516-680), the native-stream registry, and _chat's raw results.content return with: build/hold an Adapter, drive its Event stream, collect text, forward thinking, persist done.items. Delete the provider-shape branches from the base.
3. llm_native_stream.py — its handlers/_NATIVE_CREATE_KEYS/build_anthropic_thinking_kwargs move into the adapters; the module shrinks to shared helpers (or is absorbed).
4. History contract — Question/LLMBase switch from getPrompt() -> str to handing the adapter a structured message/turn so the adapter owns history and can round-trip opaque items. This is a contained Python change in the client-python SDK's Pydantic Question / QuestionHistory models (rocketride/schema/question.py) — no engine change: QuestionHistory carries the opaque items, and getPrompt stops flattening assistant reasoning to text. Single-turn works with the current flat prompt today; multi-turn is this follow-up.
5. Drivers (llm_anthropic, llm_openai, …) — their __init__ selects the adapter instead of wiring _native_stream_provider; bag['chat'] unchanged, getChat factory (chat.py:743) unchanged.
6. SDK helpers — adopt .messages.stream() / .responses.stream() + get_final_message()/get_final_response() (today we use .create(stream=True) + manual assembly). These are exactly what hand us the intact opaque items.

5. What it resolves

• Crash & leak (Anthropic typed-block content not flattened on the non-streaming path — breaks every agent node #1658) — structural: the base only ever handles normalized text. (Edward's flatten fix: flatten Anthropic typed-block content on the non-streaming path #1659 stays as the stopgap until this lands.)
• Correct multi-turn reasoning — opaque items round-trip verbatim; caching & signatures preserved.
• New/upgraded provider = one adapter, base untouched.
• Stale per-model mode — moves into AnthropicAdapter, aligned to Anthropic's current model table.

6. Risks & mitigations

• Behavior parity of streaming → keep the exact SSE thinking/text split and node API; golden-path tests per provider (chat, agent/expectJson, streaming, non-streaming).
• History contract change is broad → land behind the same chat()/ask() signatures; adapters default to single-turn if no structured history is supplied, so existing single-shot callers are unaffected.
• Cross-provider → LangChainAdapter guarantees the 18 non-reasoning providers keep working unchanged.
• Fallback → adapter stream() errors surface as today (retry/map_exception); on a mid-stream failure nothing is appended to history, so state stays consistent (mirrors the reference sse() contract).

7. Open questions for discussion

• Where should the adapter's history live across turns — on the node instance, or threaded through Question?
• Do we expose the Event stream to the node layer directly (replacing the two callbacks), or keep callbacks as a thin shim over the stream for now?
• Naming/location: llm_adapter.py vs folding into chat.py.

[github-actions[bot]]

🤖 Internal: Discord sync marker

Auto-managed by the Discord notification workflow. Stores the linked Discord message ID and forum thread ID. Do not edit or delete.

[dsapandora]

Proposed resolutions for the open questions — sharing what I'd propose so we can lock direction before I start (feedback welcome, @Rod-Christensen):

1. Where the adapter's history lives across turns → thread it through Question.
RocketRide's pipeline model keeps state in the data that flows, not on the node (nodes stay stateless / reusable), and Question.history is already the carrier. So we seed the adapter's history from Question on each call and write the opaque done.items back to it. Multi-turn reasoning state travels with the conversation object instead of being pinned to a node instance. (This is the load-bearing one — it's what makes the opaque items contract actually round-trip.)

2. Event stream vs. the two callbacks at the node layer → keep the callbacks as a thin shim over the stream.
The Event stream stays internal to the chat layer; LLMBase and the SSE thinking lane keep their current contract. This bounds the change to ChatBase and preserves the node/UI surface. We can revisit exposing the stream directly later if there's a reason.

3. Module location → a dedicated packages/ai/src/ai/common/llm_adapter.py for Event + Adapter + the concrete adapters, keeping chat.py focused on ChatBase.

If these look right, I'll start the single implementation on that basis.

[EdwardLien0426]

Adding the interim stop-the-bleeding piece here for visibility, so it lines up with this RFC.

Short-term (mine): I am gating extended thinking off at the node in llm_anthropic (_EXTENDED_THINKING_ENABLED = False) so Anthropic stops returning content blocks on the agent path. Gated at the node rather than by flipping capabilities.reasoning, because that flag is stamped by tools/sync_models and re-stamps to true on the next sync. It is a single reversible constant. Tracked in #1680, PR #1681.

Defense-in-depth: the flatten in #1659 is parked as a draft. It stays as the safety net for when thinking is turned back on.

Long-term: this RFC (the normalized adapter) is the real fix. When it lands, the node gate goes away (thinking is re-enabled behind the adapter, which round-trips reasoning correctly and keeps provider shapes out of the base), and the flatten either merges or is subsumed by the adapter.

So the sequence is: disable now (#1681) so cloud is unblocked, keep #1659 as the net, then your adapter replaces both. Flagging in case anyone hits Anthropic reasoning models in the meantime, they will get no thinking until the adapter is in.

This is just my read of how the interim and the RFC fit together, so please correct me if I have the sequencing wrong or if the gate should live somewhere else.

[dsapandora]

@dylan-savage — agreed on landing it as one cohesive change, and the Adapter/Event shape.

Before I wire the agent side, the part I want your read on is exactly Rod's caution: when do we replay thinking, and how do we keep it controlled. A few things from the agent code that shape the answer:

• The planner runs on the path that can't capture thinking today. agent_rocketride's planner is expectJson=True, and chat_string disables streaming for expectJson (chat.py:552) → it lands on the non-streaming _chat_with_retries, which never produces items. So capture can't hang off the streaming branch — the unify step has to cover the non-streaming path too, or we decouple item-capture from streaming entirely. I lean decouple.
• No native tool_use → no hard Anthropic replay constraint. The planner uses a JSON text protocol ({thought, scratch, tool_calls}), not native tool_use, so we don't hit the "thinking block must precede tool_use before tool_result" rule — replay is plain multi-turn. But the agent already carries a text thought/scratch wave→wave, so I'd like your take on how much signed provider-thinking actually buys us here vs. a future native-tool_use agent.

On control (Rod's danger) I think we need an explicit policy, not always-on:

• Toggle: opt-in flag on the node, default off.
• Model-scoped: signatures are model-specific — never replay thinking into a different model/provider. Tag items with the model, drop on mismatch.
• Never pruned/rebuilt: the wave-history trimming (rocketride_agent.py:151) must leave items untouched — verbatim replay only.

On multi I see two scopes with different risk:

1. Multi-turn within one agent loop (thinking carried wave→wave) — clean win, low risk, where I'd start.
2. Multi-instance across pipeline nodes (thinking crossing LLM nodes via the lane) — only valid same-model, and items have to ride in Question/Answer across the pipeline/websocket; more surface, more failure modes.

My inclination: land multi-turn-within-agent first (behind the toggle), keep cross-node as the deliberate next step once the control model is proven. Does that match your scoping, or do you want cross-node in from the start?

Scope check: agent_rocketride only for now — the framework agents (crewai/langchain/llamaindex/deepagent) delegate to their own SDK history, so they're a separate track. Agree?

[dylan-savage]

@dsapandora,

Decouple vs unify: unify, slightly stronger than your lean. Rather than a separate capture mechanism, implement the non-streaming/expectJson path as draining the adapter stream, same iterator, display callbacks just not wired, keep text + done.items. Capture is then structural on every path including the planner's, and _chat_with_retries stops existing as a distinct branch. Decoupled capture as a second mechanism would be something to keep in sync forever.

What signed thinking buys under the text protocol: effectively nothing, "replay is plain multi-turn" is the catch, not the escape. Anthropic doesn't just not require thinking replay outside tool-use continuations, it ignores prior-turn thinking blocks in plain multi-turn entirely. OpenAI's encrypted items are likewise only consumed across function-call chains. So wave→wave replay under the JSON protocol is a no-op at the provider: scratch isn't a stand-in for signed thinking, it's strictly more useful today. That flips the scoping conclusion: within-agent replay isn't a "clean win, low risk" first step, it's zero win until the planner speaks structured turns with native tool_use, at which point it becomes load-bearing (and for Anthropic, mandatory). I'd still land capture + tagging now to prove the plumbing, but I wouldn't put replay behind the toggle expecting it to do anything before the native-tool_use planner exists.

Policy: agreed, two tightenings. Tag with provider + model + thinking-mode (adaptive vs enabled shouldn't cross either). And I'd restate "never pruned/rebuilt" as per-turn atomicity: when trimming (:154-165) touches a turn, drop that turn's items entirely and continue, verbatim or absent, never a repaired middle state.

Scoping: agree on both counts. Cross-node stays out, beyond your serialization point, the agent host currently pins exactly one LLM node per run (host.py:29-37), so there's no in-agent consumer for cross-model handling anyway. And yes, agent_rocketride only; the framework agents are their own track.