A Thought: Streaming Agents

[shaneutt]

With the Responses API filters (#354) and the iterative request router (#786), we're touching on something that doesn't have a clean name yet. The Responses API agentic loop - where tool_dispatch writes action = "loop" to filter results, a branch chain re-enters at openai_responses_proxy, server-side tools execute inline, and the cycle repeats until the model produces a final answer... this is not really a "filter chain" in the normal sense, It's an agent. It "reasons", acts, observes, and decides whether to continue or stop. It just happens to live entirely within the lifecycle of a single HTTP request.

I'd like to propose we call this pattern Streaming Agents: agent logic that exists ephemerally on the network path born with a request and gone with the response. This is distinct from long-running agents (e.g. OpenClaw) that maintain identity and state across many requests. A Streaming Agent is the data plane executing agent-like behavior: multiple upstream exchanges, tool calls, decision points, state accumulation; all invisible to the client, who sees one request in and one response out.

Current "Streaming Agents"

The Responses API agentic loop is the most visible instance, but the same structural pattern appears across several current and planned features.

Responses API tool-call loop (today, #354)

The full chain - validate -> store -> rehydrate -> tool_parse -> responses_proxy -> stream_events -> tool_dispatch -> [branch: loop back] - is a multi-turn agent. It accumulates conversation state in ResponsesState via RequestExtensions, makes sequential inference calls, executes MCP tools and web searches between them, and decides after each response whether to loop or exit. The max_iterations: 10 on the branch chain is effectively a budget.

Semantic caching (#87)

A semantic cache is not a simple key-value lookup. The proxy needs to: embed the query, search a vector store, evaluate similarity, decide hit-or-miss, and on a miss forward to the model and persist the result. That's a multi-step decision loop with an external service call in the middle. And I'm not even convinced we wouldn't add more steps and more logic branches in time. Similar shape as the agentic loop, just with cache decisions.

RAG augmentation (#87)

Extract the user's query from the request body, call a retrieval service, inject retrieved context into the prompt, then forward to the model. Three upstream exchanges, request mutation between them, and a final response assembly. If the retrieval returns low-confidence results, you might skip injection entirely. Decisions to be made here as well by the streaming agent.

Provider fail-over with format translation (ai#287)

"On 5xx translate the request from OpenAI format to Anthropic format select a different cluster and retry"

This is response-driven re-dispatch with request mutation. A little less "agent-ish" but the proxy inspects an upstream response and makes a new, different request before the client sees anything. There are some decision points here and maybe room to grow that would end up looking more like an agent workflow in time, or in specific use cases?

P/D disaggregation (#87)

Coordinate prefill on cluster A and decode on cluster B. The proxy makes the first request, inspects the response to extract KV-cache metadata, mutates the request, routes to a different cluster, and returns only the final response. Two upstream exchanges, opaque
to the client. Could we see this having more turns in time?

Guardrails with remediation

Today, guardrails in flag mode writes results for branch chain evaluation. But imagine: detect a policy violation, call an external moderation API for a second opinion, and if confirmed, rewrite the prompt to remove the violating content and re-submit. That fits an "agent loop" pretty decently.

I'm pretty convinced guardrails is going to be a hotspot.

What do these have in common?

These share structure:

1. Classify the inbound request (format detection, body parsing, metadata extraction)
2. Make an upstream exchange (inference call, cache lookup, retrieval query, tool execution)
3. Inspect the result (tool calls returned? cache hit? retrieval confident? 5xx?)
4. Decide: loop back to step 2 with mutated state, or exit with a final response
5. Accumulate state across iterations (conversation history, tool results, cache entries, token counts)

The loop body is different in each case (different upstream targets, different decision criteria, different state shapes) but the lifecycle, observability needs, and infrastructure are identical.

Why are we getting at?

The iterative request router (#786) is effectively the first formalization of the pattern, but is it sufficient? I can see room to grow in terms of:

• Observability: The Agentic AI world around us is making a strong point that observing what choices the agent makes in addition to the actual actions taken is important. Would that naturally be important here?
• Security model: Each leg of a Streaming Agent may target a different upstream with different credentials. The SSRF prevention, credential isolation, and connection pooling that the HTTP callout filter (#358) solves need to apply uniformly across all Streaming Agent legs, not be reimplemented per use case. We also need to work out Authn/Authz inside the streaming agent.
• Budgets and cost control: The max_iterations: 10 on the Responses API branch chain is kinda a step budget. But as we continue to add on: Semantic caching, RAG, and failover all need their own bounds. A formalized pattern could provide a consistent cost-control mechanism: max iterations, max total tokens, max wall-clock time, max upstream calls.

What I'm not proposing

I'm not proposing a new abstraction layer or a rewrite. The filter pipeline, branch chains, filter results, etc don't need to be shook up. This is more about discussion and philosophical approach: What I'm suggesting is that we recognize these primitives are converging toward a common pattern, name that pattern, and let the name guide our prioritization and design decisions. Be deliberate about this.

Concretely, this might mean:

• The iterative request router (Spike: Agentic Loop Support #786) becomes "Streaming Agent Infrastructure" or something and we account for more use cases
• We design observability (spans, metrics, token accounting) with multi-leg request life-cycles in mind from the start
• We refine a consistent configuration shape for "do X, inspect the result, decide whether to loop" that operators can reuse across use cases
• Documentation and examples define and teach the pattern deliberately, not just individual filters

What I'm proposing

Discussion. This was mostly a brain dump because I'm very interested to hear your thoughts.

Some questions we might ask:

1. Does this strike you as important, or as a distraction?
2. Are there use cases I'm missing that fit the pattern? Use cases that look similar but shouldn't be grouped?
3. How should observability handle the boundary between "one client request" and "multiple upstream legs"? Nested spans? A dedicated trace attribute?

LMKWYT 🖖