Pipeline Continuations: Exchange-Level Iteration for Agentic Loops

[leseb]

Problem

Agentic AI loops need a decision after an upstream response, followed
by another upstream exchange:

call model -> read tool calls -> run tools -> call model again -> repeat

Each iteration must target a configured upstream cluster -- with
routing, load balancing, endpoint selection, TLS, retries, and health
checks -- and pass through the full filter pipeline: guardrails,
auth, token accounting, logging, model rewrite.

This cannot be solved with existing primitives:

• Branch chains evaluate during the request phase, before the
  upstream response exists. Response-derived decisions (like tool calls
  from a model) cannot trigger branch re-entry.
• Sub-requests from a filter (see #87)
  go to arbitrary endpoints and bypass the filter pipeline. Intermediate
  model calls need the same guardrails and observability as the initial
  call. A sub-request API also pushes proxy-level responsibility (cluster
  routing, TLS, retries, response replacement) into filter logic.

Why a New Primitive

What is missing is an exchange-level loop: run a complete HTTP
exchange (request filters, upstream, response filters), inspect the
result, and optionally run another complete exchange before returning a
final response to the client.

This is different from branch chains (request-phase composition within
a single exchange) and different from sub-requests (filter-owned
callouts to external services). It loops over the proxy's own machinery
-- each iteration is a full proxy exchange through the configured
pipeline targeting the configured cluster.

Proposed Mechanism: Pipeline Continuations

A continuation is a core-scheduled follow-up HTTP exchange that
runs before the root request returns a final response to the
downstream client.

run request chain -> upstream -> response/body filters -> decide
  if continue: build next request, run the configured chain
  if done: return final response to client

Key properties:

• Core owns the loop. Filters request continuations; core executes
  them. Filters never hold HTTP clients, manage TLS, or replace
  downstream responses directly.
• Targets configured clusters. Each iteration goes through the
  proxy's routing and upstream selection, not to an arbitrary endpoint
  the filter specifies.
• Full filter lifecycle per iteration. Each continuation step runs
  request filters, request body filters, upstream exchange, response
  filters, and response body filters.
• Explicit state boundary. Per-exchange state (upstream selection,
  body buffers, retry counters) resets between steps. Workflow state
  (iteration count, accumulated context) persists explicitly.
• Safety by default. Max iterations, deadline budgets, and
  cancellation are enforced by core, not by individual filters.

Non-Goals

• Do not extend branch chains into the response phase.
• Do not let filters issue direct upstream requests.
• Do not release bytes downstream before the final response (v1).
• Do not build a workflow engine or DAG executor.

What We Are Asking

1. Confirmation that the agentic loop use case cannot be solved with
   existing branch chains or other current primitives.
2. Agreement that an exchange-level loop primitive belongs in core,
   with the separation that filters request continuations while core
   executes them.
3. Guidance on the preferred direction for a follow-up design
   proposal covering the executor, filter API, state lifecycle, config
   shape, and safety semantics.

[usize]

I'm confirming that callouts from #87 do not meet your needs. It's a separate HTTP stack on purpose -- to prevent connection re-use across filter pipelines.

Continuations would resolve issues we are currently having with provider fallback in discussion 287 -- showing that you've identified a need that expands beyond the scope of the agentic loop service. That is to say, I support adding this to core.

In short, my analysis on discussion 287 leads me to believe that 1. you are right that we need this new primitive and 2. it belongs in core.

For point 3. I'll try to think through it from the perspective of the key properties first:

• Core owns the loop.

We should consider being explicit about things like mTLS and SNI being inherited by continuations. My thought is that they should be inherited, because particularly for mTLS that's our identity and it would be strange for our continuation to have a different "identity" than the parent.

• Targets configured clusters.

This makes connection re-use less problematic. This is the restriction that I couldn't add for generic http callouts, which is why it needed its own HTTP stack. Perfect. 👌🏻 ✨

• Full filter lifecycle per iteration.

Makes sense to me. And then ownership is easy because we should be able to return a response stream handle from our inner loop if we'd like. Clean.

• Explicit state boundary.

Yes. Key feature. Makes perfect sense.

---

For the shape of the API then, we minimally need

• cluster
• filter pipeline
• retry limit
• loop limit (where is it enforced?)
• recursion limit (unsure here, it depends on if someone might configure a cluster which targets the proxy itself. We should be explicit about that possibility. If yes, we can enforce via a header.)

Open questions that come to mind

• do we need to signal whether the continuation is eligible to be returned as the response from the outer loop?
  • I ask because I'm immediately thinking about cases like nested continuations returning the response stream from the innermost continuation. Maybe that is always fine and doesn't need to be disallowed? I think this would still be possible/desirable even given the non-goal of releasing bytes downstream before the final response.

Performance questions

• If we share context between all continuations, we have potential for unbounded memory growth. This may imply the need for an explicit configuration for context memory.
• We have a top level deadline associated with the outer loop. How will we respect that in our continuations?

---

Overall big ➕ 1️⃣ from me. This will make the work I've been trying to do on provider fallback easier.

[shaneutt]

➕ thank you for raising this. Agreed, that action is needed.

I expect this will take some time, and there's some relation to https://github.com/orgs/praxis-proxy/discussions/87, so I have created #786 to encompass this, made it urgent priority, and have assigned it to myself. You'll probably see a couple iterations on the description. Please subscribe/comment there!

[shaneutt]

See: #849