Idea: Build a filter around switchyard-libsy for Mixture-of-Models Routing

[usize]

NVIDIA released NeMo Switchyard on August 11 as a model routing library. The discussion in #838 and the semantic routing section of ai#74 describe the same problem space from the Praxis side. After looking at Switchyard's architecture I think there's a clean integration path that avoids reinventing the routing algorithms while keeping the classifier-as-black-box approach that I outlined in the #838 discussion.

What Switchyard Is (and Isn't)

Switchyard is a routing decision library, not a gateway. Its crate switchyard-libsy never makes HTTP calls itself. It yields a stream of Step enums:

for step in algorithm.run(request).await {
    match step {
        Step::CallModel(call) => {
            // Host fulfills the model call with its own transport
            let response = my_client.call(call).await;
            algorithm.feed(response);
        }
        Step::Done(response) => return response,
    }
}

The host owns the HTTP stack, credentials, connection pool, and retry policy. The library just says "call this model" and "here's my routing decision."

Switchyard ships four routing algorithms:

Algorithm	How it works	Overhead
LLM Classifier	A judge model classifies request complexity before routing	One model call per decision
Stage Router	Reads conversation signals (tool failures, error loops) to route without an extra model call	Near-zero
Escalation	Starts cheap, promotes to frontier after N negative judge verdicts; one-way door	One judge call per turn until lock-in
Random	Weighted traffic split for A/B testing	Zero

It also handles protocol translation (OpenAI Chat, OpenAI Responses, Anthropic Messages), but Praxis already does this with deeper coverage.

Why This Maps to a Praxis Filter

The Step::CallModel pattern maps directly to SubRequestClient. A filter would:

1. Declare BodyMode::StreamBuffer to peek at the request envelope
2. Construct a switchyard-protocol request from the classified body
3. Run the algorithm
4. Fulfill Step::CallModel instructions via SubRequestClient (sharing the server-level connection pool)
5. Write the selected cluster to ctx.filter_metadata or ctx.cluster

This is structurally identical to what ai_guardrails already does: StreamBuffer peek, external callout via SubRequestClient, routing decision based on the result.

The filter would sit in the classify -> validate and select -> rewrite pipeline that #838 converged on. The algorithm's routing decision maps to a well-known header or metadata key that downstream filters (like the proposed model_rewrite, see praxis-proxy/ai#643) can consume.

What Praxis Gets

• Four routing algorithms without implementing them.
• Stage Router is particularly interesting -- it routes on signals already present in the conversation (tool results, errors) without an extra model call. This addresses the "classification cost" concern raised in Proposal: Complexity Based Model Routing #838.
• Session-level escalation state that Switchyard already maintains. The one-way-door pattern (once escalated to frontier, stay there) is a clean answer to the "model bouncing" problem discussed in Proposal: Complexity Based Model Routing #838.

What Praxis Keeps

Everything that Switchyard doesn't do, which is most of what an AI gateway does:

• Protocol translation (deeper coverage than Switchyard)
• MCP / A2A / agentic protocol support
• Response persistence and conversation management
• Credential injection, guardrails, rate limiting
• Request validation, prompt enrichment, token accounting
• Hot-reloadable routing overlays, session affinity, load balancing
• The full L7 proxy layer (Pingora, TLS, HTTP/2, WebSocket)

Importantly, we can continue to work with llm-d on their own proposed semantic classification project, which will now have a clear counterpart and thus an API shape to work from.

Integration Shape

The dependency would be on switchyard-libsy and switchyard-protocol only. No need for switchyard-server or any of the proxy infrastructure. These are pure Rust crates with a Tokio async interface.

# In praxis-ai-filters/Cargo.toml
switchyard-libsy = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git" }
switchyard-protocol = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git" }

The filter configuration would map Switchyard's TOML concepts (targets, tiers, thresholds) into Praxis YAML:

filters:
  - filter: switchyard_route
    algorithm: llm_classifier
    mode: escalation
    classifier_cluster: classifier-backend
    threshold: 0.5
    confirmations: 2
    candidates:
      - tier: efficient
        cluster: local-vllm
      - tier: capable
        cluster: openai-frontier

Caveats

• Switchyard is pre-alpha (API will change before v1.0). A vendored snapshot or pinned commit would be prudent until it stabilizes.
• The switchyard-protocol types overlap with types in praxis-ai-apis. A thin translation layer between the two type systems would be needed. This is straightforward -- both represent the same OpenAI/Anthropic wire formats.
• Switchyard's escalation state is per-session. It would need to integrate with Praxis's session affinity mechanism or the storage layer (ai#412) for cross-request persistence.

Questions

1. Is there a version of this story where we need to ship a model with this filter? We need to think about that story. Semantic classification depends on models and deploying models is hard. Ideally we can stick to callouts to existing inference endpoints without worrying about deployment.
2. Solving the "model bouncing" problem via ratchet (only goes up, never down in complexity) is fine until a failure causes us to lose session state. What is the failure mode when a session with MoM loses semantic routing part while still engaging in turns?

References

• NVIDIA NeMo Switchyard (GitHub)
• Switchyard libsy crate docs
• LangChain Switchyard benchmark
• Kong + Switchyard integration
• Praxis Proposal: Complexity Based Model Routing #838: Complexity Based Model Routing
• Praxis ai#74: Mixture-of-Models / Intelligent Routing Epic

[yossiovadia]

Re the open question — "what is the failure mode when a session with MoM loses semantic routing state while still engaging in turns?" — the failure is silent today: the session bounces tiers mid-conversation and nobody notices until a user complains (this is essentially vllm-project/semantic-router#1439, which I reported there).

I have been sketching a complementary piece for exactly this: an async quality-sampling filter at the gateway that sends a small sample of (request + response) pairs — session-level for multi-turn traffic — to a strong judge model outside the request path, and emits per-route/per-model quality scores as metrics. With that in place, a lost-affinity failure shows up as a score drop on the affected route instead of an anecdote, with zero added latency on the hot path.

The same data has a second use that seems relevant here: Switchyard's config knobs (threshold, confirmations, tier assignments) currently have nothing to validate or calibrate them against — judge-sampled ground truth per route is exactly that missing signal.

I think the two compose cleanly: this proposal decides where requests go, the sampling layer measures whether those decisions are holding. Will write it up as its own discussion shortly.

[shaneutt]

Thank you for the analysis on this, and the proposition.

Maintainer +1

I'm in favor of someone building a POC of this in https://github.com/praxis-proxy/experimental as a next step to start exploring further.

[usize]

I added praxis-proxy/experimental#4

But @yehuditkerido is going to take over from here, to get us a nice experimental filter.

[usize]

This comment was posted by Opus 4.8 on behalf of @usize.

Following up on the maintainer +1: this is now being carried forward in praxis-proxy/experimental —

• Scaffold (filters crate, CI, identity metadata contract): feat: scaffold the filters crate — praxis-filters auto-discovery, CI, identity metadata contract experimental#1
• switchyard_route POC epic: epic: switchyard_route — Mixture-of-Models routing POC around NeMo Switchyard experimental#2

Both hang off Track A of the Standalone AI Gateway MVP epic (ai#758). Two answers to the open questions above, from reading Switchyard v0.2.0: (a) no model shipping needed — the classifier/judge is an ordinary CallModel step against any existing inference endpoint; (b) lost escalation state fails open — the ratchet resets to the efficient tier and re-earns the latch (state is in-process with a 1h TTL, so multi-replica needs session affinity for now).