Proposal: Alternative architectural approach to Heavyweight Chat Concurrency & Load Shedding (avoiding 503/502 subagent drops)

[Minus-Brain]

Hi team,
First of all, huge thanks for all the incredible work on OmniRoute! It’s an essential tool for routing LLM workloads across multiple providers.
I’d like to open a discussion regarding the recent Admission Control / Load Shedding changes introduced around v3.8.49 (chatBodyAdmission, 503 chat_admission_busy, 15s queue drop timeouts, and 502 Stream ended before producing a non-ping SSE event).
The Core Problem: Defeating the Multi-Provider Paradigm
While I understand these guards were introduced to prevent V8 Heap Out-of-Memory (OOM) crashes on constrained servers, the current load-shedding mechanism inadvertently defeats the primary purpose of an AI Gateway.
When using autonomous agentic frameworks (Cursor, Claude Code, CrewAI, AutoGen, Subagents), sending concurrent requests with large contexts (15k–100k+ tokens) is the norm, not an edge case.
With forced load shedding and ultra-short queue limits (15s):
Parallel subagent workflows completely break: Incoming concurrent calls are fast-rejected with 503 chat_admission_busy at the front door before OmniRoute even attempts to distribute them across available provider targets.
Multi-provider rotation becomes useless: Having 20 healthy backend accounts or Combo targets doesn't help if OmniRoute drops the request before routing it.
Spurious 502/503 errors poison client loops: Autonomous clients treat 503 as terminal failures or trip internal retry loops, stalling execution.
In my own benchmarks on a modest server setup, running 50 concurrent requests with ~15k tokens each never caused OmniRoute to crash or OOM. Forced front-door rejection is harming setups that have plenty of RAM to handle parallel loads.
Root Cause Analysis: Why do heavy requests consume so much RAM?
A 100,000-token text request is only ~500 KB of raw UTF-8 JSON. In theory, proxying a 500 KB payload should consume negligible memory.
However, in Node.js / V8:
JSON.parse(body) expands 500 KB of raw text into 30–50 MB of V8 Heap JS Objects (messages, tool schemas, metadata).
Pipeline Cloning: In the processing pipeline, the request body is often deep-cloned multiple times (logging, token estimation, Combo target retries, target format transformations, prompt compression).
A single heavy request can temporarily inflate to 200 MB–300 MB of V8 Heap allocations. When 5 concurrent requests hit at the same millisecond, V8 heap usage spikes by ~1.5 GB before Garbage Collection can reclaim it, triggering OOM.
The current solution treats this by shedding load at the front door (503). But we can solve the memory inflation at the source instead.
Proposed Architectural Solutions (Without 503 Front-Door Drops)

1. Zero-Copy / Pass-Through Streaming for Standard Relays
   For requests that do not require deep transformation or prompt compression, OmniRoute should avoid running JSON.parse on the full body.
   Piping raw HTTP byte streams (stream.Readable / undici) directly to the upstream provider reduces memory overhead from ~200 MB down to a few KB of buffer memory per request.
   This allows a single instance to handle hundreds of concurrent agent requests effortlessly.
2. Lazy Parsing & Elimination of Deep Clones
   Lazy Parsing: Only extract top-level routing metadata (model, stream) via lightweight inspection/regex or fast header checks, leaving the heavy messages[] array unparsed until strictly needed.
   Shallow Copies for Combos: Avoid structuredClone or JSON.parse(JSON.stringify(body)) when attempting retries across Combo targets. Using read-only references or shallow object copies prevents multi-fold memory multiplication.
3. Fully Opt-In / Configurable Admission Control
   Make aggressive queue dropping and memory-based admission control strictly opt-in or configurable via environment variables (e.g., OMNIROUTE_DISABLE_ADMISSION_CONTROL=1).
   Allow operators with sufficient server RAM to set infinite/long queue wait times (maxWaitMs) and disable front-door rejection entirely.
   Summary
   The goal of a multi-provider gateway is to swallow parallel load and distribute it cleanly. Shifting from load shedding (503 rejection) to zero-copy streaming and clone elimination would restore subagent reliability while keeping memory usage near zero.
   Would love to hear the team's thoughts on this direction!

[diegosouzapw]

Hey @Minus-Brain! Thank you for the thoughtful write-up -- this is a genuinely useful architectural discussion and you have correctly identified the tension between load-shedding and multi-provider parallelism.

You are right about the core contradiction. The admission controller was introduced to prevent V8 heap OOMs on constrained servers (15s queue, 503 when saturated). Your point that this defeats multi-provider routing is valid: 20 healthy provider targets cannot help if the request is rejected before routing.

That said, the current design is more nuanced than a hard 15s door-slam. Three things worth knowing:

1. The 15s is the default queue timeout, not a hard TTL. The admission controller uses a weighted adaptive limit (AdaptiveAdmissionController) that adjusts the concurrency cap based on actual latency and success/failure outcomes (open-sse/services/admission/controller.ts). Under light load, the effective limit is well above what a single client's agentic traffic would hit. The queue only kicks in when active requests exceed the dynamically computed limit.

2. Admission per-provider, not per-gateway. The controller operates at the request level, not the provider level. A large batch of concurrent subagent requests may trigger the 503 before OmniRoute has a chance to fan them out. This is indeed a problem for agentic frameworks.

3. We already ship three mitigations for power users:

• ADAPTIVE_ADMISSION_MAX_QUEUE_COUNT (env) raises the queue depth beyond the default 128
• ADAPTIVE_ADMISSION_MAX_QUEUE_COST (env) raises the cost budget
• The queue timeout is configurable in the admission config (defaultMaxWaitMs)

What I think should happen (and would welcome your input):

The real fix is a per-connection admission lane so that heavy agentic sessions get their own concurrency budget that does not compete with the global pool. Something like:

A client connection (identified by API key or session) gets a dedicated virtual queue. The global admission still protects the heap, but one client's burst does not 503 another client's traffic, and a heavy agentic session can saturate its own lane without hitting the shared door.

This is roughly the same shape as HTTP/2 stream prioritization applied to provider routing. Would that match your mental model? If you have a specific design sketch for how per-connection lanes would interact with combo routing and the existing fair-cost queue, I would love to read it.

The 502/503 poisoning of client retry loops is a real pain point I agree needs addressing. In the meantime, the env overrides above should let you tune the defaults for your workload.

[Minus-Brain]

Hey @diegosouzapw !

Thank you so much for the detailed explanation and for validating the core contradiction. It is fantastic to see such an active and open discussion around gateway architecture.

1. Per-Connection Admission Lanes: Absolute Spot-On Direction

Yes, Per-Connection Virtual Admission Lanes match my mental model completely. Applying the bulkheading / HTTP/2 stream-prioritization pattern to client connections and agentic sessions (API Key, X-Session-ID, or x-codex-session-id) is an enterprise-grade solution.

It solves the "Noisy Neighbor" problem instantly: a hyperactive agentic burst in one session will isolate its own queue without slamming the front door on other clients or parallel subagent streams.

2. Design Sketch & Interactions with Combo Routing

Here is how I envision the Per-Connection Lanes interacting with Combo Routing and Fair-Cost Queues:

A. Lane Isolation + Combo Fan-Out / Fallback

1. Inbound Identification: An incoming request is mapped to a Virtual Lane based on API Key > X-Session-ID > IP (fallback).
2. Combo Dispatch First: When a request enters Lane A, it should attempt to dispatch through the assigned Combo targets (Target 1 -> Target 2 -> Target 3).
3. Target-Level Saturation Check: If Target 1 is saturated (hit its provider-specific concurrency or cooldown limit), the request should immediately attempt Target 2 within the same Lane execution, rather than sitting in Lane A's queue or returning a 503.
4. Lane Buffering & Pacing: If all candidate targets in the Combo are temporarily saturated, the request pauses in Lane A's queue with pacing up to defaultMaxWaitMs.

B. Dynamic Session Lane Lifecycle (Auto-Eviction)

Autonomous agents (Cursor, Claude Code, CrewAI) spin up short-lived subagents that create and destroy sessions frequently.
Virtual Lanes should be lightweight in-memory structures with an idle TTL (for example, auto-pruned after 60s of inactivity) so the Lane map itself never causes a memory leak over long uptime.

C. Backpressure & Retry Poisoning Mitigation

• When a Lane reaches its concurrency cap, prefer server-side wait / pacing up to defaultMaxWaitMs over an instant 503 chat_admission_busy.
• If a 503 must be emitted (due to extreme global heap pressure), ensure the response includes a structured Retry-After header so client agents back off gracefully instead of poisoning their internal retry loops or breaking the session.

3. Immediate Workaround

Thank you for highlighting ADAPTIVE_ADMISSION_MAX_QUEUE_COUNT, ADAPTIVE_ADMISSION_MAX_QUEUE_COST, and defaultMaxWaitMs. Tuning these environment variables provides an immediate breather for heavy workloads while the per-connection lane design matures.

I think building Per-Connection Lanes will make OmniRoute uniquely resilient for agentic workloads. I would be happy to help test any experimental PRs or nightlies whenever you begin prototyping.

[diegosouzapw]

Hey @Minus-Brain! Excellent follow-up -- your design sketch is solid and I agree on every point.

Your lane isolation + combo interaction model is exactly right. The key insight (and the most important part to get right in code) is step A.3: "if Target 1 is saturated, try Target 2 within the same lane before queuing." This is what distinguishes a lane from a simple per-key rate limiter -- the lane is a routing scope, not a bottleneck.

Tracking issue opened: #9654 with your full design sketch incorporated (lane isolation, combo interaction, auto-eviction TTL, Retry-After headers) plus acceptance criteria and implementation ideas. I will start prototyping on a feat/admission-lanes branch once the current release cycle wraps up.

Regarding testing: much appreciated. Once there is a branch with a working prototype, I will ping you on #9654 and you can pull it from a nightly or a pre-release build. The combo interaction part (step A.2-A.3) is the trickiest to get right -- your real-agentic-framework traffic patterns (Cursor, Claude Code subagents) are exactly the load profile this needs to validate against.

Thank you again for the quality of this discussion -- it went from "here is a problem" to "here is the spec" in two messages, which is rare and valuable.

[Minus-Brain]

Awesome! Thank you so much for creating issue #9654 and taking this forward into a full spec.

I will subscribe to #9654 and keep an eye out for the test build on the feat/admission-lanes branch. Really looking forward to benchmarking the real-world subagent traffic against the prototype.

Thanks again for the great collaboration and open discussion!

[diegosouzapw]

Hey @Minus-Brain! Thank you again for the thoughtful update and benchmark framing.

Your thread is the right one for the architecture work, and issue #9654 already tracks it.

If you have concrete latency/error numbers from agentic traffic after branch changes, post them in #9654 and we can tune the lane model against real workload data.

No extra action needed on this thread right now.

[diegosouzapw]

Thanks for the detailed proposal -- the admission-control/load-shedding work you're referencing has moved forward since this was posted: see #9176 (bounded phase-aware admission replacing the binary heavyweight-chat lease) and #10054 (adaptive admission promoted to the primary multi-agent scheduling path). If you're still seeing 503/502 subagent drops on a current version, please open a fresh issue with your version and a reproduction -- the architecture has changed enough that the original analysis may no longer map 1:1.