Adaptive Routing

🌍 View in other languages

Adaptive Routing: Routing Events, Quality Feedback & Explainability

This document describes the feedback-driven adaptive routing foundation added to OmniRoute. It is deliberately small: it introduces a typed routing-outcome channel, an online quality signal that feeds the existing auto-combo scorer, an optional OpenTelemetry exporter, and an explainability endpoint. It does not replace the existing resilience stack (circuit breaker, connection cooldown, model lockout, health matrix, autopilot) — it complements it.

1. Architectural context

OmniRoute is a data plane with a request hot path and a control/intelligence plane. The hot path must stay fast, memory-efficient, asynchronous, resilient and predictable. Evaluation, quality scoring, experiments and historical analysis belong to the control plane.

AI Agent / IDE
      │
      ▼
┌─────────────────────┐
│    OmniRoute        │   data plane (fast, sync, in-memory)
│  routing / failover │
│  health / guardrail │
│  cache / streaming  │
└──────────┬──────────┘
           │ RoutingEvent (fire-and-forget, ~0.2µs)
           ▼
┌─────────────────────┐
│  Feedback sinks     │   control plane (async, best-effort)
│  quality tracker    │
│  OTel exporter      │
│  explain store      │
└──────────┬──────────┘
           ▼  quality score
      auto-combo scorer

What was already there (audited, not duplicated)

Concept	Existing implementation
Availability (can we send traffic?)	Circuit breaker (CLOSED/DEGRADED/OPEN/HALF_OPEN, DB-persisted), connection cooldown, model lockout
Health reporting	providerHealthMatrix.ts, providerHealthAutopilot.ts
Shadow traffic	open-sse/services/combo/shadowRouting.ts
Guardrails	src/lib/guardrails/ (pre/post hooks)
Exact cache	src/lib/semanticCache.ts (signature-based)
Evaluators / eval-driven routing	src/lib/evals/, open-sse/services/evalRouting.ts
Combo decision explainability	open-sse/services/combo/decisionTrace.ts
Dashboard real-time events	src/lib/events/eventBus.ts (UI notification channel, unknown payloads, 100-entry history)

The routing-event layer is not a re-implementation of eventBus: that bus is the dashboard's real-time notification channel (typed event names, opaque payloads, UI consumers). RoutingEvent is a typed outcome struct (latency/tokens/cost/outcome/finish-reason) consumed by the control plane's feedback sinks (quality tracker, OTel exporter, explain store).

What was missing (added here)

1. A typed routing-outcome event + sink abstraction (RoutingEvent / RoutingEventSink). decisionTrace is combo-scoped and in-memory-only; comboMetrics are cumulative counters; call_logs is raw async persistence. None is a typed, sink-based outcome channel that a quality tracker, an OTel exporter, or a Future-AGI-style evaluator can subscribe to.
2. An online quality signal (EWMA) for output quality — the scorer previously proxied "quality" only through static task fitness and opt-in eval pass-rates.
3. An optional, dependency-free OTel exporter using GenAI semantic conventions.
4. An explainability endpoint returning the real routing decisions + quality state.

2. Routing Events (feedback foundation)

Files: open-sse/services/routing/events.ts, .../index.ts

A RoutingEvent carries only routing metadata:

interface RoutingEvent {
  requestId: string;
  provider: string;
  model: string;
  strategy: string; // "auto" | "priority" | "direct" | ...
  latencyMs: number;
  ttftMs: number | null;
  inputTokens: number | null;
  outputTokens: number | null;
  cost: number | null;
  retries: number;
  fallbackUsed: boolean;
  outcome: RoutingOutcome; // allowlisted union
  status: number | null;
  finishReason: string | null;
  connectionId: string | null;
  ts: number;
}

RoutingEventSink is a Send+Sync-style trait in TypeScript:

interface RoutingEventSink {
  readonly name: string;
  record(event: RoutingEvent): void; // must be O(1), no sync I/O
}

The hot path calls emitRoutingEvent(event) once per completed request (the streaming-completion callback, the non-streaming success path, and the malformed-200 failure path in handleChatCore). Dispatch is synchronous fan-out to registered sinks, but each sink only enqueues/updates in-memory state. No synchronous database writes, no network I/O on the hot path.

Default sinks:

• MemoryRoutingEventStore — bounded (500) ring buffer, newest-first, for the explain endpoint.
• QualityTracker consumer — updates the EWMA quality estimate.
• OtlpHttpsEventSink — optional, enabled only when OMNIROUTE_OTEL_ENDPOINT (or OTEL_EXPORTER_OTLP_ENDPOINT) is set.

Measured overhead (honest comparison)

npm run bench:routing-events on this workstation (100k iterations; sub-µs ops measured as aggregate µs/op because per-op percentiles are below performance.now() timer resolution):

Scenario	µs/op	ops/s
baseline (scoring only)	~0.045	~22 M
baseline + RoutingEvent (2 sinks)	~0.168	~5.9 M
baseline + event + OTel enqueue	~0.163	~6.1 M
concurrent (8 interleaved bursts)	~0.18	—

The event-dispatch delta over baseline scoring is ~0.12 µs/request; the OTel sink only enqueues (O(1) buffer push), adding nothing measurable. These numbers are machine-specific and relative — not a production guarantee. The v1 "~0.2 µs" figure was an aggregate estimate; this methodology separates the scoring baseline from the event-dispatch cost.

3. Quality Signal (feedback-driven provider state)

Files: open-sse/services/routing/quality.ts

v2 separates operational from semantic quality:

• Operational — derived from the routing hot path (HTTP 4xx/5xx, connection failures, 429s, malformed responses, stream interruptions, finish_reason=length, zero-output successes, latency/TTFT EWMA). A 200 is NOT treated as semantic quality.
• Semantic — the actual value of the generated output. ONLY ever produced by an evaluator via setSemanticQuality(). It is null until one provides it and never leaks into the operational score.

Per-(provider, model) state (EWMA + bounded counters):

• successEwma — EWMA (α=0.2) of outcome success.
• latencyEwma / ttftEwma — EWMA of latency (α=0.1).
• samples, anomalies, rateLimited, semantic, semanticConfidence.
• recencyMs — how recently the model was last observed.

Confidence / sample awareness

confidence = clamp01(samples / 50), and the score returned to the scorer is blended toward the neutral midpoint:

score = 0.5 + confidence * (operational - 0.5)

Consequences (verified by tests):

• A cold provider (0 samples) scores 0.5 — not unfairly penalized, but unable to dominate a provider with thousands of solid observations.
• A provider with 7 lucky successes is pulled toward 0.5 (never dominates from optimistic initialization).
• A provider with 50+ samples converges to its true operational score.
• Degradation and recovery are gradual (EWMA), and one isolated failure does not destroy a healthy provider.

ProviderQuality exposes { operational, semantic, confidence, samples, anomalies, rateLimited, successEwma, latencyEwmaMs, ttftEwmaMs, recencyMs }.

This feeds the auto-combo scorer as the quality scoring factor:

• ScoringFactors.quality / ScoringWeights.quality in open-sse/services/autoCombo/scoring.ts.
• DEFAULT_WEIGHTS: health 0.1905 → 0.1605, quality 0.03. Sum stays 1.0.
• buildAutoCandidates populates candidate.quality from the tracker; candidates without data default to neutral 0.5 (a cold candidate is neither boosted nor penalized).

The closed loop:

RoutingEvent → QualityTracker → getQualityScore → auto-combo quality factor
      ↑                                                    │
      └────── request outcome (handleChatCore) ←────────────┘

Hard exclusion vs soft penalty

The quality signal is a soft adaptive preference only. Hard exclusion stays with the existing resilience stack: circuit breaker OPEN, quota exhausted, auth failure, model lockout — none of these are affected by the quality score. A provider whose quality score dips temporarily is de-preferenced, never hard-disabled.

3b. Canonical stream timing (TTFT / ITL)

Files: open-sse/utils/streamTiming.ts

createStreamTiming() is the single instrumentation seam for the streaming path, wired into createSSEStream (open-sse/utils/stream.ts):

• markByte() — first upstream chunk received.
• markForward() — first chunk forwarded to the client (used for TTFT).
• markInterrupted() — stream timeout/abort/error before a clean finish.
• ttft() = first-forwarded-SSE-chunk latency. This is NOT token-level TTFT — a single SSE chunk may carry zero/one/many tokens. Documented precisely.
• avgItlMs() = mean inter-chunk gap (a chunk-latency proxy for ITL).

TTFT/ITL/interrupted flow into the RoutingEvent (ttftMs, itlMs) and are exported as GenAI/OmniRoute span attributes by the OTel sink.

4. OpenTelemetry / GenAI observability

Files: open-sse/services/routing/otel.ts

• Dependency-free OTLP/HTTP JSON exporter (uses global fetch, no @opentelemetry/* SDK).
• Spans follow GenAI semantic conventions (gen_ai.provider.name, gen_ai.request.model, gen_ai.usage.input_tokens/output_tokens, gen_ai.completion.finish_reason, gen_ai.system) plus OmniRoute routing attributes (outcome, status, ttft, retries, fallback).
• record() only enqueues into a bounded buffer (O(1)); a background timer flushes via POST {endpoint}/v1/traces asynchronously. Under overload the oldest events are dropped (dropped counter) — never backpressure the data plane.
• Disabled unless configured. OMNIROUTE_OTEL_ENDPOINT (or OTEL_EXPORTER_OTLP_ENDPOINT) must be set; otherwise the sink is not registered and zero OTel code runs.

5. Explainability

• GET /v1/explain/routing returns the recent RoutingEvents (the real decisions, newest first) and the per-provider/model quality snapshot.
• Auth mirrors /v1/combos (Bearer API key or dashboard session; anonymous on single-user local deployments with REQUIRE_API_KEY=false).
• Combo-level per-invocation traces remain available via the existing decisionTrace.ts (header X-OmniRoute-Combo-Trace).
• Safety: events carry only routing metadata, never prompts/bodies/credentials.

6. Evaluation-plane integration (Future AGI readiness)

OmniRoute treats Future AGI (or any evaluator) as a potential intelligence/evaluation backend, not a dependency. The seams:

• A RoutingEventSink can forward events to an evaluator asynchronously.
• The MemoryRoutingEventStore + quality snapshot give an evaluator the raw decision stream.
• A future Evaluator (deterministic, local judge, HTTP, WASM) would consume events/traces and return a QualityScore that feeds the same getQualityScore/quality-factor path.
• Existing eval-driven routing (open-sse/services/evalRouting.ts) already re-orders combo targets by eval_runs pass-rates when enabled.

No evaluation runs synchronously on the request path, and the gateway operates fully with the evaluator absent.

7. Final architectural review

1. What remains on the synchronous hot path? Routing/scoring, guardrail pre-checks, cache lookup, and one emitRoutingEvent fan-out (~0.12 µs over baseline scoring) to in-memory sinks.
2. What moved to asynchronous processing? OTel export (timer + fetch), call_logs/usage persistence, semantic-cache writes, quality is in-memory and O(1) (no async needed).
3. How does a routing outcome become feedback? handleChatCore emits a RoutingEvent → QualityTracker updates EWMA state → getQualityScore feeds the auto-combo quality factor.
4. How does quality influence future routing? A low quality score reduces the weighted score of that provider/model in scoreAutoTargets, so degraded models are gradually de-preferenced and recover as their EWMA improves.
5. How can Future AGI integrate without becoming a dependency? Via the RoutingEventSink interface / a future Evaluator adapter — no hardcoded dependency.
6. What happens when the evaluator is unavailable? Routing is unaffected; quality falls back to neutral (1.0) for models with no observed signal.
7. What happens when telemetry is unavailable? The OTel sink simply isn't registered; the rest of the routing layer runs unchanged.
8. What happens under overload? The OTel buffer drops oldest events; quality and the ring buffer are bounded by construction; no backpressure.
9. How does provider state recover after degradation? EWMA re-converges as successes accumulate; warmup keeps cold models neutral; the circuit breaker independently recovers via HALF_OPEN probes.
10. Which proposed features were intentionally NOT implemented, and why?
    • Shadow traffic / experiments — already implemented (combo/shadowRouting.ts); not re-built.
    • Guardrails — already implemented (src/lib/guardrails/); not duplicated.
    • Semantic cache — already implemented (src/lib/semanticCache.ts); not duplicated.
    • A full experiment-management platform, dataset tooling, prompt-optimization platform, vector DB, or mandatory external OTel infrastructure — out of scope for a lean data plane.
    • A Rust RoutingEvent struct — the data plane is TypeScript; the TS type is the adapted equivalent.

8. Configuration reference

Variable	Default	Effect
OMNIROUTE_OTEL_ENDPOINT	unset	When set, enables the OTLP/HTTP traces exporter (e.g. http://collector:4318).
OTEL_EXPORTER_OTLP_ENDPOINT	unset	Fallback alias for the OTLP endpoint.
OTEL_SERVICE_NAME	omniroute	service.name resource attribute.

9. Tests

• tests/unit/routing-events.test.ts — event normalization, status classification, bounded ring buffer, sink fan-out + isolation.
• tests/unit/routing-quality.test.ts — EWMA warmup, failure/success recovery, anomaly penalties, 429 transient handling, snapshot, reset.
• tests/unit/routing-scoring-quality.test.ts — weight integrity, neutral default, quality factor ranking.
• tests/unit/routing-otel.test.ts — enable gating, GenAI span payload, async flush, drop-under-overload.
• tests/unit/routing-events-concurrency.test.ts — thousands of events, ring buffer boundedness, throwing-sink isolation, interleaved async bursts, reset-during-inserts.
• tests/unit/routing-adaptive-e2e.test.ts — deterministic end-to-end loop via the real scoreAutoTargets scorer: healthy → degrade → recover → blip, plus cold-start and lucky-cold-provider scenarios.
• tests/unit/stream-timing.test.ts — TTFT (first-forwarded-chunk), ITL, first-byte vs first-forward, interruption, malformed/empty chunk safety.

10. Pre-existing issues status (Phase 18)

Issue	Status	Notes
omniglyph export mismatch	FIXED (environmental)	node_modules was out of sync with package-lock.json (installed 1.3.1 vs locked 1.4.0). Running npm install omniglyph@1.4.0 restored the locked version; type errors dropped to 0. Manifests unchanged.
Stale getKnownContextOverflow tests	KNOWN — not fixed	combo-context-overflow-compression-probe.test.ts imports a function that no longer exists in open-sse/services/combo.ts (only comments reference it). Fixing requires re-implementing or re-writing those tests — unrelated architectural churn.
combo-runtime-unit-concurrency.test.ts DB isolation	KNOWN — not fixed	Test-harness SQLite-isolation assertion fails when run directly; fails identically on the base branch.
i18n llm.txt drift	KNOWN — not fixed	docs/i18n/*/llm.txt differ from root; pre-existing, blocks the docs-sync pre-commit gate.

Environmental vs code issues are kept distinct; no unrelated failures are hidden behind changed test filters.