Skip to content

feat: route returns RoutingOutcome #459

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
grahamking merged 4 commits into from
Aug 18, 2026
Merged

feat: route returns RoutingOutcome #459

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
c9aeb73
feat: `route` returns `RoutingOutcome`
grahamking Aug 17, 2026
0eeadeb
Feedback
grahamking Aug 17, 2026
0a2c466
Update AdvisorGate to RoutingOutcome
grahamking Aug 18, 2026
fa098e9
Fix ruff error
grahamking Aug 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 9 additions & 11 deletions crates/libsy-llm-client/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,9 @@ headers, makes the call with a shared `reqwest::Client`, and decodes the reply
back into a [`switchyard_protocol::Response`] — buffered or streamed.

It also pairs the client with a libsy algorithm: [`run`] drives
[`Algorithm::run_stream`] and serves every model call the algorithm offloads, so a
host that just wants the answer never has to drive the step stream itself.
[`Algorithm::run_stream`], serves routing-time calls, and consumes the terminal routing outcome.
When routing has not already produced the answer, `run` makes the terminal call and owns backend
retries plus ordered candidate fallback.

It depends on `switchyard-libsy`, `switchyard-protocol`, and
`switchyard-translation`; no server, no provider SDK.
Expand Down Expand Up @@ -143,10 +144,10 @@ async fn stream(

### Routing an algorithm

[`run`] takes a libsy algorithm and a [`ClientRouter`], and returns the final response plus
the trace of decisions the algorithm published. Each offloaded `CallModel` carries an ordered
`models` list. The router resolves and tries those candidates in order; `ClientRouter::single`
is the single-provider case:
[`run`] takes a libsy algorithm and a [`ClientRouter`], and returns the algorithm-selected
[`ModelId`] plus the final response. Routing-time `CallModel`s are served while the algorithm
runs. The terminal outcome either already contains the answer or supplies the selected model and
ordered fallbacks for the client to try. `ClientRouter::single` is the single-provider case:

```rust
use std::sync::Arc;
Expand All @@ -160,12 +161,9 @@ async fn route(
request: Request,
) -> switchyard_libsy::Result<String> {
let clients = ClientRouter::single(client);
let (trace, _response) =
let (selected_model, _response) =
switchyard_llm_client::run(algorithm, clients, request, None).await?;
Ok(trace
.last()
.map(|decision| decision.selected_model_id().to_string())
.unwrap_or_default())
Ok(selected_model.to_string())
}
```

Expand Down
11 changes: 8 additions & 3 deletions crates/libsy-llm-client/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,9 @@
//! streamed responses.
//!
//! [`run()`] pairs the client with a libsy algorithm: it drives
//! [`switchyard_libsy::Algorithm::run_stream`] and serves every model call the algorithm
//! offloads, so a host that just wants the answer does not have to drive the step stream
//! itself.
//! [`switchyard_libsy::Algorithm::run_stream`], serves routing-time calls, and makes the terminal
//! answer call from the routing outcome when needed. A host that just wants the answer does not
//! have to drive the step stream itself.

pub mod backend;
pub mod client;
Expand All @@ -32,3 +32,8 @@ pub use observation::{LlmCallObservation, RunObservation, RunObserver};
pub use raw::RawResponse;
pub use run::{ClientRouter, run};
pub use switchyard_translation::RawEventStream;

/// Registers process-wide compatibility gauges with the global meter provider.
pub fn initialize_metrics() {
metrics::initialize();
}
97 changes: 86 additions & 11 deletions crates/libsy-llm-client/src/metrics.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,39 @@
//! Metric labelling inherited from Python

use std::time::Duration;
use std::{
sync::OnceLock,
sync::atomic::{AtomicU64, Ordering},
};

use opentelemetry::metrics::ObservableGauge;
use opentelemetry::{KeyValue, global};
use switchyard_libsy::Result;
use switchyard_protocol::{ModelId, Response};

static TOTAL_REQUESTS: AtomicU64 = AtomicU64::new(0);
static TOTAL_ERRORS: AtomicU64 = AtomicU64::new(0);
static TOTAL_GAUGES: OnceLock<(ObservableGauge<u64>, ObservableGauge<u64>)> = OnceLock::new();

/// Registers process-wide compatibility gauges with the installed global meter provider.
pub fn initialize() {
TOTAL_GAUGES.get_or_init(|| {
let meter = global::meter("switchyard");
let requests = meter
.u64_observable_gauge("switchyard.total_requests")
.with_callback(|observer| {
observer.observe(TOTAL_REQUESTS.load(Ordering::Relaxed), &[]);
})
.build();
let errors = meter
.u64_observable_gauge("switchyard.total_errors")
.with_callback(|observer| {
observer.observe(TOTAL_ERRORS.load(Ordering::Relaxed), &[]);
})
.build();
(requests, errors)
});
}

pub(crate) const fn is_retryable_http_status(status: u16) -> bool {
status == 408 || status == 429 || (status >= 500 && status <= 599)
Expand Down Expand Up @@ -61,24 +92,68 @@ pub(crate) fn record_upstream_attempt(status: Option<u16>) {
);
}

/// Records what routing cost on top of the call that served the run: classifier
/// calls, target resolution, and decision publishing.
pub(crate) fn record_routing_overhead(
algorithm: &str,
run: Duration,
call_duration: Duration,
) -> Duration {
// Saturating: the two clocks start a moment apart, so a run that is all
// routed call can come out fractionally negative.
let overhead = run.saturating_sub(call_duration);
/// Records the time needed to produce the routing outcome, including classifier calls,
/// target resolution, request rewrites, and decision publishing.
pub(crate) fn record_routing_overhead(algorithm: &str, overhead: Duration) {
global::meter("switchyard")
.f64_histogram("switchyard.routing_overhead_ms")
.build()
.record(
overhead.as_secs_f64() * 1000.0,
&[KeyValue::new("algorithm", algorithm.to_string())],
);
overhead
}

/// Records one terminal model call made after routing, preserving the libsy call metric surface.
pub(crate) fn record_answer_call(
algorithm: &str,
selected_model: &ModelId,
duration: Duration,
result: &Result<Response>,
) {
let attributes = [
KeyValue::new("algorithm", algorithm.to_string()),
KeyValue::new("selected_model", selected_model.to_string()),
KeyValue::new("outcome", if result.is_ok() { "ok" } else { "error" }),
];
let meter = global::meter("switchyard");
meter
.u64_counter("switchyard.llm_calls")
.build()
.add(1, &attributes);
meter
.f64_histogram("switchyard.llm_call_duration_ms")
.build()
.record(duration.as_secs_f64() * 1000.0, &attributes);
}

/// Records one terminal routed request after the algorithm has produced an outcome.
pub(crate) fn record_routed_request(
selected_model: &ModelId,
answer_duration: Option<Duration>,
result: &Result<Response>,
) {
TOTAL_REQUESTS.fetch_add(1, Ordering::Relaxed);
let attributes = [KeyValue::new("model", selected_model.to_string())];
let meter = global::meter("switchyard");
if result.is_ok() {
meter
.u64_counter("switchyard.requests")
.build()
.add(1, &attributes);
if let Some(duration) = answer_duration {
meter
.f64_histogram("switchyard.model_call_latency_ms")
.build()
.record(duration.as_secs_f64() * 1000.0, &attributes);
}
} else {
TOTAL_ERRORS.fetch_add(1, Ordering::Relaxed);
meter
.u64_counter("switchyard.errors")
.build()
.add(1, &attributes);
}
}

#[cfg(test)]
Expand Down
8 changes: 4 additions & 4 deletions crates/libsy-llm-client/src/observation.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,11 @@ use std::time::Duration;

use switchyard_protocol::{ModelId, Usage};

/// One completed model call observed at the algorithm offload boundary.
/// One completed model call observed while serving an algorithm run.
#[derive(Clone, Debug)]
pub struct LlmCallObservation {
/// Model selected for the completed call.
pub selected_model: ModelId,
/// Whether this call generated an answer rather than a routing verdict.
pub is_answer_call: bool,
/// Whether the call completed successfully.
pub is_success: bool,
/// Time spent waiting for the model call to resolve.
Expand All @@ -26,8 +24,10 @@ pub struct LlmCallObservation {
/// One request-scoped observation emitted by the algorithm runner.
#[derive(Clone, Debug)]
pub enum RunObservation {
/// A completed model call.
/// A completed model call requested by the algorithm for routing work.
LlmCall(LlmCallObservation),
/// A completed terminal model call made from the routing outcome.
AnswerCall(LlmCallObservation),
/// Routing time recorded by the `switchyard.routing_overhead_ms` metric.
RoutingOverhead(Duration),
}
Expand Down
Loading
Loading