diff --git a/README.md b/README.md index d84cae242..45cf3cbde 100644 --- a/README.md +++ b/README.md @@ -26,6 +26,7 @@ algorithm you write yourself. Choose the launcher path to run Claude Code, Codex CLI, or OpenClaw through Switchyard. Choose the server path to run Switchyard as a standalone proxy. +Choose the library path to embed routing in your own Rust application. ### Launcher Path @@ -92,6 +93,36 @@ curl http://localhost:4000/health For a complete configuration and a test request, follow [Getting Started](docs/getting_started.md). +### Library Path + +`switchyard-libsy` embeds the routing algorithms in your own Rust application. +It never calls a model itself: an algorithm decides which target to use and +hands every model call back to you, so it drops into an existing proxy, gateway, +or agent runtime without owning an HTTP stack. Pair it with +`switchyard-llm-client` when you want the calls made for you. + +```toml +[dependencies] +switchyard-libsy = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git" } +switchyard-protocol = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git" } +``` + +See [Getting Started](docs/getting_started.md#library-path) for setup and the +algorithm list, or the [`switchyard-libsy`](crates/libsy/README.md) crate docs. + +## Routing Strategies + +| Strategy | Use it when | Route `type` | +|---|---|---| +| [LLM Classifier](docs/routing_algorithms/llm_classifier_routing.md) | Request content should decide whether a turn needs the weak or strong tier. | `llm_classifier` | +| [Stage Router](docs/routing_algorithms/stage_router_routing.md) | Signals already in the conversation, such as tool results and errors, should route most turns without an extra model call. | `stage_router` | +| [Escalation Router](docs/routing_algorithms/escalation_router_routing.md) | Every turn runs on the weak tier first, and a judge reads that answer to decide whether to send the same request to the strong tier. | `llm_classifier` with `mode = "escalation"` | +| [Random](docs/routing_algorithms/random_routing.md) | You need a fixed traffic split for A/B tests, baselines, or cost experiments. | `random` | + +A `passthrough` route registers one target under one model ID with no routing +decision. See the [Routing Overview](docs/routing_algorithms/overview.md) for +the common route shape and self-hosted targets. + ## Architecture ```mermaid diff --git a/docs/getting_started.md b/docs/getting_started.md index 7519bada6..b737cf01d 100644 --- a/docs/getting_started.md +++ b/docs/getting_started.md @@ -1,11 +1,13 @@ # Getting Started with Switchyard -Switchyard has two native Rust execution paths: +Switchyard has three native Rust execution paths: - **Launcher path:** install the Python-distributed CLI and launch Claude Code, Codex, or OpenClaw through the packaged Rust server binding. - **Server path:** build and run the standalone Rust server for API clients and custom deployments. +- **Library path:** embed the routing algorithms directly in your own Rust + application with `switchyard-libsy`. ## Launcher Path @@ -233,7 +235,50 @@ export SWITCHYARD_TELEMETRY_OPT_OUT=1 --- -### Next steps +## Library Path + +Use this path when you want routing inside your own Rust application rather than +behind a proxy. `switchyard-libsy` never calls a model itself: an algorithm +picks a target and hands the model call back to you. + +### Add the dependencies + +```toml +[dependencies] +async-trait = "0.1" +futures = "0.3" +switchyard-libsy = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git" } +switchyard-protocol = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git" } +tokio = { version = "1", features = ["macros", "rt"] } +``` + +### Choose an algorithm + +| Type | Purpose | +|---|---| +| `LlmTaskClassifier` | Ask a judge model to choose an efficient or capable target. | +| `StageRouter` | Route from signals already in the conversation, such as tool results and errors, with an optional judge fallback. | +| `LlmTaskClassifier` with escalation | Every turn runs on the efficient target first, and a judge reads that answer to decide whether to send the same request to the capable target. | +| `Random` | Select among any number of targets, uniform or weighted. | + +These are the same strategies the server exposes as route types, so a deployment +can move between the server and library paths without changing routing +behaviour. + +### Drive the algorithm + +An algorithm yields a stream of steps. Each `Step::CallLlm` is a model call your +host performs over its own transport, and the run ends with +`Step::ReturnToAgent` carrying the final response. Serving those calls yourself +is what lets libsy embed in a host that already owns its HTTP stack, retries, +and credentials. + +For the request, response, and streaming types the steps carry, see +[`switchyard-protocol`](../crates/protocol/README.md). + +--- + +## Next steps - [Core Concepts](core_concepts.md): LLM clients, targets, and routes - [`switchyard-server`](../crates/switchyard-server/README.md): server configuration,