Repository navigation
v0.1.0: one API for Groq, Gemini, and Ollama, with automatic failover
Pre-releaseModelSwitch is a self-hosted gateway written in Rust. Your app calls one
API, and ModelSwitch handles the providers behind it. If one fails, your
request moves to the next.
This is the first release. It's ready to try with your own apps, but
not ready to expose to the public (see Limitations).
Highlights
One API, several providers. Send requests to POST /v1/generate
using Groq, Google Gemini, or a local Ollama model. Changing provider is
a config change, not a code change.
Automatic failover. Define an alias such as fast with an ordered
list of models. Brief failures get one retry, then ModelSwitch moves
down the list. Every request has a limit on attempts and total time, so
it can't hang or retry forever. Need one exact model? Pin it with
groq/openai/gpt-oss-20b, and it will never switch.
See what happened. Every response tells you which provider answered
and how many attempts it took. Each request and attempt is saved to
PostgreSQL with its outcome, latency, and token counts as reported by
the provider. Prometheus metrics are available on a separate internal
port.
Private by default. Prompts and responses are never logged or
stored. API keys are stored only as hashes, and revoking one takes
effect immediately.
Get started
You need Rust and PostgreSQL. The README walks through
setup, and the built-in echo provider lets you try the API without any
provider account.
Limitations
- Don't expose it to the public yet. There are no rate limits or
spending caps, so anyone with a valid key can use your provider
credit without limit. - Text only, with no streaming.
- No cost estimates yet. Token counts are recorded.
- Ollama support hasn't been tested against a live server yet.
- If a client disconnects mid-request, that request isn't recorded.
- On Gemini's free tier, the newest models can be too slow for the
default timeout.
Rate limits and spending caps are planned next. See the
roadmap.