Radicalbit AI Gateway is finally here!

[bigmoby]

Radicalbit AI Gateway is finally here!

👋 Today we are releasing the first public version of the Radicalbit AI Gateway.

The Gateway sits between your applications and your model providers. Simply point your code at a single endpoint, define your models and routes in a single YAML file, and the Gateway handles the rest: provider credentials, fallbacks, guardrails, caching, and limits. If you are running LLMs in production across more than one provider, this is the layer that keeps your infrastructure from turning into glue code scattered across every service.

📚 Documentation: https://docs.ai-gateway.radicalbit.ai/

What it does

🔀 Multi-strategy routing — keyword, token length, context length, time-of-day (cron), budget ratio, external text classification, and semantic routing via embedding similarity. Cost control becomes a routing decision.

🛡️ Guardrails in the config, not in the code — pattern matching (contains, starts_with, ends_with, regex), PII detection and redaction via Microsoft Presidio, and LLM-as-a-Judge with your own prompt templates. Enforced at the gateway layer, with block or warn behaviors and per-message-role scoping.

⚡ Caching — exact-match and semantic caching with configurable TTL, similarity threshold and distance metric, plus exact caching support for audio transcription requests.

🚦 Limits — rate, token and budget limiting per route, independently combinable, audio-duration-based limiting for transcription routes, and rate limiting for Model Context Protocol (MCP) servers.

🔄 Reliability — automatic model fallback chains that work across providers.

📊 Observability — 20+ Prometheus metrics, end-to-end OpenTelemetry tracing with ClickHouse storage and custom OTLP exporters (Jaeger, Tempo, …). A usage dashboard with cost and token breakdown by route, group, and key. Request tagging and tag-based trace filtering.

🖥️ Built-in UI — projects, configuration editor with load → approve → serve lifecycle, config import/export, groups and API keys, usage dashboard and trace inspection.

🤖 Configure in plain English — describe the route you want and get valid YAML, either from the Generate button in the UI or from your IDE via the Radicalbit Skills plugin.

🚨 Real-time Alerts & Notifications — proactive email notifications triggered on critical gateway events: guardrail interventions (input/output violations, PII detected/redacted, LLM judge breaches). Rules are configurable per route with instant delivery and customizable recipient lists.

🔌 Providers — native support for OpenAI, Anthropic, Google Gemini, DeepSeek, Mistral and Azure OpenAI, plus any OpenAI-compatible endpoint (Ollama, vLLM, OpenRouter, on-premises deployments).

🔌 MCP proxy — expose Model Context Protocol servers through the same route, the same API key with the same trace view as your models. One endpoint per route fans out across several upstreams over Streamable HTTP or stdio, namespacing tools as {alias}__{tool}.

What a route looks like

chat_models:
  - model_id: primary
    model: openai/gpt-4o
    credentials:
      api_key: !secret OPENAI_API_KEY
  - model_id: backup
    model: anthropic/claude-3-5-sonnet
    credentials:
      api_key: !secret ANTHROPIC_API_KEY

routes:
  customer-service:
    chat_models: [primary, backup]
    guardrails: [presidio_analyzer, presidio_anonymizer]
    fallback:
      - target: primary
        fallbacks: [backup]
    rate_limiting:
      algorithm: fixed_window
      window_size: 30 seconds
      max_requests: 20

That is the whole setup. Start the Gateway, send your requests to the customer-service route, and you get PII filtering, a fallback chain, and a rate limit without a line of application code.

Tell us what breaks

We built this based on real production needs, but yours might be different.

• Got feedback or questions? Open a discussion here if something is missing, unclear, or behaves in a way you did not expect.
• Found a bug? Please report it under Issues.
• Need setup advice? Questions about a specific provider or deployment setup are always welcome too!

[mmariniello90]

🚀 Let's go!

[Rehanguards]

Congrats on the launch! How are you handling latency overhead on the guardrails hot-path within the gateway? If you're looking for ultra-fast, sub-millisecond deterministic checks to drop bad inputs before hitting LLM routes, I'd love to see if we can collaborate on a native integration!

[bigmoby]

Really appreciate the energy and the effort you're putting into this, the benchmark data is great to see! 🙌

For something like a native integration, it'd help a lot to have a bit more structure around the proposal so we (and anyone else following this thread, of course) can properly evaluate it.
Would you be up for opening a new Discussion with:

• An overview of what you have in mind and how it would plug into the existing guardrail pipeline (in-process vs. sidecar, how it'd sit alongside our guardrails: config, block/warn behaviors, etc.)

• A bit more detail on the benchmark setup (payload shapes, hardware, how it compares against the baseline without the extra check)

That'll give the proposal a proper home and make it much easier to move forward on.

Thanks again for the interest and the work you're putting in!
Looking forward to seeing it! 🚀

[Rehanguards]

Thanks @bigmoby! Appreciate the feedback.

I'll put together a structured RFC in a new Discussion thread covering the in-process pipeline integration, config/block-warn semantics, and detailed baseline comparison metrics. Will link it here as soon as it's posted! 🚀

[Rehanguards]

@bigmoby I've published the detailed RFC covering the pipeline architecture, execution modes, and comparative benchmarks here: [https://github.com//discussions/116]

Looking forward to your thoughts on this! 🚀

[Rehanguards]

Hello are you here?

[bigmoby]

Hi! Yes, I’m here...and I’ve read the new Discussion and my colleague’s detailed response, which reflects our current perspective as a team. Thanks for taking the time to structure the proposal! 👍
Let’s continue in #116 so we can keep the conversation focused in one place.