Launching the Langfuse AI Gateway

[hassiebp]

We're launching the Langfuse AI Gateway in public beta

The AI Gateway sits between your apps or coding agents and your model providers. Point your client at the gateway instead of OpenAI or Anthropic and use a Langfuse gateway key instead of a provider key. The request itself stays the same. Every call is recorded in Langfuse, with no SDK or instrumentation needed.

Features

• Gateway keys: give each person a user key and each CI job or shared service a service account key. Provider credentials stay in Langfuse, so you can rotate a provider key or revoke one person's access without touching anyone else. Add key metadata such as team or cost_center to break spend down the way your company is organized.
• Model access: choose which models each provider credential allows. Clients call a model as <credential>:<model>, and coding agents can keep sending plain provider model names. See model IDs.
• Analytics: cost, requests, and tokens per user, API key, and model. Owners and admins see the whole organization. People with the AI Gateway role see their own usage.
• Tracing: every call becomes a Langfuse generation, so you can use the same search, evaluations, annotation, and dashboards you already use. You choose what is recorded with the ingestion mode. USAGE (the default) records the model, tokens, cost, and latency. FULL adds prompts and outputs. Self-hosted deployments can also use NONE, which only routes requests.
• APIs: the native OpenAI Responses, OpenAI Chat Completions, and Anthropic Messages APIs, including streaming. The gateway doesn't convert between formats, so keep the format your client already uses. See the REST API reference.
• Integrations: the OpenAI SDK, Anthropic SDK, and Vercel AI SDK. For coding agents: Claude Code, Codex, GitHub Copilot, OpenCode, and Pi. You can also connect any other third party solution that allows you to configure a custom base URL for your LLM calls.
• Where it runs: All Langfuse Cloud regions, each with its own gateway host. Traces are stored in your organization's region. You can also self-host the gateway with the Helm chart.

Get started

An admin sets up the gateway, then everyone else sends a first request. For how keys, credentials, and routing fit together, see core concepts.

Good to know during the beta:

• Behavior, routes, and headers may change.
• On Langfuse Cloud, each organization has per-minute rate limits based on its plan. They may change during the beta. Contact us if you need more.
• Each recorded generation counts as a billable unit, and your provider bills model usage to your own provider account. See how gateway usage is billed.
• Check the known limitations before you rely on gateway traces for audits or cost reporting.

What's next

• Spend limits: enforce budgets on the gateway
• Fallbacks and smart model routing
• Guardrails and redaction
• More providers: Bedrock, Vertex / AI Studio support as well as custom upstreams that support the Chat Completions, Responses, or Messages APIs

Please share your feedback!

We'd love to hear how it works for you. Reply in this thread with:

• what works, what's confusing, what's missing
• what would make the gateway useful for your team. Please 👍 existing requests rather than duplicating them.
• for issues, please see how to report a gateway issue. Common errors and fixes may already have the answer.

Thank you for trying out the Langfuse AI Gateway 🫶🏾