Repository navigation
Otoroshi LLM Extension — v0.0.90
Breaking changes
- Otoroshi 18 only — the extension now builds with Scala 3 for Otoroshi 18, based on Play 3 and Pekko
(#198). This release needs Otoroshi18.0.0or later: deployments still on Otoroshi 17 stay on0.0.86.
The fat jar no longer embeds the Scala standard library, Otoroshi provides the very same one.
Added
AI Studio
- AI Studio — a console for the teams that build with AI, served at
/extensions/cloud-apim/ai-studio
and reachable from the backoffice home page (#204). In a few clicks a team gets a workspace: an
OpenAI-compatible endpoint, its own providers and provider keys (BYOK, with their health), models, API keys,
users, guardrails, routing, activity, logs, and reporting with costs and environmental impact. A workspace is
made of regular Otoroshi entities, so what is done in the studio shows in the backoffice and the other way
round. It is turned on from the danger zone (AI Studio section), and its activity pages read the active
User Analytics (PostgreSQL) exporter. AI Studio is experimental. - Chat and playgrounds — conversations, regenerate, models compared side by side, attached files, tool
selection, cost simulation and answers rendered as markdown (#206). Every other model type has its
playground (images, audio, embeddings, moderation, OCR, video, decisions), with image edition refined step by
step (#220), a Responses playground showing the summary of the reasoning, a Translation playground for the
audio models that translate, and an MCP playground calling the tools a workspace exposes (#222). An API page
explains how to call the workspace. - Tools and MCP — MCP servers and connectors per workspace, tool functions with a
web_fetchtemplate, and
HTTP tool calls that can convert what they fetch to markdown (#207). The MCP server of a workspace can be
exposed in meta mode (list_servers,list_tools,get_tool_schema,search_tools,execute), with an
optional semantic tool search. A new tool is offered to every provider, and a new provider to every tool. - Admin API for workspaces — everything the studio does on a workspace is an admin API route of the
extension (/api/extensions/cloud-apim/extensions/ai-extension/studio/workspaces/...), to create and manage
workspaces without the UI (#205). Every route checks the rights of its caller the way the Otoroshi admin API
does: a service account limited to a tenant manages the workspaces of that tenant, a member only sees the
pages they can read and their own usage, and secrets come back masked.
Decision models
- Decision models — a new kind of model answering typed questions (
noul,choice,score) about a
state, with probabilities, through the System One API (Jev and alike) (#208). Native System One servers,
Cloudflare Workers AI, or any text provider answering the questions. They come with their entity, a
/systemoneendpoint and plugin, a guardrail, a workflow function, fallbacks, budgets and an AI Studio
playground. - OpenAI decisions API — decision models also answer
POST /v1/decisions, on the/decisionspath of the
LLM OpenAI Compatible API plugin and with the newOpenAI decisions backendplugin, in the OpenAI format
whatever provider serves them (#224). OpenAI is a new decision model provider (gpt-6-luna), a question a
model declines comes back as arefusal, which the decision guardrail denies. - Pictures in decisions — pictures go to the decision models that look at them (OpenAI, Cloudflare,
self-hosted servers, Jev 1.14 as TypeSafe assets with theTypeSafe-Danger-Zone-*headers) and are refused
with a400by the others (#230)./systemonereads TypeSafe assets, and gzip, zstd or multipart bodies. - Decision router workflow node — sends a workflow down the path a decision model picks, with a
min_confidenceunder which a default path runs instead (#214). Available in the workflow designer.
Routing
- Smart and intent routers — two new routing models of the Otoroshi router, next to
code-router,
auto-routerandfusion-router(#217).smart-routerhas a decision model rate how demanding a request is
and sends it to the cheapest candidate good enough for it;intent-routerpicks the candidate whose
description fits the request. Both keep answering when the decision model does not, and are configured from
the providers page or the Routing page of AI Studio. - Coding index of the code router — the Artificial Analysis Coding Index is now a bundled data file of 258
models (27 before), refreshed with the other data files byscripts/update-data-files.mjs(#216). Model
names are matched word by word, so a model is never given the score of an earlier version. - A specific model wherever a provider is referenced — semantic cache, guardrails, search engines, agents
and plugins can name the model to use next to the provider, instead of its default model (#203).
Budgets and costs
- Budgets per consumer —
scope.partition_bytakes an Otoroshi expression and counts the budget apart for
each of its values (#227):${user.email}gives everyone their own limit,${apikey.metadata.team}each
team,${ctx.ai.model}each model. The admin API lists the consumers of a budget (/partitions) and reads
or resets one of them (?partition=), workers push what each consumer consumed, and the analytics show a
line by consumer. - Budget extensions — a budget can allow extensions that the people it counts take themselves
(POST …/budgets/:id/consumption/_extend), up to a number per cycle set by the administrator (#227). The
limit grows for the current cycle only and every extension is audited. AI Studio offers both, and its chat
proposes the extension when a message is refused. - Custom budget error message —
error_messagein the action on exceed of a budget, returned to the
client of a blocked call (#196). The response always carriesbudget_exceeded: true. - Cost tracking for every model type — image, audio, embedding, moderation, OCR and video models are now
priced, not only chat models (#199). The costs reported by OpenRouter are used as they are,
require_known_costsmakes a provider refuse (and hide) the traffic it cannot bill, and the
costs-tracking-provider/costs-tracking-modelmetadata work on every kind of model. - Provider quota alerts — an Otoroshi alert when a provider starts refusing calls, telling throttling
(LLMProviderQuotaExceededAlert) from an account out of credit (LLMProviderCreditExhaustedAlert), and
another one when it serves again (#200). A provider known to be refusing is skipped until the time it gives
(Retry-After,x-ratelimit-reset-*).
Models and providers
- Reasoning continuity — the reasoning of the previous assistant turns, whatever field it comes in
(reasoning,reasoning_content,reasoning_details, Anthropicthinkingblocks,/responsesreasoning
items), is sent back to each provider the way it takes it:reasoning_detailsfor OpenRouter, signed
thinkingblocks for Anthropic,reasoning_contentfor DeepSeek and vLLM / SGLang servers,thinking
chunks for Mistral,reasoningfor Groq and Ollama, reasoning items on native/responses, and nothing for
the APIs that refuse it (#228). Answers carry it the OpenRouter way, asthinkingblocks on/messagesand
as reasoning items on/responses, streamed too. The tool loops run by the gateway, provider contexts, cache
keys and persistent memory keep it, and a signed reasoning is never sent changed. - Models catalog and metadata —
/models?enriched=truedescribes each model (kind, endpoints,
capabilities, known price), withkind,endpointandhas_costfilters (#202). The LiteLLM price table,
the models.dev catalog and the EcoLogits data are bundled, and refreshed with one script. - One
/modelslisting for every model type — the models endpoint of the unified API lists the chat,
embedding, image, audio, moderation, OCR and decision models of the route in one call, asking each
connection once. - Arkane Cloud — new provider (#209).
MCP
- MCP 2026-07-28 — the Streamable HTTP endpoint (
McpRespEndpoint) serves the stateless revision of MCP
withprotocol_version: "2026-07-28", while clients opening withinitializekeep being served2025-11-25
on the same endpoint (#192). No session, so any Otoroshi instance answers any call;MCP-Protocol-Version,
Mcp-Method,Mcp-NameandMcp-Param-*headers checked against the body, so a load balancer, a WAF or a
plugin can route on them;ttlMs/cacheScopecaching hints;server/discoverandsubscriptions/listen.
MCP connectors get a matchinghttp_2026_07_28transport, and thehttpone takes aprotocol_version
metadata.
Observability and data protection
- Built-in dashboards — with an active User Analytics (PostgreSQL) exporter, the extension installs ten
dashboards in the Otoroshi user dashboards: overview, costs, tokens & cache, budgets, ecological impact,
performance, reliability, consumers, MCP overview, MCP tools & security (#194). Every widget is a query you
can reuse in your own dashboards and alert on. - Event redaction with Rampart — a data exporter transformer removing personal data from the events of the
gateway with the Rampart model, before they leave Otoroshi (#201). The credentials of the entities never
enter an event.
Fixed
- Fallbacks, load balancers and routers — a call handed over to another provider was counted twice on
budgets, in metrics and in the environmental impact, two providers falling back on each other never ended, a
failing fallback was called twice, and a handed over stream ended with two usage chunks (#210). Such a call
is now priced, measured and metered once, by the provider that served it, and counts for the budgets of both. - Calls made on behalf of another — the calls a guardrail, a knowledge base search, a tool, a semantic
cache, a router or a fusion panel make are counted for the model that served them and for the caller, and
the call that triggered them keeps its own provider, usage and cost (#211). The usage reported by moderation
providers is read. - Budgets in cluster mode — a leader rejected every budget delta of a worker of a previous version, a
worker lost what it consumed while pushing its deltas, and a delta was counted in the current cycle whatever
the cycle it was consumed in (#212). - Streamed calls failing with
limit of 13 reached— a non-200 answer to a streamed call hid the real
error of the provider behind a generic502. The error body is now read without blocking, logged and
returned (#195). - Sampling parameters that were never set — Azure OpenAI, Groq, Mistral, OVH and x.ai sent default
temperature/top_pvalues, refused by the models that do not support them (#197). - Token limit —
max_tokenswas renamed tomax_completion_tokensfor every provider of the OpenAI client,
which many OpenAI-compatible APIs refuse: it is now renamed for OpenAI only. The Anthropic proxy sends the
limit under the name each provider knows, and Azure OpenAI getsmax_completion_tokenswhen its API version
knows it (#213). - Models only served on the Responses API (codex and pro models) failed on
/responseswithout the
responsesoption of the provider: they are now always sent to the native endpoint (#221). - Image edits on OpenAI-compatible providers — the prompt was never sent, the images were sent as
fileand
the size twice./images/editsalso accepts several images asimage[], the way the OpenAI SDKs send them
(#219). - Audio translations read the transcription settings and sent the response format as the temperature; they
now read their own settings, and Whisper models are known to translate (#222). - AI agent router in the workflow designer — it could not be saved (
object is not iterable) and offered no
usable path to the model (#215). - Deleting an AI Studio workspace left its MCP virtual server behind (#225).
- Prices of OVH, Scaleway and Hugging Face models, and of a model whose name is written in another case
(#199). - Entities without a description — prompt contexts, budgets, MCP connectors and the other model entities
created without adescriptionwere refused, they are now read with an empty one. - Anthropic proxy — a request with an
output_configother than a JSON schema no longer writes a debug file
in the working directory of Otoroshi.
Release Infos
Contributors
Thanks to @davlgd, @jonathan-varela-maif, @mohamed-amir-ben-khadda-maif and @Zwiterrion for their reports.