Skip to content

0.0.90

Latest

Choose a tag to compare

@github-actions github-actions released this 23 Oct 10:22
· 46 commits to main since this release

Otoroshi LLM Extension — v0.0.90

Breaking changes

  • Otoroshi 18 only — the extension now builds with Scala 3 for Otoroshi 18, based on Play 3 and Pekko
    (#198). This release needs Otoroshi 18.0.0 or later: deployments still on Otoroshi 17 stay on 0.0.86.
    The fat jar no longer embeds the Scala standard library, Otoroshi provides the very same one.

Added

AI Studio

  • AI Studio — a console for the teams that build with AI, served at /extensions/cloud-apim/ai-studio
    and reachable from the backoffice home page (#204). In a few clicks a team gets a workspace: an
    OpenAI-compatible endpoint, its own providers and provider keys (BYOK, with their health), models, API keys,
    users, guardrails, routing, activity, logs, and reporting with costs and environmental impact. A workspace is
    made of regular Otoroshi entities, so what is done in the studio shows in the backoffice and the other way
    round. It is turned on from the danger zone (AI Studio section), and its activity pages read the active
    User Analytics (PostgreSQL) exporter. AI Studio is experimental.
  • Chat and playgrounds — conversations, regenerate, models compared side by side, attached files, tool
    selection, cost simulation and answers rendered as markdown (#206). Every other model type has its
    playground (images, audio, embeddings, moderation, OCR, video, decisions), with image edition refined step by
    step (#220), a Responses playground showing the summary of the reasoning, a Translation playground for the
    audio models that translate, and an MCP playground calling the tools a workspace exposes (#222). An API page
    explains how to call the workspace.
  • Tools and MCP — MCP servers and connectors per workspace, tool functions with a web_fetch template, and
    HTTP tool calls that can convert what they fetch to markdown (#207). The MCP server of a workspace can be
    exposed in meta mode (list_servers, list_tools, get_tool_schema, search_tools, execute), with an
    optional semantic tool search. A new tool is offered to every provider, and a new provider to every tool.
  • Admin API for workspaces — everything the studio does on a workspace is an admin API route of the
    extension (/api/extensions/cloud-apim/extensions/ai-extension/studio/workspaces/...), to create and manage
    workspaces without the UI (#205). Every route checks the rights of its caller the way the Otoroshi admin API
    does: a service account limited to a tenant manages the workspaces of that tenant, a member only sees the
    pages they can read and their own usage, and secrets come back masked.

Decision models

  • Decision models — a new kind of model answering typed questions (noul, choice, score) about a
    state, with probabilities, through the System One API (Jev and alike) (#208). Native System One servers,
    Cloudflare Workers AI, or any text provider answering the questions. They come with their entity, a
    /systemone endpoint and plugin, a guardrail, a workflow function, fallbacks, budgets and an AI Studio
    playground.
  • OpenAI decisions API — decision models also answer POST /v1/decisions, on the /decisions path of the
    LLM OpenAI Compatible API plugin and with the new OpenAI decisions backend plugin, in the OpenAI format
    whatever provider serves them (#224). OpenAI is a new decision model provider (gpt-6-luna), a question a
    model declines comes back as a refusal, which the decision guardrail denies.
  • Pictures in decisions — pictures go to the decision models that look at them (OpenAI, Cloudflare,
    self-hosted servers, Jev 1.14 as TypeSafe assets with the TypeSafe-Danger-Zone-* headers) and are refused
    with a 400 by the others (#230). /systemone reads TypeSafe assets, and gzip, zstd or multipart bodies.
  • Decision router workflow node — sends a workflow down the path a decision model picks, with a
    min_confidence under which a default path runs instead (#214). Available in the workflow designer.

Routing

  • Smart and intent routers — two new routing models of the Otoroshi router, next to code-router,
    auto-router and fusion-router (#217). smart-router has a decision model rate how demanding a request is
    and sends it to the cheapest candidate good enough for it; intent-router picks the candidate whose
    description fits the request. Both keep answering when the decision model does not, and are configured from
    the providers page or the Routing page of AI Studio.
  • Coding index of the code router — the Artificial Analysis Coding Index is now a bundled data file of 258
    models (27 before), refreshed with the other data files by scripts/update-data-files.mjs (#216). Model
    names are matched word by word, so a model is never given the score of an earlier version.
  • A specific model wherever a provider is referenced — semantic cache, guardrails, search engines, agents
    and plugins can name the model to use next to the provider, instead of its default model (#203).

Budgets and costs

  • Budgets per consumer — scope.partition_by takes an Otoroshi expression and counts the budget apart for
    each of its values (#227): ${user.email} gives everyone their own limit, ${apikey.metadata.team} each
    team, ${ctx.ai.model} each model. The admin API lists the consumers of a budget (/partitions) and reads
    or resets one of them (?partition=), workers push what each consumer consumed, and the analytics show a
    line by consumer.
  • Budget extensions — a budget can allow extensions that the people it counts take themselves
    (POST …/budgets/:id/consumption/_extend), up to a number per cycle set by the administrator (#227). The
    limit grows for the current cycle only and every extension is audited. AI Studio offers both, and its chat
    proposes the extension when a message is refused.
  • Custom budget error message — error_message in the action on exceed of a budget, returned to the
    client of a blocked call (#196). The response always carries budget_exceeded: true.
  • Cost tracking for every model type — image, audio, embedding, moderation, OCR and video models are now
    priced, not only chat models (#199). The costs reported by OpenRouter are used as they are,
    require_known_costs makes a provider refuse (and hide) the traffic it cannot bill, and the
    costs-tracking-provider / costs-tracking-model metadata work on every kind of model.
  • Provider quota alerts — an Otoroshi alert when a provider starts refusing calls, telling throttling
    (LLMProviderQuotaExceededAlert) from an account out of credit (LLMProviderCreditExhaustedAlert), and
    another one when it serves again (#200). A provider known to be refusing is skipped until the time it gives
    (Retry-After, x-ratelimit-reset-*).

Models and providers

  • Reasoning continuity — the reasoning of the previous assistant turns, whatever field it comes in
    (reasoning, reasoning_content, reasoning_details, Anthropic thinking blocks, /responses reasoning
    items), is sent back to each provider the way it takes it: reasoning_details for OpenRouter, signed
    thinking blocks for Anthropic, reasoning_content for DeepSeek and vLLM / SGLang servers, thinking
    chunks for Mistral, reasoning for Groq and Ollama, reasoning items on native /responses, and nothing for
    the APIs that refuse it (#228). Answers carry it the OpenRouter way, as thinking blocks on /messages and
    as reasoning items on /responses, streamed too. The tool loops run by the gateway, provider contexts, cache
    keys and persistent memory keep it, and a signed reasoning is never sent changed.
  • Models catalog and metadata — /models?enriched=true describes each model (kind, endpoints,
    capabilities, known price), with kind, endpoint and has_cost filters (#202). The LiteLLM price table,
    the models.dev catalog and the EcoLogits data are bundled, and refreshed with one script.
  • One /models listing for every model type — the models endpoint of the unified API lists the chat,
    embedding, image, audio, moderation, OCR and decision models of the route in one call, asking each
    connection once.
  • Arkane Cloud — new provider (#209).

MCP

  • MCP 2026-07-28 — the Streamable HTTP endpoint (McpRespEndpoint) serves the stateless revision of MCP
    with protocol_version: "2026-07-28", while clients opening with initialize keep being served 2025-11-25
    on the same endpoint (#192). No session, so any Otoroshi instance answers any call; MCP-Protocol-Version,
    Mcp-Method, Mcp-Name and Mcp-Param-* headers checked against the body, so a load balancer, a WAF or a
    plugin can route on them; ttlMs / cacheScope caching hints; server/discover and subscriptions/listen.
    MCP connectors get a matching http_2026_07_28 transport, and the http one takes a protocol_version
    metadata.

Observability and data protection

  • Built-in dashboards — with an active User Analytics (PostgreSQL) exporter, the extension installs ten
    dashboards in the Otoroshi user dashboards: overview, costs, tokens & cache, budgets, ecological impact,
    performance, reliability, consumers, MCP overview, MCP tools & security (#194). Every widget is a query you
    can reuse in your own dashboards and alert on.
  • Event redaction with Rampart — a data exporter transformer removing personal data from the events of the
    gateway with the Rampart model, before they leave Otoroshi (#201). The credentials of the entities never
    enter an event.

Fixed

  • Fallbacks, load balancers and routers — a call handed over to another provider was counted twice on
    budgets, in metrics and in the environmental impact, two providers falling back on each other never ended, a
    failing fallback was called twice, and a handed over stream ended with two usage chunks (#210). Such a call
    is now priced, measured and metered once, by the provider that served it, and counts for the budgets of both.
  • Calls made on behalf of another — the calls a guardrail, a knowledge base search, a tool, a semantic
    cache, a router or a fusion panel make are counted for the model that served them and for the caller, and
    the call that triggered them keeps its own provider, usage and cost (#211). The usage reported by moderation
    providers is read.
  • Budgets in cluster mode — a leader rejected every budget delta of a worker of a previous version, a
    worker lost what it consumed while pushing its deltas, and a delta was counted in the current cycle whatever
    the cycle it was consumed in (#212).
  • Streamed calls failing with limit of 13 reached — a non-200 answer to a streamed call hid the real
    error of the provider behind a generic 502. The error body is now read without blocking, logged and
    returned (#195).
  • Sampling parameters that were never set — Azure OpenAI, Groq, Mistral, OVH and x.ai sent default
    temperature / top_p values, refused by the models that do not support them (#197).
  • Token limit — max_tokens was renamed to max_completion_tokens for every provider of the OpenAI client,
    which many OpenAI-compatible APIs refuse: it is now renamed for OpenAI only. The Anthropic proxy sends the
    limit under the name each provider knows, and Azure OpenAI gets max_completion_tokens when its API version
    knows it (#213).
  • Models only served on the Responses API (codex and pro models) failed on /responses without the
    responses option of the provider: they are now always sent to the native endpoint (#221).
  • Image edits on OpenAI-compatible providers — the prompt was never sent, the images were sent as file and
    the size twice. /images/edits also accepts several images as image[], the way the OpenAI SDKs send them
    (#219).
  • Audio translations read the transcription settings and sent the response format as the temperature; they
    now read their own settings, and Whisper models are known to translate (#222).
  • AI agent router in the workflow designer — it could not be saved (object is not iterable) and offered no
    usable path to the model (#215).
  • Deleting an AI Studio workspace left its MCP virtual server behind (#225).
  • Prices of OVH, Scaleway and Hugging Face models, and of a model whose name is written in another case
    (#199).
  • Entities without a description — prompt contexts, budgets, MCP connectors and the other model entities
    created without a description were refused, they are now read with an empty one.
  • Anthropic proxy — a request with an output_config other than a JSON schema no longer writes a debug file
    in the working directory of Otoroshi.

Release Infos

  • the documentation is available here
  • release is available here

Contributors

Thanks to @davlgd, @jonathan-varela-maif, @mohamed-amir-ben-khadda-maif and @Zwiterrion for their reports.