Skip to content

v0.9.0

Choose a tag to compare

@cldmv-bot cldmv-bot released this 14 Nov 00:54
· 4 commits to master since this release
22a7336

llm-translator v0.9.0 Changelog

Release Date: November 2026
Release Type: Minor
Branch: next → master


Overview

Version 0.9.0 adds four drivers: Azure OpenAI, OpenRouter, Hugging Face Text Generation Inference (TGI) and LocalAI. All four used to run on the generic compatible driver at best. Azure OpenAI didn't work at all without rewriting URLs by hand. Each driver now handles its server's own URL layout, auth, reply quirks and error shapes.

To make Azure possible, a driver can now build its own request path through an optional path leaf, and api-key joins bearer and x-api-key as an auth style. Upstreams on these servers move to their new drivers automatically, and self-hosted servers get a few more startup probes. See the upgrade notes.

None of the four has been run against a real server yet. They're built from the providers' API docs and source and tested against fixtures.


✨ Features

Azure OpenAI driver (#145)

An azure driver, matched by host name (*.openai.azure.com, *.cognitiveservices.azure.com, *.services.ai.azure.com) with no request of its own. The key goes in the api-key header.

  • By default requests go to the v1 API (<endpoint>/openai/v1/chat/completions, the model named in the body, no api-version).
  • Set apiVersion on the upstream to use the classic URLs instead: <endpoint>/openai/deployments/<deployment>/chat/completions?api-version=<version>. deployment is a second new upstream field and defaults to the request's model. The model list and embeddings follow the same layout.
  • A base URL pasted with /openai or /openai/v1 on the end works too.
  • Token limits go out as max_completion_tokens. With an apiVersion dated before 2024-09-01 they go out as max_tokens, the only field those versions know.
  • A prompt rejected by the content filter (HTTP 400, code: "content_filter") keeps that code. Its message lists the categories that were filtered, with their severity. Filtered replies already got stop_details.category in v0.8.0.

Entra ID tokens aren't built in; send one through the upstream's headers. See docs/drivers.md. Fixes #43.

OpenRouter driver (#146)

An openrouter driver for openrouter.ai, matched by host name. https://openrouter.ai, https://openrouter.ai/api and https://openrouter.ai/api/v1 all work as the base URL.

  • reasoning_details (reasoning.text, reasoning.summary and reasoning.encrypted items) comes back to OpenRouter unmodified and in order on the next turn, streamed or not.
  • OpenRouter can answer HTTP 200 with an error on the choice instead of a message. That's now a real upstream error with the provider's status and error type, and in a stream it ends the stream with an error event.
  • Reasoning effort is sent as reasoning: { effort }, and Anthropic thinking with a budget_tokens (and no explicit effort) as reasoning: { max_tokens }. A reasoning object an OpenAI client sends itself is passed through as is.
  • The final usage chunk and the : OPENROUTER PROCESSING keep-alive comments are handled.

Fixes #41.

Hugging Face TGI driver (#147)

A tgi driver, detected by a strict match on GET /info. It needs router: "text-generation-router", a string model_id and version, and a numeric max_total_tokens, so Text Embeddings Inference, which answers /info as text-embeddings-router, isn't claimed. vLLM now steps aside for it.

TGI's replies are normalised for the deviations older releases are known for:

  • tool_calls sent as a single object;
  • numeric tool-call ids, and empty ids on streamed fragments, which used to open a new tool_use block per fragment;
  • tool arguments as an object instead of a JSON string;
  • the eos_token and stop_sequence finish reasons;
  • the text_completion object name.

Structured output is sent in TGI's grammar shape: a JSON schema becomes response_format: { type: "json_schema", value: { name, schema } }. A client's own grammar is accepted and folded into response_format. TGI refuses tools and a grammar together, so tools win and the grammar is dropped. Fixes #38.

LocalAI driver (#148)

A localai driver, detected by the <version> (<commit>) shape of LocalAI's GET /version, or by its /.well-known/localai.json discovery document. vLLM, which also answers /version, steps aside for it.

  • Reasoning is read from LocalAI's reasoning field (and reasoning_content) and sent back as reasoning.
  • LocalAI sets parallel tool calls per model in its YAML (function.parallel_calls) and ignores parallel_tool_calls. When an Anthropic client sets disable_parallel_tool_use, the driver keeps only the first tool call of each reply and stream. An OpenAI client's parallel_tool_calls: false is forwarded but has no effect on LocalAI.
  • repetition_penalty is sent as repeat_penalty, and LocalAI's extra sampling fields (top_k, min_p and others) are passed through.

docs/drivers.md covers parallel_calls and the case where a model emits tool calls as plain text, with the LocalAI function: settings that fix it. Fixes #39.

📚 Documentation

🔧 Dependencies

No dependency updates. @cldmv/slothlet and @cldmv/slothlet-types stay at 3.23.0.


Upgrade notes

  • Upstreams on these servers change driver. An Azure OpenAI or OpenRouter host is now bound to azure or openrouter, and a TGI or LocalAI server to tgi or localai, where they used to run on compatible (or, for TGI and LocalAI, could be taken for vLLM). Set backend: "compatible" on the upstream to keep the old behaviour.
  • Azure upstreams that rewrote URLs by hand. If you pointed an upstream at a full deployment URL to make compatible work, point it at the resource endpoint instead and set apiVersion (and deployment if it differs from the model name).
  • More startup probes. A server no earlier driver claims now also gets GET /info and GET /.well-known/localai.json, and vLLM's own check also asks for /info when /version doesn't name vLLM. The probes run in parallel and are bounded by detectTimeout. Set backend on an upstream to skip probing.
  • No other changes are needed.

coverage

Metric Coverage
Statements 100.0%
Branches 100.0%
Functions 100.0%
Lines 100.0%

Avg: 100.0% · 05d19bf · Node lts/*

👥 Contributors