Skip to content

v0.10.0

Choose a tag to compare

@cldmv-bot cldmv-bot released this 14 Nov 17:13
· 3 commits to master since this release
5b56be2

llm-translator v0.10.0 Changelog

Release Date: November 2026
Release Type: Minor
Branch: next → master


Overview

Version 0.10.0 adds four things. Anthropic requests to vLLM can go to its own /v1/messages instead of being converted, a pool skips vLLM upstreams that aren't ready and routes to LoRA adapters, five hosted OpenAI-compatible providers (Groq, Together, Fireworks, xAI and Gemini) get drivers, and Ollama can be used through its native /api/chat protocol.

Everything is opt-in except the provider drivers, which bind by host name, and the readiness probes, which only run when a model has more than one endpoint. One behaviour change applies to upstreams with backend: "vllm" set by hand; see the upgrade notes.

None of this has been run against a real server yet. It's built from the providers' API docs and source and tested against fixtures.


✨ Features

Native /v1/messages for vLLM (#152)

vLLM 0.11.1 and later serve the Anthropic Messages API themselves. A new nativeMessages setting (true, false or "auto", default false) sends an Anthropic-front request to that endpoint with only normalisation, instead of converting it to Chat Completions and back. "auto" uses it when the detected or configured version is 0.11.1 or later, and converts when the version is unknown.

  • The system prompt is sent as one string with the billing header block removed.
  • vLLM sends no ping event, so one follows message_start.
  • Thinking blocks vLLM sends without a signature (before 0.17) get the translator's marker signature.
  • With toolErrors on, failed tool results carry the marker text, as in a conversion, since vLLM ignores is_error.
  • The upstream is called with the bearer key only; the client's x-api-key and anthropic-beta headers aren't forwarded.

A request ending in an assistant prefill, one with document blocks, and one with deferred tools (unless toolSearch is native) are still converted. An upstream that answers 404 to /v1/messages with something other than an Anthropic error is converted from then on, until the next detect(), and a fallback event is emitted once. status() shows whether an upstream is on the native endpoint. See docs/drivers.md. Resolves #31.

Skip unhealthy upstreams and tag LoRA adapters (#153)

When more than one endpoint serves a model, upstreams whose driver names a health path (vLLM: GET /health) are checked before the pool is ordered, and the ones that aren't ready go last. A lone endpoint is still tried.

  • New pool.healthInterval (milliseconds a verdict is trusted, default 10000, 0 turns probing off) and pool.healthTimeout (default 2000).
  • endpoint.status() and status() carry ready, readyReason and checkedAt. A health event with kinds down and up fires when an upstream changes state, and health.check() probes every upstream right away.
  • vLLM lists each LoRA adapter in /v1/models as a model whose parent is the base model. Adapters are kept apart in the detection facts (facts.adapters), are routable like any other model, and carry parent in model.list() and GET /v1/models on both fronts. An upstream with one base model and some adapters is still the single model unknown names fall back to.
  • A model list fetched live now updates what a vLLM upstream serves, so adapters loaded or unloaded at runtime start or stop being routable. See the upgrade notes.

See docs/drivers.md and docs/server.md. Resolves #32.

Presets for hosted OpenAI-compatible providers (#154)

groq, together, fireworks, xai and gemini drivers, bound by host name (api.groq.com, api.together.ai, api.fireworks.ai, api.x.ai, generativelanguage.googleapis.com) or chosen with backend for a proxy in front of the provider. The provider's path root is added for you, so the host alone works as the upstream url.

The provider facts sit in one table, PRESETS in src/lib/presets.mjs: the path root, the token field (max_tokens or max_completion_tokens), the reasoning field, how thinking and effort are sent, whether top_k is accepted, whether the stream takes stream_options, and the request fields the provider rejects (Groq's logprobs and logit_bias, xAI's stop and penalties on reasoning models). The driver folders are one-line wrappers over it, and settings on an upstream still overrides any value. See docs/drivers.md. Resolves #44.

Ollama's native /api/chat protocol (#155)

An api: "native" setting on an ollama upstream (default "openai", the existing /v1 path) talks to Ollama's own chat API, which exposes the controls /v1 hides: think, options and format. Both fronts are converted to and from it, streamed or not.

  • Tool arguments are JSON objects and tool results are matched by tool_name. Ollama sends no tool-call ids, so each call gets a generated one.
  • think takes the effort level when there is one, else the thinking switch. message.thinking is read back as a thinking block (or the reasoning field on the OpenAI front) and sent on assistant turns.
  • Sampling fields go into options, response_format and output_config.format become format, and images go as base64 strings.
  • The stream is newline-delimited JSON, buffered across chunks so a line split at any byte is parsed once complete.
  • tool_choice other than auto, n above 1, logit_bias and logprobs are dropped and recorded in state.dropped. The model list and embeddings stay on /v1.

Setting api: "native" on an upstream bound to a driver without a native API is an error at detection. See docs/drivers.md and docs/protocol-support.md. Resolves #35.

📚 Documentation

🔧 Dependencies

No dependency updates.


Upgrade notes

  • An upstream with backend: "vllm" and no models list now serves what the server last listed. An explicit backend skips detection, so such an upstream had no model list of its own and a model list fetched from the server was never kept. Now every successful fetch (model.list(), GET /v1/models) replaces the upstream's served models, adapters included, with what the server returned, and requests for those models route to it. An upstream with a configured models list keeps that list, and backend: "auto" upstreams get the same refresh on top of what detection found.
  • Pools probe vLLM upstreams. A model with more than one endpoint now gets a GET /health on each vLLM endpoint, at most once per pool.healthInterval. A server that answers something other than 2xx or 5xx (a proxy without that path) is left alone until the next detect(). Set pool.healthInterval: 0 to turn probing off.
  • Provider hosts change driver. api.groq.com, api.together.ai (and api.together.xyz, us-api.together.ai), api.fireworks.ai, api.x.ai and generativelanguage.googleapis.com upstreams are now bound to groq, together, fireworks, xai or gemini, where they ran on compatible. Set backend: "compatible" on the upstream to keep the old behaviour.
  • nativeMessages and Ollama's api: "native" are off unless you set them.
  • No other changes are needed.

coverage

Metric Coverage
Statements 100.0%
Branches 100.0%
Functions 100.0%
Lines 100.0%

Avg: 100.0% · 60e4777 · Node lts/*

👥 Contributors