Skip to content

v0.12.0

Choose a tag to compare

@cldmv-bot cldmv-bot released this 18 Nov 02:54
· 1 commit to master since this release
71f196e

llm-translator v0.12.0 Changelog

Release Date: November 2026
Release Type: Minor
Branch: next → master


Overview

Version 0.12.0 is about running the server in production. It adds an optional Prometheus /metrics endpoint, and TLS, per-key rate limits and CORS for the built-in server. All four are off unless you turn them on, so an existing deployment behaves as before.

One default changes: a failed tool result sent to a Chat Completions upstream is now marked with toolErrorMarker out of the box. Set toolErrors: false to keep the old behavior.


✨ Features

Prometheus metrics (#168)

GET /metrics serves Prometheus text format (0.0.4) with no extra dependency. Turn it on with --metrics, LLM_TRANSLATOR_METRICS=1, "server": { "metrics": true } in the config file, or the metrics: true option of server.start and server.plugin. When it's off, /metrics is an unknown route. The endpoint uses the same apiKey / authorize authentication as every other route.

Metric Type Labels
llm_translator_requests_total counter front, route, driver, code
llm_translator_request_duration_seconds histogram front, route, driver, code
llm_translator_tokens_total counter driver, type
llm_translator_errors_total counter kind
llm_translator_upstream_up gauge upstream, driver

Labels are bounded: route patterns, never URLs, and never models, request ids or keys. A client that disconnects is counted as code="499" and not as an error. Counters are in memory per server. Requests to unmatched paths and the scrapes themselves are not counted. See docs/server.md. Resolves #51.

TLS, per-key rate limits and CORS (#170)

Each is a server.start option and a server key in the config file, and each has CLI flags and environment variables. cors and rateLimit also work on the Fastify plugin.

  • TLS: tls: { key, cert, ca, passphrase } takes file paths and serves HTTPS. Flags are --tls-key, --tls-cert, --tls-ca and --tls-passphrase. A missing or unreadable file, or key without cert, fails start-up with invalid_config. Files are read once, with no reload.
  • Rate limit: rateLimit: { requests, windowMs = 60000 } is a fixed window per configured API key, and per remote address for anything else. Flags are --rate-limit and --rate-limit-window. Over the limit the client gets a 429 in its front's error format with retry-after, before any upstream is contacted. The limit doesn't apply to /, /api/hello, /metrics or preflights. Limits are per process, and behind a reverse proxy every request shares the proxy's address.
  • CORS: cors is a list of allowed origins, or "*"; the flag is --cors-origin (repeatable). Allowed origins get CORS headers on every reply, including errors and streams. Preflights get a 204 without an API key and outside the rate limit. Requests without an Origin header are untouched.

When requests come in, CORS preflights are answered first, then metrics counts the request, then the rate limit applies, then the API key is checked. A request refused with 429 shows up in the metrics as code="429". See docs/server.md. Resolves #52.

Failed tool results are marked by default (#169)

Chat Completions has no error flag on tool messages, so a failed tool reached the model as ordinary output. toolErrors now defaults to true: a tool_result with is_error: true gets toolErrorMarker ("[error] ") in front of its text on every OpenAI-protocol driver, including vLLM's native /v1/messages passthrough. Upstreams that take is_error natively are unaffected. toolErrorInference stays off by default, because a tool that just prints the marker would be read as a failure. See docs/settings.md. Resolves #124.

📚 Documentation


Upgrade notes

  • Failed tool results now carry a marker. If a model or prompt depended on seeing failed tool output unchanged over Chat Completions, set toolErrors: false.
  • Metrics, TLS, rate limits and CORS are opt-in. No other changes are needed.

coverage

Metric Coverage
Statements 100.0%
Branches 100.0%
Functions 100.0%
Lines 100.0%

Avg: 100.0% · ab1a862 · Node lts/*

👥 Contributors