Olla v0.1.0

What's in this release

Olla is a high-performance proxy and load balancer for LLM infrastructure.

Quick Start

# Docker
docker pull ghcr.io/thushan/olla:v0.1.0

# Binary (see assets below)
./olla --config config.yaml

What's New in v0.1.0

It's a big release and acknowledges the work we did to get here too with everyone's contributions to get us to this milestone! We'd have called this v0.0.30 but really, it's the first of the month and Olla has passed a lot of the stability checkpoints now.

There are lots of things to be aware of, documented in our Olla v0.1.0 Release Article or see the Full Release Writeup.

Behaviour Changes

• Request counts can halve, and a backend 5xx now counts as a failure. v0.0.29 counted every request that named a model twice, and earlier releases counted a backend 5xx as a success. total_requests, per-endpoint requests, success_rate, total_failures and the matching olla_* metrics now count each request once per backend attempt, so the status verdict, the low success rate issue and any alert on these may change (#267).
• Anthropic and Claude Code traffic is now rate limited. /olla/anthropic/v1/messages, /messages/count_tokens and /models take the same limits as the proxy routes (100 requests per minute per IP, burst 50, 1000 globally) and share each client's per-IP bucket. Many sessions behind one IP (NAT, CI, parallel agents) may see 429s, so raise server.rate_limits.per_ip_requests_per_minute and burst_size (#257).
• per_ip_requests_per_minute: 0 no longer turns rate limiting off. The global limit (default 1000/min, burst 50) still applies. Set global_requests_per_minute: 0 as well to disable both (#256).
• A request is never retried once part of the response has reached the client. A GET that failed mid-stream used to be replayed on the next backend. It now ends with the error (#260).
• SGLang and LMDeploy pass Anthropic requests through. Needs SGLang v0.5.9+ or LMDeploy v0.13.0+. LMDeploy requests carrying tools are still translated. For older builds, override the profile with zz-sglang.yaml or zz-lmdeploy.yaml and set api.anthropic_support.enabled: false (#242).
• Profiles declare the paths they serve. The shipped profiles now list /v1/responses, /v1/messages, /v1/rerank and Ollama's /api/ps and /api/version. A custom profile serving these paths needs the same entries (#240).
• Anthropic translation uses each backend's own chat path. Docker Model Runner gets /engines/v1/chat/completions and Lemonade /api/v1/chat/completions. If no backend serves a chat path, the request gets a 404 (#241).
• Ollama vision and tool models are typed differently. Vision models are vlm with the vision capability and tools maps to function_calling. ?type=llm no longer lists vision models; ?capability=chat still does (#250).
• /olla/ollama/api/ps merges every Ollama host. Entries carry olla_endpoint, a partial answer sets X-Olla-Partial, and the route is rate limited. The /olla/ollama/api/list stub is gone and /api/blobs returns 501 (#243).
• "Request completed" log lines carry http_status, and a backend 5xx logs at WARN instead of INFO. Check any log-based alerting (#253).
• Discovery retry_attempts and retry_backoff are now honoured. A negative retry_backoff fails startup (#245).
• Name patterns tag rerankers and embedders on backends that report no kind. A custom models.yaml with its own name_patterns needs the two new entries (#248).
• X-RateLimit-* headers are omitted when no rate limit applies (#262).
• Additive: olla.type in /olla/models and a top-level type on /olla/models/{id} (#249).

Text Embeddings Inference

• New Text Embeddings Inference (TEI) backend with embeddings, rerank and classification routes, model kinds taken from TEI's /info, and /olla/tei/ provider prefixes. Thanks to @MartinEmrich for the contribution (#231).
• Rerankers and embedders are recognised by name on backends that report no kind (#248), and the models API exposes the kind (#249).
• Model-less requests are narrowed by capability. A request naming a model skips filtering, and a model-less request gets 503 when the only capable backend is down (#236).
• Ollama /api/embed and embedding models now route correctly in a fleet that includes TEI (#247).

Dashboard

• Herd columns, a column picker, an issues tooltip and a filter on the Endpoints panel (#265).

Reliability and Stats

• New proxy.retry controls failover (#260).
• A backend 5xx is no longer logged as a normal completion, and failed requests carry their status in the log (#253).
• Model discovery retries failed fetches (#245), tolerates bad responses, and no longer sends backend credentials to HTTP_PROXY or across hosts (#223, #234).
• Fixed a crash under concurrent model discovery and requests (#233), and a model hosted on several backends no longer flips its listed id (#246).
• Services stop in reverse order when startup fails. Thanks to @SDTBU-FXL (#225).
• Unknown keys in config.yaml are reported at startup (#258), listed once each (#263).
• Config and docs no longer promise settings Olla ignores, such as max_concurrent_requests in a profile (#255).

Rate Limiting and Request Handling

• The global limit and the per-IP limit are independent tiers, checked per-IP first, and every 429 reports zero remaining. Thanks to @matthewjhunter for the global-limiter fix (#256).
• One client can no longer flood the log with rejections (#237).
• Chunked bodies on Anthropic routes are capped at the translator limit (#261).
• Model routing no longer depends on the Content-Type header (#238), and Content-Encoding: identity bodies are read (#238).
• /olla/models/{id} works for IDs containing slashes (#244).
• Ollama model names without a tag resolve to :latest (#235), and Ollama vision and tool capabilities are read from the backend (#250).
• LM Studio model types and capabilities are reported correctly.

Docs and Tooling

• make smoke-build-local checks build-local from clean and dirty build dirs (#239, #252).
• ollamock gained Ollama /api/embed, /api/ps and capabilities in /api/tags (#251).
• Docs and README updated for TEI, retry, rate limits and the herd dashboard (#254, #259, #268).

Full detail, with the links for each change, is in the v0.1.0 release notes.

Acknowledgements

• @MartinEmrich - added the Text Embeddings Inference backend (#231).
• @matthewjhunter - fixed the global rate limiter being skipped by the per-IP short circuit, and reported the retry-on-partial-response, chunked-body cap and unknown-config-key issues.
• @SDTBU-FXL - fixed the service rollback order on failed startup (#225).

Full Changelog: v0.0.29...v0.1.0