v0.3.1

The built-in client is the default everywhere

Until now, an install with the llm CLI on its PATH used llm unless Settings said otherwise. From 0.3.1 the built-in client is the default whether or not llm is installed. llm remains a choice on Settings, Model client (or BENCH_CLIENT=llm), for its plugins. An install that already chose a client on Settings keeps its choice.

Why: pacing, the usage ledger, quota records, streaming, route mode's tool calls and the native Anthropic and Gemini endpoints all live in the built-in client. Calls through llm get none of them.

Docs brought up to date

• README: a badge per API the gateway speaks (OpenAI and Responses, Anthropic Messages, Gemini, Ollama), agents with tool calling, and the MCP Registry. Install from source needs no pip install. Rate limits and the usage ledger are described, the Limits section is rewritten for 0.3 (route mode's reach, what is tested against fakes only, the scheduler living in the server), and the hosting section is "Docker and hosting".
• AGENTS, USAGE, SECURITY and PRIVACY: the same client and key-file wording; USAGE explains Onboard as Full / Quick / Quick + ad hoc and the model roles, vision included.

v0.3.0

Route mode: agent clients on a collection

A panel's several answers can't drive an agent's tool loop, so a collection now has a third reply mode beside all and merged: route. Ask for bench/<collection>:route, or set it as the collection's reply.

• One member answers each turn, picked by its record (useful verdicts, then suite findings). One new conversation in four goes to the member heard from least, so the record doesn't narrow to one model. Members cooling down after a rate limit go last.
• Tools, streaming and images pass through. OpenAI chat relays the upstream stream exactly. Anthropic Messages, OpenAI Responses (including Codex-style custom tools), Gemini and Ollama translate tool definitions, calls and results both ways.
• Sticky: a conversation stays on the model that started it, held for an hour idle, so an agent isn't switched mid-task.
• Failover: a 429, a 5xx, a pacing refusal, no first token within 90 s, or an empty reply moves the turn to the next member before anything reaches the client.
• Recording: every call goes in the usage ledger, and each conversation is one record per model that answered it, with a verdict optional. Route conversations get their own table on the Gateway card and don't count toward Owed.
• Skipped, with the reason given: retired and ad hoc members; members that can't call tools when the request has tools, or can't see when it has images; vision specialists on a turn without images; and members whose window is too small.
• Panels still refuse tools; the 400 now points to :route.

Claude Code, tested end to end:

export ANTHROPIC_BASE_URL=http://127.0.0.1:8122
export ANTHROPIC_API_KEY=<a gateway key, or any text on this machine>
export ANTHROPIC_MODEL=bench/core:route
export ANTHROPIC_DEFAULT_HAIKU_MODEL=bench/fast:route

GATEWAY.md also covers Codex and other Responses clients, opencode, and OpenAI- or Ollama-speaking agents. Those are documented but not yet tested against the real clients.

Native Anthropic and Gemini endpoints

An endpoint can now declare api: "anthropic" or api: "gemini" (a new API choice in the Add endpoint dialog). The built-in client speaks that API natively: its key header, its stream, its usage and its errors, with pacing, quota records and the ledger as for any endpoint. No new dependency: this replaces the plan's aisuite backend. So far it has been checked against fake servers only.

Also

• Every collection is listed in all three modes on /v1/models, /api/tags and /v1beta/models, so agents' model pickers can choose :route.
• An output budget above a model's known maximum is cut to it (Claude Code asks every model for 32,000 tokens).

Known limits: route mode doesn't send turns to members on native or Responses endpoints yet, and the capability probe records tool and reasoning support as unknown for native-API models.

v0.2.2

The gateway speaks more APIs

• Besides OpenAI chat completions, a collection now answers Anthropic Messages (/v1/messages), Ollama (/api/chat, /api/generate, /api/tags), OpenAI Responses (/v1/responses) and Gemini (generateContent, streamGenerateContent). Point an Anthropic, Gemini or Ollama client at the gateway and pick bench/core. GATEWAY.md has a curl example for each.
• With stream: true, each seat's answer is sent as that seat finishes, with keep-alives in between, instead of everything at the end.
• Requests that carry tools are refused clearly for now; agent clients come with route mode in the next release.

The built-in client knows the bench

• Calls stream from the provider, with reasoning kept apart from the answer.
• Real token counts and tokens per second in the perf results, and a note when a provider serves a different model than the one asked for.
• A usage ledger in the data folder (one file a month), which the cost report now reads.
• Rate-limit headers and 429 bodies become quota records with exact reset times, shown on the Quotas page.
• Stored limits are now enforced: a call waits when the wait fits, or is refused before it goes out. Connection limits gain a monthly cap, for trial keys such as Cohere's 1,000 calls a month.
• A free model that went paid is not called until you allow it; an empty reasoning reply is retried once with a bigger budget.

Models

• A vision role for models that read images well but are weak at text-only work (OCR and document models). Only image requests seat them; on a text request a member with this role is skipped and named.
• Onboard Full, Quick or Quick + ad hoc in the picker, and bench onboard --quick [--ad-hoc]. Quick probes now and leaves the suite to the scheduled passes; Quick + ad hoc also keeps the bench from ever calling it on its own.
• The Ad hoc help text now says where an ad hoc model is still called: as a collection member or merger, or when named.

Hosting

• Templates for Fly.io, Railway and Render, and a VPS setup with Caddy for HTTPS, under deploy/ and render.yaml. HOSTING.md compares cost, sleep, disks and HTTPS. Render has a Deploy button.
• The image takes a platform's PORT and public address, and owns a root-mounted /data.

Publishing

• Releases now also list the MCP server in the official MCP Registry (io.github.WendellResearch-AI/benchwarmers), after npm and PyPI.

v0.2.1

Install from PyPI

Benchwarmers is now on PyPI as well as npm: pipx install benchwarmers, uv tool install benchwarmers, or try it without installing with uvx benchwarmers. Python 3.11+, no dependencies. Every install, PyPI, npm or a checkout, runs the same launcher, so the commands, help and first run are the same everywhere.

bench: the dashboard from the command line

bench (or benchwarmers) now does what the dashboard does, headless, against a running server (BENCHWARMERS_URL, or --url):

• bench ask core "..." -f file.py, or pipe a diff in; answers on stdout, the seats' call ids on stderr
• bench verdict <call> --useful --real "..." --false "..."
• bench models, bench model retire|restore|role|rename|max-tokens|ad-hoc|retry
• bench candidates --new, bench onboard <served> --wait, bench dismiss
• bench collections, bench collection show|save|rm (save changes only what you give; --add, --remove)
• bench endpoints, bench endpoint add|test|key|rm|default (keys from a hidden prompt or stdin, never an argument)
• bench keys, bench key create|revoke, bench settings gateway|client|scheduler, bench status

--json on every command, examples in every --help, confirmation before destructive commands (--yes for scripts), and exit code 3 when no server is running. With an admin token set, the CLI sends it as a bearer (BENCHWARMERS_ADMIN_TOKEN). Where another bench (Frappe's) comes first on PATH, type benchwarmers.

Dashboard

• Collection tags narrow the member list; Discover's Add shows progress from the click.
• The onboarding picker says what each model takes besides text; image models get their role automatically.
• Onboarding records why it stopped; Retire confirms in the page.
• A tab icon.
• /api/status, /api/models and /api/gateway-keys serve JSON; a fresh install with no endpoint answers instead of dropping the connection.

Releases

Publishing a GitHub release now publishes npm and PyPI by Trusted Publishing (no stored tokens; npm provenance attached). CI runs every Python on macOS too, and a test that failed on Linux and macOS is fixed.

v0.2.0

From Claude Code and other agents

• MCP server: benchwarmers mcp (or npx -y @wendellresearch-ai/benchwarmers mcp) is an MCP server on stdio with three tools: list_collections, ask (a collection, a prompt, optionally merged and with) and verdict. It is a thin client of a running Benchwarmers server; BENCHWARMERS_URL and BENCHWARMERS_KEY point it at a remote one.
• Claude Code plugin: /plugin marketplace add WendellResearch-AI/benchwarmers, then /plugin install benchwarmers@benchwarmers. Adds the MCP server, /benchwarmers:ask <collection> <prompt>, and a skill for working with a panel: verify before acting, a verdict per seat, every seat named.

Fixes

• Retiring a model takes it out of every collection it was in; Restore puts it back. The gateway skips a retired member instead of calling it.
• A seat that answered stays ok when llm could not write its own log (concurrent seats locking logs.db), in the gateway and in the suite and probes.
• The Models tab lists decision models, under the connection they come from.
• The gateway address the dashboard shows follows BENCH_BIND, BENCH_PORT and BENCHWARMERS_PUBLIC_URL instead of a fixed 127.0.0.1:8123; a reply's bench.verdict is a path on whichever server answered.

Dashboard

• The collection dialog has a search box over the members, by alias or served name.

v0.1.1

Put me in, coach. Every model on your bench gets its minutes.

Onboarding

• Pick several, then Onboard or Dismiss. Ticks survive a search, so you can gather models from different connections; Dismiss acts on every ticked model, and the dismissed list brings any back.
• New models first, in their own group at the top, and every connection's group folds away (remembered per browser).
• No more "unknown connection". A model just added to Open WebUI is grouped under its real connection straight away, named live from Open WebUI's connection list (base URLs only, never keys).
• Progress in the header, not the popup. The popup closes once the batch is queued; a chip beside the button fills by each model's five steps, then turns green, or red when one failed. Details lists every model's state, alias, reason and log. "New results · Reload" waits until the batch ends.

Docs and look

• A new README: the name on its bench as the logo, what Benchwarmers is best at, the related tools it builds on, and a new diagram of the panel loop.
• A social card on the hardwood court, and the court's three-point lines now centred on the rims (in the hardwood theme too).
• SECURITY, PRIVACY, TERMS and USAGE. Outside contributions aren't open yet; report a vulnerability through SECURITY.md.

v0.1.0

The first release of Benchwarmers: a self-hosted panel gateway for LLMs. Ask the whole bench, keep every answer, learn who to trust.

What's in it

• Panel gateway. Any OpenAI-compatible client asks for bench/<collection>; the collection's members and explorer seats (the models unused longest) all answer, each labelled by model and seat. :merged adds one answer that says where the seats agreed and disagreed. Images pass through to the models that take them.
• Verdicts. Every seat has a call id. Record whether an answer was useful and which claims were real or false; verdicts choose the next panels, and the confidently wrong show as such.
• Dashboard. At a glance, Benchmarks, Quota management, Logs, Endpoints, Models, Collections, Discover, Settings and Info, in three themes.
• Endpoints. Sign in with OpenRouter, NanoGPT, Hugging Face, Pollinations, a ChatGPT plan, GitHub Models or a local Ollama, or paste a key for any OpenAI-compatible API. Hubs (Open WebUI, LiteLLM, CLI Proxy API) through their own adapters.
• Onboarding, in bulk. Pick one model or several; they are registered, probed and run through the suite one at a time, with progress in the header.
• Health. A morning probe, auto-retire after one confirming call (always restorable), rate limits learned from providers' own errors, and a catalog of model facts from models.dev, OpenRouter and FreeLLMAPI.
• The suite. Real bugs from public projects (SWE-rebench-V2), claims about real code, and your own repositories seeded with realistic bugs.
• Discover. Public catalogs merged into one list of models worth adding, with every route this install has to each.
• Runs anywhere. Python 3.11 to 3.13 on Windows, macOS and Linux with only the standard library at its core, or Docker. On 127.0.0.1 with no login by default; beyond it only with an admin token, a login and revocable gateway keys.

Before you start

It is early: run it from a checkout (no installer yet), no streaming replies, and the LiteLLM and CLI Proxy API adapters are not yet proven against a real hub. Probes, onboarding and the suite spend calls on your keys. See the README, USAGE.md and SECURITY.md.