Skip to content

v0.2.2

Choose a tag to compare

View all tags
@bikemeardsley bikemeardsley released this 23 Oct 20:37
· 10 commits to main since this release
v0.2.2
e45b948

The gateway speaks more APIs

  • Besides OpenAI chat completions, a collection now answers Anthropic Messages (/v1/messages), Ollama (/api/chat, /api/generate, /api/tags), OpenAI Responses (/v1/responses) and Gemini (generateContent, streamGenerateContent). Point an Anthropic, Gemini or Ollama client at the gateway and pick bench/core. GATEWAY.md has a curl example for each.
  • With stream: true, each seat's answer is sent as that seat finishes, with keep-alives in between, instead of everything at the end.
  • Requests that carry tools are refused clearly for now; agent clients come with route mode in the next release.

The built-in client knows the bench

  • Calls stream from the provider, with reasoning kept apart from the answer.
  • Real token counts and tokens per second in the perf results, and a note when a provider serves a different model than the one asked for.
  • A usage ledger in the data folder (one file a month), which the cost report now reads.
  • Rate-limit headers and 429 bodies become quota records with exact reset times, shown on the Quotas page.
  • Stored limits are now enforced: a call waits when the wait fits, or is refused before it goes out. Connection limits gain a monthly cap, for trial keys such as Cohere's 1,000 calls a month.
  • A free model that went paid is not called until you allow it; an empty reasoning reply is retried once with a bigger budget.

Models

  • A vision role for models that read images well but are weak at text-only work (OCR and document models). Only image requests seat them; on a text request a member with this role is skipped and named.
  • Onboard Full, Quick or Quick + ad hoc in the picker, and bench onboard --quick [--ad-hoc]. Quick probes now and leaves the suite to the scheduled passes; Quick + ad hoc also keeps the bench from ever calling it on its own.
  • The Ad hoc help text now says where an ad hoc model is still called: as a collection member or merger, or when named.

Hosting

  • Templates for Fly.io, Railway and Render, and a VPS setup with Caddy for HTTPS, under deploy/ and render.yaml. HOSTING.md compares cost, sleep, disks and HTTPS. Render has a Deploy button.
  • The image takes a platform's PORT and public address, and owns a root-mounted /data.

Publishing

  • Releases now also list the MCP server in the official MCP Registry (io.github.WendellResearch-AI/benchwarmers), after npm and PyPI.
Assets 2
Loading