Skip to content

Releases

rigtune 0.3.0

Choose a tag to compare

@Aagam-Bothara Aagam-Bothara released this 19 Nov 06:17

rigtune 0.3.0 is the release behind the November 2026 results.

On this year's models and engines (vLLM 0.33.0, SGLang 0.5.23):

  • Qwen3.5-9B: rigtune's pick ran at least 1.8× the throughput of every other setup: stock vLLM and SGLang, vLLM's auto_tune, BentoML's llm-optimizer, and settings from each engine's own docs.
  • Gemma 4 12B: it ran 1.5–1.7× the best expert setting.
  • Answers: unchanged on all 1,319 GSM8K items in every case.

What the results don't show: most of the Qwen gain comes from the model's built-in MTP head, and only 1–8 users were measured. Full numbers, method and limits are in the benchmarks.

Highlights

  • Check its pick against the alternatives yourself. rigtune compare --verdict measures rigtune's pick against stock engines, other tools' picks (--rows) and expert settings (--expert) on your card. It scores each as a win, tie or loss, and --check-answers gsm8k makes sure no row wins on worse answers.
  • More speculative decoding. --speculative all now tries a checkpoint's own MTP head, EAGLE-3 heads, Gemma 4 assistants and draft models on vLLM and SGLang. Suffix decoding is offered wherever it actually loads.
  • Accuracy gate. --eval-set serves a configuration only if its answers are measurably no worse than stock. GSM8K is bundled.
  • 2026 models. Hybrid-attention models like Qwen3.5 and Gemma 4 are planned with the right KV memory per layer.
  • Current engines. Checked with vLLM 0.33.0 and SGLang 0.5.23; other versions get a warning, not a refusal.
  • Your own goals. --optimise one metric within --limit bounds, with cost per million tokens and energy per request at a stated arrival rate.
  • Multi-turn traffic. --sessions replays conversations turn by turn and measures prefix reuse. --kv-tier tries a CPU KV-cache tier when your working set outgrows the GPU.
  • Every trial recorded. Each calibration is logged to a trial ledger, and rigtune ledger verify re-checks a profile against it.
  • Safer serving. An optional API key, opt-in request limits, and /health returns 503 until the engine is really ready.

Upgrading

Profiles cached by 0.2.x are recalibrated once, on first use, because cached profiles now record the constraints and model revision they were measured with.

pip install -U rigtune
pip install "vllm==0.33.0"

Full list of changes: CHANGELOG.md


One small thing: the changelog heading says 0.3.0 was released on 2026-11-18, and if you publish today it's the 19th. It doesn't matter much, but say the word and I'll fix the date before you push.

v0.2.1

Choose a tag to compare

@Aagam-Bothara Aagam-Bothara released this 12 Nov 06:21

First working release as rigtune (formerly PolyServe). 0.2.0 was published without its code and is yanked.

v0.1.0

Choose a tag to compare

@Aagam-Bothara Aagam-Bothara released this 15 Sep 20:14

PolyServe 0.1.0

First release. PolyServe is an autotuner for LLM serving: it measures serving settings on your GPU and your traffic, then serves the fastest configuration that meets your latency target behind an OpenAI-compatible API.

Install

pip install polyserve
pip install "vllm==0.29.0"      # NVIDIA driver 580+; older drivers: "vllm==0.11.0" "transformers>=4.56,<5"
polyserve serve Qwen/Qwen2.5-3B-Instruct --workload chat

Linux, Python 3.10–3.13. Optional: pip install arctic-inference==0.1.1 adds suffix decoding; SGLang can run from its own environment via SGLANG_PYTHON; llama.cpp needs llama-server on PATH.

What it does

  • Probes the machine, drops every configuration that cannot fit in memory, and benchmarks the rest on your workload across vLLM, SGLang and llama.cpp.
  • Tunes weight precision, memory, batch size, prefill budget, KV-cache type, prefix caching and speculative decoding (n-gram lookup, suffix decoding, draft models), then re-measures the best candidates and chooses on those runs.
  • Judges latency at the 95th percentile time to first token by default.
  • Calibrates on your own prompts with --workload-file, and caches the result per machine, model and workload.
  • Opt-in 4-bit (AWQ, GPTQ) and 8-bit W8A8 checkpoints, since they can cost answer quality.

Results (on prompts calibration never saw)

  • Llama 3.1 8B: +93% on an A40, +59% on an A100, +71% on an RTX 4090 (where stock vLLM does not start), and +24% on an H100 over stock vLLM with fp8 weights.
  • Qwen2.5-3B on an A40, with suffix decoding available: +122% over stock vLLM.
  • Calibration takes 22–56 minutes and paid for itself within 0.5–2.6 hours of busy serving.

Limits

  • The gain comes from measuring on your card and traffic, not from the order of the search: given the same time, random sampling of the same settings matched it on two GPUs and beat it on a third.
  • Load is closed-loop (a fixed number of concurrent users); calibration does not check answer quality.
  • vLLM-CPU, pre-Turing GPUs and --power have not run on real hardware.

Full results and methods: https://github.com/Aagam-Bothara/polyserve/blob/main/docs/benchmarks.md
Usage and every option: https://github.com/Aagam-Bothara/polyserve/blob/main/docs/usage.md