v0.6.0

whichllm 0.6.0

Thanks for waiting. 0.5.0 came out in March, and the 23 releases since then were all patches. This is the first new minor version, and it's a big one. 62 pull requests went into it, and they reach almost every part of the tool.

Before cutting it, I went back through all 78 issues and 176 pull requests and replayed the commands you ran. Most of what you hit is fixed here. Three things were still broken, and #261 fixed those.

Ready in a second or two

A cold start used to mean hundreds of Hugging Face requests plus scraped leaderboards, and one rate limit could stall the lot. Now whichllm reads a prebuilt index that GitHub Pages serves fresh every day. One request brings the catalog, the config.json shape of every family, real GGUF file sizes and the merged benchmark scores. --live still fetches everything directly if you want that.

Speed estimates checked against real runs

I fitted the GPU speed model to 379 published llama.cpp runs on 194 GPUs. Leave any one GPU out, predict it from the rest, and the median error is 9%. It used to be 33%. CPU estimates are fitted to 255 LocalScore results.

On my RTX 5080 with DDR5-5600, whichllm says Qwen3.6-35B-A3B at Q4_K_M runs at 70 tok/s. llama.cpp measured 79.

Big MoE models on small GPUs

That 35B model is a 22 GB file, and the card has 16 GB. When a mixture-of-experts model doesn't fit in VRAM, whichllm keeps the routed experts in RAM, the way llama.cpp's --n-cpu-moe does, and estimates speed from what each token actually reads. Every placement has to clear the same usable speed of 8 tok/s, so a model that answers at 3 tok/s can't outrank one that answers at 25.

Run what it picks

Under the ranking you now get the llama-server command and the LM Studio settings that run the top pick the way it was estimated, and --json puts a recipe on every row, Ollama included. whichllm run chats through llama-cli on your GPU when llama.cpp is installed. If the model is already in your LM Studio library, it opens that file instead of downloading it again (#186).

Measure it on your machine

whichllm bench measures real decode speed through llama.cpp, Ollama, LM Studio or any OpenAI-compatible server, and prints it next to the estimate. whichllm report turns the result into a prefilled GitHub issue. You read the whole report first, and you're the one who submits it. whichllm doesn't send anything.

I'd love your numbers. My own measurements come from a single RTX 5080, and everything else is public data. Verified reports go into the estimates, and three plain llama.cpp results that agree with the estimate move a GPU to high confidence.

Context length, finally

@tachijuan asked for it back in May (#34). The new Context column shows each model's own limit and how much of it still fits the way the row runs. 256k→18k means the model takes 256k tokens but stops fitting entirely on your GPU past 18k.

More about your hardware

whichllm hardware now shows how much RAM is free, each NVIDIA card's PCIe link, and a RAM bandwidth it measures while you wait. On a machine with several GPUs, mixed cards too, whichllm plans weights and KV cache per device (#189). Jetson Orin boards show up as GPUs instead of dropping to CPU-only.

Breaking change

--ram-bandwidth now sets system RAM bandwidth. Before, it was an alias of --bandwidth, and --bandwidth is still the flag for a discrete GPU's VRAM. On an iGPU or unified memory, --ram-bandwidth keeps setting the GPU's bandwidth, so --vram 8 --ram-bandwidth 68 works as before.

Thanks

@hannibal-lee built the LM Studio reuse (#187) and the per-device multi-GPU plan (#193, #245). @donghee-ai fixed Jetson Orin detection (#218). And to everyone who opened an issue: a lot of these fixes started with you.

How it was checked

All 1,804 tests pass on the release commit, and the suite passed on native Windows too, where six symlink tests skip. The golden set ranks the published index for 29 hardware setups, from a 4 GB GTX 1650 to a 128 GB M4 Max, and all 562 checks pass. Three llama.cpp runs on my RTX 5080 machine came within 30% of their estimates.

Upgrade

uv tool upgrade whichllm
pip install -U whichllm
brew upgrade whichllm

Everything else is in CHANGELOG.md.