Repository navigation
Releases
v0.6.1 — Multi-GPU dashboard cards and llama.cpp PCI bus device ordering
Changes
- Render a Dashboard metric card for every detected GPU (supporting 3+ GPUs) with a responsive auto-fit grid instead of only displaying
GPU 0andGPU 1. - Update the Dashboard VRAM Budget subtitle to reflect the actual number of detected GPUs (
Used across all N GPUs,Used across both GPUs,Used on GPU, orNo GPUs detected). - Set
CUDA_DEVICE_ORDER=PCI_BUS_IDwhen probingllama-server --list-devicesand launchingllama.cppsoCUDA0,CUDA1,CUDA2, etc. matchnvidia-smiPCI bus device numbering (consistent with vLLM and SGLang). - Show GPU model names alongside
CUDA0/CUDA1/CUDA2in the Launch tab'sllama.cpp devices:summary line.
v0.6.0 — SGLang and TGI engines, engine versions on the dashboard
Changes
- Refresh SGLang integration and parser choices for SGLang v0.5.22. Correct host binding, GPU selection, executable choice, and multimodal guidance.
- Add Hugging Face TGI v3.3.7 as a Docker engine, with setup guidance and conservative fit estimates. TGI is archived upstream.
- Support Podman's Docker command wrapper for TGI GPU selection, and require the exact launch image tag when detecting its availability. Clean temporary token files after failed container exits and support multiple selected GPUs.
- Show installed llama.cpp, vLLM, SGLang and Docker engine versions beside cached latest GitHub release versions on the Dashboard.
- Keep engine versions readable when an executable path is long or the dashboard is narrow.
- Expose llama.cpp prompt chunk reuse and vLLM CPU KV cache offloading.
- Keep SGLang API keys out of launcher controls because v0.5.22 logs its server arguments. Correct its quantization choices and align model fit recommendations with the Launch default.
v0.5.7 — model discovery and saved settings
Changes
- Downloaded models remain visible when a newer cache revision contains metadata without weights.
- Favorites now have Edit and Archive controls. Archiving shows the recovery-file path and saves removed settings in one private file.
- Reload previous settings restores the latest model configuration, including after its server entry is removed, with the observed load outcome.
- Status badges and checkmarks use bright green, yellow and red in both themes.
- Interface text, documentation and examples use general hardware guidance.
- llama.cpp source updates retain accurate build numbers, and prompt/console settings are checked against the selected server.
Resolves #28, #30, #31 and #32. See PR #33 for implementation and validation details.
The frontend production build, package-content comparison, Python/catalog parsing and live API checks completed. Automated regression suites and model inference runs were not performed.
Python wheel and source archive are attached with SHA-256 checksums. Engine installations are managed separately from the launcher package.
v0.5.6 — Launch favorites and benchmark log
What's New in v0.5.6
- Named launch favorites: save a launch and relaunch it later.
- Benchmark log with run-to-run comparison.
- Favorites never save
host, credentials or internal fields. A relaunch follows the current Settings LAN toggle, so a favorite saved with LAN access on cannot reopen the LAN after it is switched off. Set an explicit--hostagain on each launch.
v0.5.5 — New llama.cpp & vLLM flags (reasoning, MoE/FFN offload, lazy mode, speculative draft, host, prompts)
What's New in v0.5.5
- Main-section
llama.cppandvLLMlaunch controls:- Essential:
--temp(LLAMA_ARG_TEMPERATURE) and--system-prompt. - Connection:
--host(LLAMA_ARG_HOST) on bothllama.cppandvLLM, overriding the Settings LAN toggle when set. - Performance & memory: moved
-ncmoe/--n-cpu-moe N(LLAMA_ARG_N_CPU_MOE) out of Advanced into the main performance section.
- Essential:
llama.cppAdvanced launch controls:- Offload & loading:
--moe-cache-mib N(LLAMA_ARG_MOE_CACHE_MIB),--n-cpu-ffn N(LLAMA_ARG_N_CPU_FFN), and--lazy-mode MODE(auto,on,off;LLAMA_ARG_LAZY_MODE). - Context, server & prompt options:
--context-shift/--no-context-shift(--ctx-shift,LLAMA_ARG_CONTEXT_SHIFT),--slots/--no-slots(LLAMA_ARG_ENDPOINT_SLOTS),--color [on|off|auto], and--system-prompt-file FNAME. - Reasoning / thinking controls:
--reasoning [on|off|auto](LLAMA_ARG_REASONING),--reasoning-format FORMAT(auto,deepseek,deepseek-legacy,none;LLAMA_ARG_THINK),--reasoning-effort LEVEL(default,minimal,low,medium,high,xhigh,max;LLAMA_ARG_REASONING_EFFORT), and--reasoning-budget N(LLAMA_ARG_THINK_BUDGET). - Top 5 speculative decoding (
--spec-draft-*) controls:--spec-draft-model,--spec-draft-ngl,--spec-draft-n-max,--spec-draft-n-min, and--spec-draft-p-min.
- Offload & loading:
- Validation & traffic-light advisor:
- Blocks
--lazy-mode onwhen combined with--no-mmap, warns when--moe-cache-mibis set without CPU MoE offload, and marks custom memory fit for--n-cpu-ffnand--spec-draft-model.
- Blocks
v0.5.1 — MTP draft-ahead setting, measured llama.cpp memory, freeze watchdog
- New llama.cpp setting "MTP words drafted ahead" (1–16), shown when MTP is
on. It was fixed at 3. The default is now 1: on gpuhost (two RTX 5060 Ti),
drafting 3 ahead crashed or froze llama.cpp three times on 2026-10-05 across
two Qwen3.8 27B files, while 1 held up. An out-of-range value blocks launch
with a plain message. - The llama.cpp memory estimate now follows llama.cpp's own buffer reports,
measured on gpuhost with Qwen3.8 27B and Gemma 4 12B. The working memory
grows with the micro-batch size (ubatch) and context, and compressed
conversation memory needs an extra full-size copy of one layer. Each of
llama-server's 4 default conversation slots keeps its own Mamba-style state
and sliding window. The flat 0.7 GB allowance is gone. Qwen3.8 27B at 262k
context with ubatch 512 was estimated at 25.0 GB and used 26.1 GB; it is
now estimated at 27.6 GB, slightly high because llama.cpp keeps the word
table (1.3 GB) in system RAM. Draft-MTP's cache is counted at full size,
as llama.cpp keeps it, and the 1.5 GB allowance is replaced by measured
parts. - Close to the limit, the advice now explains that llama.cpp doesn't refuse
a model that is too big: it quietly runs the overflow on the CPU, which can
be 10–40x slower. - New freeze watchdog for llama.cpp (Advanced, default 3 minutes, 0 = off).
If the server keeps accepting requests but stops working on them, the
launcher stops it and the Servers page explains why. A long prompt being
read never counts as frozen.
v0.5.0
What's Changed
- v0.5.0: exact llama.cpp memory estimate (incl. draft-MTP), text-only counts, warned launch over the estimate by @jimdawdy-hub in #25
Full Changelog: v0.4.8...v0.5.0
v0.4.8 — Sidebar shows the real version
- Fixed the sidebar showing a stale version (it said v0.4.5). It now reads the version from the server, and a test refuses a hand-typed version.
v0.4.7 — Reach the GUI through tailscale serve
- Added
--allow-host NAME(repeatable) so the GUI can be reached through atailscale serveaddress or similar private name. - The server still listens on 127.0.0.1 only; the DNS-rebinding guard accepts only those exact extra names and refuses wildcards.
- Engine source updates still require a direct local connection.
v0.4.6 — Tensor splitting with quantized KV cache
Fix the launcher blocking llama.cpp tensor splitting when K/V cache uses q8_0 or q4_0. Recent llama.cpp builds support this combination; Flash Attention on/auto remains required.
- Preserve selected cache types and GPU proportions in generated commands.
- Explain that NVFP4 model weights and KV-cache quantization are separate settings.
- Warn that older engines may need an update, layer splitting, or f16 cache.
- Add builder, advisor, and API regressions. Tensor placement still reports unknown fit.
Merged fix: PR #22. Package and runtime versions are now 0.4.6.
Validation: 430 tests passed on the release tree (one inherited deprecation warning). No real-model GPU inference or performance benchmark was run. Publishing this release does not restart or update the running GUI.