Skip to content

Releases

v0.6.1 — Multi-GPU dashboard cards and llama.cpp PCI bus device ordering

Choose a tag to compare

@jimdawdy-hub jimdawdy-hub released this 01 Nov 04:10

Changes

  • Render a Dashboard metric card for every detected GPU (supporting 3+ GPUs) with a responsive auto-fit grid instead of only displaying GPU 0 and GPU 1.
  • Update the Dashboard VRAM Budget subtitle to reflect the actual number of detected GPUs (Used across all N GPUs, Used across both GPUs, Used on GPU, or No GPUs detected).
  • Set CUDA_DEVICE_ORDER=PCI_BUS_ID when probing llama-server --list-devices and launching llama.cpp so CUDA0, CUDA1, CUDA2, etc. match nvidia-smi PCI bus device numbering (consistent with vLLM and SGLang).
  • Show GPU model names alongside CUDA0/CUDA1/CUDA2 in the Launch tab's llama.cpp devices: summary line.

v0.6.0 — SGLang and TGI engines, engine versions on the dashboard

Choose a tag to compare

@jimdawdy-hub jimdawdy-hub released this 25 Oct 06:41
3cf9bca

Changes

  • Refresh SGLang integration and parser choices for SGLang v0.5.22. Correct host binding, GPU selection, executable choice, and multimodal guidance.
  • Add Hugging Face TGI v3.3.7 as a Docker engine, with setup guidance and conservative fit estimates. TGI is archived upstream.
  • Support Podman's Docker command wrapper for TGI GPU selection, and require the exact launch image tag when detecting its availability. Clean temporary token files after failed container exits and support multiple selected GPUs.
  • Show installed llama.cpp, vLLM, SGLang and Docker engine versions beside cached latest GitHub release versions on the Dashboard.
  • Keep engine versions readable when an executable path is long or the dashboard is narrow.
  • Expose llama.cpp prompt chunk reuse and vLLM CPU KV cache offloading.
  • Keep SGLang API keys out of launcher controls because v0.5.22 logs its server arguments. Correct its quantization choices and align model fit recommendations with the Launch default.

v0.5.7 — model discovery and saved settings

Choose a tag to compare

@jimdawdy-hub jimdawdy-hub released this 20 Oct 05:56
3187022

Changes

  • Downloaded models remain visible when a newer cache revision contains metadata without weights.
  • Favorites now have Edit and Archive controls. Archiving shows the recovery-file path and saves removed settings in one private file.
  • Reload previous settings restores the latest model configuration, including after its server entry is removed, with the observed load outcome.
  • Status badges and checkmarks use bright green, yellow and red in both themes.
  • Interface text, documentation and examples use general hardware guidance.
  • llama.cpp source updates retain accurate build numbers, and prompt/console settings are checked against the selected server.

Resolves #28, #30, #31 and #32. See PR #33 for implementation and validation details.

The frontend production build, package-content comparison, Python/catalog parsing and live API checks completed. Automated regression suites and model inference runs were not performed.

Python wheel and source archive are attached with SHA-256 checksums. Engine installations are managed separately from the launcher package.

v0.5.6 — Launch favorites and benchmark log

Choose a tag to compare

@jimdawdy-hub jimdawdy-hub released this 18 Oct 04:49
91fbf7b

What's New in v0.5.6

  • Named launch favorites: save a launch and relaunch it later.
  • Benchmark log with run-to-run comparison.
  • Favorites never save host, credentials or internal fields. A relaunch follows the current Settings LAN toggle, so a favorite saved with LAN access on cannot reopen the LAN after it is switched off. Set an explicit --host again on each launch.

v0.5.5 — New llama.cpp & vLLM flags (reasoning, MoE/FFN offload, lazy mode, speculative draft, host, prompts)

Choose a tag to compare

@jimdawdy-hub jimdawdy-hub released this 16 Oct 05:46

What's New in v0.5.5

  • Main-section llama.cpp and vLLM launch controls:
    • Essential: --temp (LLAMA_ARG_TEMPERATURE) and --system-prompt.
    • Connection: --host (LLAMA_ARG_HOST) on both llama.cpp and vLLM, overriding the Settings LAN toggle when set.
    • Performance & memory: moved -ncmoe / --n-cpu-moe N (LLAMA_ARG_N_CPU_MOE) out of Advanced into the main performance section.
  • llama.cpp Advanced launch controls:
    • Offload & loading: --moe-cache-mib N (LLAMA_ARG_MOE_CACHE_MIB), --n-cpu-ffn N (LLAMA_ARG_N_CPU_FFN), and --lazy-mode MODE (auto, on, off; LLAMA_ARG_LAZY_MODE).
    • Context, server & prompt options: --context-shift / --no-context-shift (--ctx-shift, LLAMA_ARG_CONTEXT_SHIFT), --slots / --no-slots (LLAMA_ARG_ENDPOINT_SLOTS), --color [on|off|auto], and --system-prompt-file FNAME.
    • Reasoning / thinking controls: --reasoning [on|off|auto] (LLAMA_ARG_REASONING), --reasoning-format FORMAT (auto, deepseek, deepseek-legacy, none; LLAMA_ARG_THINK), --reasoning-effort LEVEL (default, minimal, low, medium, high, xhigh, max; LLAMA_ARG_REASONING_EFFORT), and --reasoning-budget N (LLAMA_ARG_THINK_BUDGET).
    • Top 5 speculative decoding (--spec-draft-*) controls: --spec-draft-model, --spec-draft-ngl, --spec-draft-n-max, --spec-draft-n-min, and --spec-draft-p-min.
  • Validation & traffic-light advisor:
    • Blocks --lazy-mode on when combined with --no-mmap, warns when --moe-cache-mib is set without CPU MoE offload, and marks custom memory fit for --n-cpu-ffn and --spec-draft-model.

v0.5.1 — MTP draft-ahead setting, measured llama.cpp memory, freeze watchdog

Choose a tag to compare

@jimdawdy-hub jimdawdy-hub released this 06 Oct 02:02
b77e76a
  • New llama.cpp setting "MTP words drafted ahead" (1–16), shown when MTP is
    on. It was fixed at 3. The default is now 1: on gpuhost (two RTX 5060 Ti),
    drafting 3 ahead crashed or froze llama.cpp three times on 2026-10-05 across
    two Qwen3.8 27B files, while 1 held up. An out-of-range value blocks launch
    with a plain message.
  • The llama.cpp memory estimate now follows llama.cpp's own buffer reports,
    measured on gpuhost with Qwen3.8 27B and Gemma 4 12B. The working memory
    grows with the micro-batch size (ubatch) and context, and compressed
    conversation memory needs an extra full-size copy of one layer. Each of
    llama-server's 4 default conversation slots keeps its own Mamba-style state
    and sliding window. The flat 0.7 GB allowance is gone. Qwen3.8 27B at 262k
    context with ubatch 512 was estimated at 25.0 GB and used 26.1 GB; it is
    now estimated at 27.6 GB, slightly high because llama.cpp keeps the word
    table (1.3 GB) in system RAM. Draft-MTP's cache is counted at full size,
    as llama.cpp keeps it, and the 1.5 GB allowance is replaced by measured
    parts.
  • Close to the limit, the advice now explains that llama.cpp doesn't refuse
    a model that is too big: it quietly runs the overflow on the CPU, which can
    be 10–40x slower.
  • New freeze watchdog for llama.cpp (Advanced, default 3 minutes, 0 = off).
    If the server keeps accepting requests but stops working on them, the
    launcher stops it and the Servers page explains why. A long prompt being
    read never counts as frozen.

v0.5.0

Choose a tag to compare

@jimdawdy-hub jimdawdy-hub released this 05 Oct 23:45
2bc1715

What's Changed

  • v0.5.0: exact llama.cpp memory estimate (incl. draft-MTP), text-only counts, warned launch over the estimate by @jimdawdy-hub in #25

Full Changelog: v0.4.8...v0.5.0

v0.4.8 — Sidebar shows the real version

Choose a tag to compare

@jimdawdy-hub jimdawdy-hub released this 05 Oct 04:14
6aa612d
  • Fixed the sidebar showing a stale version (it said v0.4.5). It now reads the version from the server, and a test refuses a hand-typed version.

v0.4.7 — Reach the GUI through tailscale serve

Choose a tag to compare

@jimdawdy-hub jimdawdy-hub released this 05 Oct 02:56
1fc106d
  • Added --allow-host NAME (repeatable) so the GUI can be reached through a tailscale serve address or similar private name.
  • The server still listens on 127.0.0.1 only; the DNS-rebinding guard accepts only those exact extra names and refuses wildcards.
  • Engine source updates still require a direct local connection.

v0.4.6 — Tensor splitting with quantized KV cache

Choose a tag to compare

@jimdawdy-hub jimdawdy-hub released this 05 Oct 00:52

Fix the launcher blocking llama.cpp tensor splitting when K/V cache uses q8_0 or q4_0. Recent llama.cpp builds support this combination; Flash Attention on/auto remains required.

  • Preserve selected cache types and GPU proportions in generated commands.
  • Explain that NVFP4 model weights and KV-cache quantization are separate settings.
  • Warn that older engines may need an update, layer splitting, or f16 cache.
  • Add builder, advisor, and API regressions. Tensor placement still reports unknown fit.

Merged fix: PR #22. Package and runtime versions are now 0.4.6.

Validation: 430 tests passed on the release tree (one inherited deprecation warning). No real-model GPU inference or performance benchmark was run. Publishing this release does not restart or update the running GUI.