Skip to content

Kayon v1.5.1 — honest fit reserve

Choose a tag to compare

View all tags
@caesarlab caesarlab released this 17 Jul 10:30
· 6 commits to main since this release
v1.5.1
e58a029

If you are on 1.5.0, Kayon will offer this update itself. Click Download, then Relaunch to update.

Kayon was too pessimistic about what fits

  • The fit engine reserved 1.5 GB of your VRAM beyond the model weights and KV cache. The real figure is about 256 MB. On an 8 GB card that is 16% of your VRAM, held back for nothing.
  • So Kayon has been telling people models would not fit when they fit comfortably. If a quant you wanted showed as a split or a no, check it again.
  • The 1.5 GB was not measured. It was a figure carried over from a CUDA build that Kayon does not actually ship. This release measures the real runtime instead of trusting the number.

The reserve is now read from your model, not assumed

  • It turns out this reserve has almost nothing to do with model size. It is the output buffer, so it scales with the model's vocabulary. A 70B model needs the same amount as a 3B one with the same tokenizer.
  • Kayon now reads each model's vocabulary out of its own file and computes the figure. Predicted against what llama.cpp actually allocates: 96.0 MB predicted vs 97.1 observed at one vocabulary, 250.5 vs 254.5 at another.
  • Every model in the catalog gets its own number now, from 250 MB for Llama up to 512 MB for Gemma 4.

The runtime is Vulkan, and now says so

  • Kayon's README, settings screen and documentation all claimed a CUDA runtime. The bundled llama.cpp has no CUDA backend in it; it is the Vulkan build. Every one of those claims is corrected.
  • Vulkan is the deliberate choice. The CUDA build plus its runtime is about 600 MB against Vulkan's 20 MB, and a 30x download is too much to pay for faster prompt processing. It is slower than CUDA on NVIDIA, and that is the trade.
  • The GPU details on the dashboard still mention CUDA. That is your driver's CUDA version read from NVIDIA's own library, which is true and stays.

Kayon will no longer guess an offload

  • If Kayon cannot compute an honest verdict for a model, it now refuses to load it rather than falling back to putting every layer on the GPU. That fallback was most likely to fire on exactly the small cards it would then run out of memory on.

Windows will warn that the publisher is unknown. Kayon is not code-signed yet; that is separate from the update signature and is coming.

Assets 4
Loading