Skip to content

v0.4.6 — Tensor splitting with quantized KV cache

Choose a tag to compare

@jimdawdy-hub jimdawdy-hub released this 05 Oct 00:52
· 22 commits to master since this release

Fix the launcher blocking llama.cpp tensor splitting when K/V cache uses q8_0 or q4_0. Recent llama.cpp builds support this combination; Flash Attention on/auto remains required.

  • Preserve selected cache types and GPU proportions in generated commands.
  • Explain that NVFP4 model weights and KV-cache quantization are separate settings.
  • Warn that older engines may need an update, layer splitting, or f16 cache.
  • Add builder, advisor, and API regressions. Tensor placement still reports unknown fit.

Merged fix: PR #22. Package and runtime versions are now 0.4.6.

Validation: 430 tests passed on the release tree (one inherited deprecation warning). No real-model GPU inference or performance benchmark was run. Publishing this release does not restart or update the running GUI.