Skip to content

Releases: kvmem/kvmem-llama.cpp

Release list

v0.16.0-rc2 — Windows CUDA prebuilt

v0.16.0-rc2 — Windows CUDA prebuilt Pre-release
Pre-release

Choose a tag to compare

View all tags
@wangleye wangleye released this 19 Sep 00:56
v0.16.0-rc2
4837d45

v0.16.0-rc2 — Windows CUDA prebuilt

English | 中文

English

Native Windows prebuilt packages compiled with CUDA 13.2.86, including the browser chat interface. IQ3 is the recommended model; download the prepared files without local quantization.

Downloads

  • For normal use: download kvmem-v0.16.0-rc2-windows-x86_64-cuda13.2.86.zip. It includes the server, CLI, matching CUDA DLLs, Web UI, and PowerShell launch scripts.
  • For model conversion: optionally download the ZIP with quantizer in its name. It is not needed to run prepared models.
  • Source: kvmem-v0.16.0-rc2-source.zip includes the pinned llama.cpp source with the patches already applied.
  • Each ZIP has a matching .sha256 checksum file.

Download the model files separately:

Quick start

Windows x64; compiled CUDA targets: sm_75/80/86/89/90/120a. Tested on RTX 5060 Ti 16GB. A compatible NVIDIA driver and the Microsoft Visual C++ x64 runtime are required. CUDA Toolkit, Visual Studio, and Node.js are not needed to run the package. The CPU must support AVX2/FMA/F16C/BMI2.

Open PowerShell in the extracted runtime directory. Replace the model paths and GPU index with your own:

powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\windows\start-iq3.ps1 `
  -Model 'D:\models\Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf' `
  -Mmproj 'D:\models\mmproj-Qwen3.8-27B-Q5_K-MIX.gguf' -Gpu 0

After the model loads, open http://127.0.0.1:18200/ . Keep the terminal open; press Ctrl+C to stop the server. For text-only startup, see the README inside the runtime package.

Changes in rc2

  • Native Windows/MSVC support and Web UI assets located relative to the executable.
  • Historical KV storage in host memory on Windows.
  • llama.cpp-style -ctk/-ctv and --cache-type-k/--cache-type-v options, with --kv-dtype retained.
  • Quantized K and V must use the same type, for example -ctk q8_0 -ctv q8_0. Mismatched types fail immediately, before model loading.
  • Separate runtime and optional quantizer packages. IQ3 is the primary recommendation; IQ4 remains an optional experimental configuration.

Validation

The six-architecture build passed the following checks on RTX 5060 Ti 16GB: 8 basic tests, 52 KV argument cases, 3 IQ3 plain-inference checks, and 4 short-text checks with KVMem Q8 + MTP3/ReplaySSM at 8K context. Web UI homepage and static assets, launch scripts, and split-package checks also passed. Build metadata, a source manifest, and validation records are included in the packages.

Linux/WSL2 users should continue using the rc1 prebuilt packages.

中文

原生 Windows 预编译包，使用 CUDA 13.2.86 构建，包含浏览器聊天界面。推荐 IQ3 模型，下载后无需自行量化。

下载

  • 普通用户： 下载 kvmem-v0.16.0-rc2-windows-x86_64-cuda13.2.86.zip。包含 server、CLI、配套 CUDA DLL、Web UI 和 PowerShell 启动脚本。
  • 需要转换模型： 可选下载名称带 quantizer 的 ZIP；普通运行不需要。
  • 源码： kvmem-v0.16.0-rc2-source.zip 包含固定版本的 llama.cpp 及已应用补丁。
  • 每个 ZIP 都有配套 .sha256 校验文件。

模型单独下载：

启动

Windows x64，CUDA 目标 sm_75/80/86/89/90/120a，实测 RTX 5060 Ti 16GB。需要兼容的 NVIDIA 驱动和 Microsoft Visual C++ x64 运行库；无需安装 CUDA Toolkit、Visual Studio 或 Node.js。CPU 需支持 AVX2/FMA/F16C/BMI2。

在解压目录打开 PowerShell，替换模型路径和 GPU 编号：

powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\windows\start-iq3.ps1 `
  -Model 'D:\models\Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf' `
  -Mmproj 'D:\models\mmproj-Qwen3.8-27B-Q5_K-MIX.gguf' -Gpu 0

加载后访问 http://127.0.0.1:18200/ 。保持终端开启，Ctrl+C 停止服务。纯文本启动方式见包内 README。

本次更新

  • 原生 Windows/MSVC 支持和相对于程序目录的 Web UI 加载。
  • Windows 使用主机内存保存历史 KV。
  • 支持 llama.cpp 风格 -ctk/-ctv、--cache-type-k/--cache-type-v，保留 --kv-dtype。
  • 量化 K/V 必须为相同类型，例如 -ctk q8_0 -ctv q8_0；不匹配时在加载模型前立即报错。
  • 运行包与可选量化工具分开；IQ3 是主要推荐，IQ4 仅保留为备选测试配置。

验证

本次六架构产物在 RTX 5060 Ti 16GB 上通过：8 项基础测试、52 项 KV 参数检查、3 项 IQ3 普通推理及 4 项 KVMem Q8＋MTP3/ReplaySSM 短文本测试（8K 上下文）。Web UI 首页和静态资源、启动脚本、拆分打包检查通过。包内附有构建信息、源码清单和测试记录。

Linux/WSL2 用户继续使用 rc1 预编译包。

Loading

v0.16.0-rc1 — Chat fixes and Linux CUDA prebuilt

v0.16.0-rc1 — Chat fixes and Linux CUDA prebuilt Pre-release
Pre-release

Choose a tag to compare

View all tags
@wangleye wangleye released this 16 Sep 08:33
v0.16.0-rc1
db4bc22

v0.16.0-rc1

This pre-release fixes thinking-budget handling and repeated tool-call IDs, adds Jinja template and reasoning-effort configuration, and provides an experimental Linux/WSL2 CUDA binary package.

Changes

  • Correct zero-budget thinking-prefix handling and validate request budget overrides.
  • Support custom Jinja templates, template kwargs and top-level reasoning_effort (top-level takes precedence).
  • Give requests and tool calls distinct IDs across requests and server restarts.
  • Simplify recipe launchers around the MTP3 / F16 draft KV / ReplaySSM defaults.
  • Add portable build/packaging scripts and an issue #1 short-prompt comparison script.

Binary package

kvmem-v0.16.0-rc1-linux-x86_64-cuda13-sm120a.tar.gz

  • Linux/WSL2 x86_64; Ubuntu 22.04 / glibc 2.35 baseline.
  • CUDA target sm_120a; AVX2/FMA/F16C/BMI2 CPU required.
  • Bundles CUDA runtime/cuBLAS and license texts; CUDA Toolkit is not needed.
  • Requires a compatible NVIDIA driver and host C/C++ runtime. Recipe launchers also need Python 3.10+, Bash and ss.
  • No model weights included. See the README inside the package for model paths, launch commands and diagnostics.

The matching source asset includes the pinned llama.cpp source with the maintained patch already applied. It is more convenient for rebuilding this binary than GitHub's automatically generated source archives, which omit submodule contents. SHA-256 files are provided for both assets.

Loading