KVMem v0.17.0 · llama.cpp · Source only / 仅源码

KVMem v0.17.0 · llama.cpp · Source only / 仅源码

仅源码，不包含 v0.17.0 预编译运行包。Source only: no v0.17.0 prebuilt runtime is included.

GitHub 的 Latest 指向本源码版。想直接运行，请按下面的后端与系统选择已有运行包；这些包对应各自的旧版或预发布源码，并不包含本页全部新功能。

GitHub Latest points to this source release. For a ready-to-run package, use the navigation below; each package belongs to its own earlier or prerelease source and does not include all v0.17.0 features.

用途 / Audience	后端 / Backend	模型 / Model	下载 / Download	状态 / Status
Windows NVIDIA RTX 30/40/50	NInfer	.ninfer	v0.18.0 RC1	预发布 / Prerelease
Windows NVIDIA	llama.cpp	GGUF	CUDA RC3	预发布 / Prerelease
Linux / WSL2 NVIDIA	llama.cpp	GGUF	CUDA RC3	预发布 / Prerelease
Windows / Linux / WSL2 AMD	llama.cpp	GGUF	ROCm Beta2	测试版 / Beta
Windows Bonsai 2	Bonsai 专用 / Dedicated	PTQ1 GGUF	prism.3	专用实验版 / Experiment
自行编译 / Build from source	llama.cpp	GGUF	v0.17.0 source	仅源码 / Source only

自行编译 / Build from source: v0.17.0 ZIP · v0.17.0 tar.gz

v0.17.0 changes / v0.17.0 更新说明

v0.17.0 — Source release / 源码发布

KVMem keeps long conversation KV history in host RAM and retrieves relevant blocks into a bounded GPU working set, reusing previously computed KV across turns. This source release brings the changes since v0.16.0-rc3 to the existing v0.17.0 tag (commit 37805065ab5b2e311cb556005eaae40b3476416f).

KVMem 将长对话的历史 KV 保存在主机内存中，按需检索相关块到有界的 GPU 工作集，并在多轮对话间复用已计算的 KV。本次源码发布基于已有的 v0.17.0 tag，汇总 v0.16.0-rc3 之后的更新。

Source release only: no new v0.17.0 prebuilt binaries are included. Existing rc3 CUDA and ROCm beta 2 downloads come from earlier source and do not include all features below.

本次仅发布源码，不附带新的 v0.17.0 预编译运行包。现有 rc3 CUDA 与 ROCm beta 2 下载包基于较早源码，未包含以下全部功能。

English — Main features

• Multiple conversation KV caches with optional NVMe spill. Retain several conversations' host KV histories and reuse matching prefixes when switching back. Configure session count with --kvmem-conversations; soft RAM limits and an optional NVMe quota move inactive sessions by LRU. The active session stays in RAM, requests run one at a time, and disk snapshots are reusable only within the current server process. Startup cleanup reclaims marked, unlocked orphaned cache directories while preserving live servers' files. RAM cache · NVMe cache
• Experimental CUDA multi-GPU inference. Two explicitly selected CUDA GPUs support layer and tensor splitting with one logical KVMem history. Layer mode stores each layer's KV on its owning GPU; tensor mode uses llama.cpp Meta-device KV shards. Current KVMem transfers are synchronous, MTP constraints depend on the split mode, and row splitting or mixed backends are unsupported. Layer mode · Tensor mode
• Independent vision-projector placement. --mmproj-device selects CPU or a named GPU for the vision projector independently of the language model. CUDA builds with Vulkan support can also place the projector on a Vulkan GPU. Device guide
• OpenAI Responses API. /v1/responses and /responses support non-streaming requests and SSE streaming, including function-call output and image input through the existing vision pipeline when --mmproj is loaded. File IDs are unsupported; image detail is controlled by server settings. API notes
• Experimental ROCm support on Windows and Linux/WSL2. Native HIP builds and launchers integrate KVMem and GDN replay. Build scripts discover distribution-provided LLVM, honor compiler overrides, and can build without Ninja. GPU compatibility depends on the installed ROCm SDK. ROCm guide
• Long-running workload fixes. Linux IQ3/IQ4 launchers limit glibc allocator arenas by default to reduce observed RSS growth. Additional changes include batched CUDA layer-capture readback, multimodal replay-budget corrections, and CUDA KV-layout/OOM recovery fixes. These are targeted reliability and overhead improvements. Host-memory diagnosis · Capture batching

中文 — 主要特性

• 多会话 KV 缓存，可选 NVMe 转存。 单个服务可保留多个对话的主机端 KV 历史，切回对话时匹配前缀并复用缓存。--kvmem-conversations 控制会话数量，内存软上限与可选 NVMe 配额按 LRU 转存不活跃会话。活动会话仍驻留内存，请求逐个执行；磁盘快照仅在当前服务进程内复用，重启后不恢复。启动时会清理已标记、未被占用的旧缓存目录，并保留仍在使用的文件。内存缓存 · NVMe 缓存
• 实验性 CUDA 双卡推理。 显式选择两张 CUDA 显卡后，可使用 layer 或 tensor 分卡，并共用一份逻辑 KVMem 历史。Layer 模式将各层 KV 保存在所属显卡，tensor 模式使用 llama.cpp Meta 设备的 KV 分片。当前 KVMem 传输采用同步方式，MTP 约束随模式不同；暂不支持 row 分卡或混合后端。Layer 模式 · Tensor 模式
• 视觉投影器独立选卡。 --mmproj-device 可为视觉投影器单独选择 CPU 或指定显卡，不改变语言模型的设备分配。启用 Vulkan 支持的 CUDA 构建还可将视觉头放到 Vulkan 显卡。设备说明
• OpenAI Responses API。 /v1/responses 和 /responses 支持非流式请求与 SSE 流式事件，包含函数调用输出；加载 --mmproj 后，图片输入复用现有视觉处理路径。暂不支持文件 ID，图片 detail 由服务端设置控制。API 说明
• 实验性 Windows 与 Linux/WSL2 ROCm 支持。 原生 HIP 构建和启动脚本接入 KVMem 与 GDN replay；构建脚本可发现发行版提供的 LLVM，遵循编译器覆盖设置，并支持无 Ninja 构建。具体显卡兼容性取决于本机 ROCm SDK。ROCm 指南
• 长时间运行的稳定性修复。 Linux IQ3/IQ4 启动器默认限制 glibc allocator arena 数量，缓解已观测到的 RSS 持续增长；同时包含 CUDA layer 捕获读回合批、多模态 replay 预算修正，以及 CUDA KV 布局与 OOM 恢复修复。这些是针对性的稳定性和开销改进。内存诊断 · 捕获合批

Full release notes / 完整发布说明 · Changes since v0.16.0-rc3 / 版本差异