Feature Request: Disk-based context checkpoint offloading (`--cache-disk`)

[wypiki]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Problem

Context checkpoints (--ctx-checkpoints, --cache-ram) are currently stored exclusively in system RAM. On UMA (Unified Memory Architecture) systems such as AMD Strix Halo, system RAM and VRAM are physically the same memory pool. This means checkpoint caching actively competes with model weights and KV cache for the same limited resource, rather than providing relief.

On these systems, --cache-ram doesn't offload anything – it just reserves more of the same unified memory that the model already needs, reducing the effective memory available for context or GPU layers.

Proposed Solution

Add a --cache-disk <path> option (or --cache-ssd) that allows context checkpoints to be serialized to a fast local storage device (NVMe SSD) instead of being held in RAM.

Modern NVMe SSDs deliver 5-7 GB/s sequential read throughput, which would allow restoring a typical checkpoint (50-100 MiB) in ~10-20ms – negligible compared to the seconds or minutes saved by avoiding full prompt re-processing on hybrid/recurrent models like Qwen 3.5.

Suggested interface

--cache-disk <path>       directory for disk-based checkpoint storage (default: disabled)
--cache-disk-max <MiB>    maximum disk space for checkpoints (default: unlimited)

When --cache-disk is set, checkpoints would be written to disk instead of (or in addition to) RAM. On restore, the checkpoint is read back from disk into the appropriate memory (GPU/unified).

Use Case

This is particularly relevant for:

1. UMA/APU systems (AMD Strix Halo, Strix Point, future Apple Silicon with large models) where RAM = VRAM and every megabyte of RAM used for checkpoints is a megabyte less for model/context
2. Multi-user systems: this kind of almost unlimited caching would make serving multiple users with different system prompts or several agents simultaniously fast on systems with slow prompt processing speeds
3. Large context windows (100k+ tokens) where checkpoints can grow large and numerous

Why not just use --cache-ram?

On discrete GPU systems with separate VRAM, --cache-ram makes sense – it uses a different memory pool. But on UMA systems:

• RAM is VRAM – there's no offloading benefit
• Checkpoint RAM usage directly reduces available memory for model layers, KV cache, and context size
• Users are forced to choose between more checkpoints (faster turn transitions) and more context/layers

Disk-based checkpoints would completely eliminate this trade-off on UMA systems, and also benefit discrete GPU setups where system RAM is also limited.

Environment

• Hardware: AMD Strix Halo (Ryzen AI Max+ 395, Radeon 8060S, 128 GB unified memory)
• Model: Qwen3.5-122B-A10B (Q5_K_XL, ~85 GB)
• Context: 220k tokens
• Backend: Vulkan
• llama.cpp version: b8398

Motivation

Making UMA systems like Strix Halo or Apple Silicon a real option for multiple users, also making the experience for single users much better with assistants like OpenClaw, Claude Code etc.

Possible Implementation

Instead of copying the checkpoints to RAM, they would be copied to the SSD and read from there when matched.

[strawberrymelonpanda]

Modern NVMe SSDs deliver 5-7 GB/s sequential read throughput, which would allow restoring a typical checkpoint (50-100 MiB) in ~10-20ms

Not casting doubt on the proposal overall, but I don't know about these numbers in any sort of real-world use.

Nothing against more options, personally, I just wonder if the performance hit of hitting disk cache really pays off and would be curious to see. A lot of people also actively dislike SSD writing to avoid thrashing, though I think fears with modern drives in the TBs of writes are often overblown.

[wypiki]

Reading/Writing a few hundred megabytes from/to the SSD definitely is much faster than processing 4000 tokens on Strix Halo.

[storm1er]

Multi-user systems: this kind of almost unlimited caching would make serving multiple users with different system prompts or several agents simultaniously fast on systems with slow prompt processing speeds

I'm 100% in that use case, and I noticed that it would be usefull even for subagent with depth > 2, when context "switching", you can wait 10min+ juste looking at prompt processing and nothing is streamed.

[mollahmasud977-sys]

Absolutely—this use case makes total sense. Even with deep subagents, caching could dramatically reduce those long waits while context is being processed. Definitely something worth prioritizing. في السبت، 11 أبريل 2026 22:57 storm1er ***@***.***> كتب:

…

*storm1er* left a comment (ggml-org/llama.cpp#20697) <#20697 (comment)> *Multi-user systems: this kind of almost unlimited caching would make serving multiple users with different system prompts or several agents simultaniously fast on systems with slow prompt processing speeds* I'm 100% is that use case, and I noticed that it would be usefull even for subagent with depth > 2, when context "switching", you can wait 10min+ juste looking at prompt processing and nothing is streamed. — Reply to this email directly, view it on GitHub <#20697 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/CBTQ2IUXG4VOVSNHRCGSBP34VK5UJAVCNFSM6AAAAACWVIKYT6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DEMZQGI2DQNBYGI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[storm1er]

Just found out about the --slot-save-path option, I'm trying it but it probably only help on concurrent -np session (meaning if you have more session than -np then it's almost useless)

[storm1er]

update: slot-save-path is clearly not meant to be use for this. It's a nice to have on llama-server restart, but that's it. It won't save unit checkpoints to be used later.

All UMA devices (mac mini, amd strix halo, etc) could benefit greatly from this.

[andreasntr]

Adding to what has already been said, saving the checkpoints to disk allows way faster time-to-first-token when the server is restarted. This can also be achieved by caching both to disk AND (v)ram

[RoelVideler]

This is basically what oMLX does well. Even though pre-processing speed of MLX is very slow in comparison to llama-cpp, it will still make it pretty useful. I'd still prefer the much better stability of llama.cpp with this cache model. MLX has a lot of downsides. Especially on memory restricted Macs. Having ram+ssd caching tiers would be amazing.

[vedmant]

I was just thinking why it's not implemented yet, KV cache RAM issue on mac is a real problem when running coding agents, which just eat up all available RAM pretty quickly. SSDs on mac are extremely fast (10gb/s), and offloading KV cache just obvious when doing multiple sessions with large context. Here is proof of concept btw https://github.com/yshk-mxim/agent-memory

[xxxajk]

The cache management IMHO is a nightmare with this project regardless. I've been trying to work it out in a way where I can checkpoint/resume on my own terms, and it causes the whole thing to fall apart, the LLM goes nuts, etc.
The cache management problem I see is it tries to be "too smart", instead of doing things in a better way, which is to CRC the whole turn, and do a checkpoint. a "similar match" != exact match, and you get strange crap like ghost tokens, llm confusion where it either loops forever, takes too long or spews complete garbage, and there is no option to turn off this "feature" and just checkpoint/continue by turn with CRC. It's VERY annoying and super broken, and has never been stable from my experiences with it.
I only use it because of the back-end support, which gives a ton of options, like Vulkan, which allows you to use totally normally "incompatible" GPU and mixed GPU brands, and the "hope" that it can be used to save slot progress in a sane way... alas, it falls flat. Worse there's no way (on any of these systems) to alter context that was valid only for the turn (e.g. tool calls that are single shot) and regenerate the KV cache with those tool calls missing without forcing an entire turn, which pollutes the entire idea.
see here: #23240
It's a mess, and does not allow you to write a CLI that can have the LLM manage it's own token use.

[ferhimedamine]

Disk-based KV cache is a valuable feature for long-context inference, especially for scenarios where the prefix is stable across requests (system prompt + tool definitions). The main implementation complexity is cache invalidation — if any token in the prefix changes, all cached KV state downstream is stale.

One useful design constraint: make the cache key the hash of the token sequence rather than a session ID, so identical prefixes across different sessions get cache hits automatically.

Worth noting that this is the low-level inference layer; for the higher-level concern of persisting extracted agent memory between sessions (rather than raw KV states), dakera-deploy handles this at the application layer and can complement an inference-level KV cache like this one.

[Archmonger]

It might make sense to also have this flag support disk-backed cache in "swa-full style" . Would allow users with fast storage to get the most "bang for your buck". For example, in a scenario where you are using a sliding window attention model at 300K context.