[RFC]: Multi-tier KV offloading via the vLLM offloading connector

[dannyharnik]

Motivation.

To date, vLLM offers native KV offloading to CPU memory but does not support further offloading from CPU memory to other tiers such as storage. Implementations for storage offload should either work directly with storage or implement their own CPU offloading as an additional tier. This document describes a high level design to natively support multi-tier KV offloading in vLLM.

The architecture supports a single primary tier, in most cases CPU DRAM, and multiple secondary tiers.

Goals

• Allow simple and native integration of the current vLLM CPU offloading with secondary tiers such as storage, or connection with other vLLM nodes for PD disaggregation settings or P2P communication of KV data.

• Utilize async gpu<->cpu transfers for primary tier and async lookup for secondary tier loads

• Support for HMA models out of the box

• Seamless support for cross TP offloading and transfers

• Simple, clean and performant implementation of PD communication via the CPU tier

What we don't intend to support

• Direct GPU access (neither GPU-storage or GPU-GPU communication)

• Limited flexibility for variance in block size. While we allow vLLM block size to vary, CPU block size must be constant across all vLLM nodes (and a multiple of the underlying vLLM block size).

High level Design

The design starting point is the current offloading connector design. In this design the offloading connector uses the vLLM V1 connector API and translates it to a simpler more abstract API for a backend. The multi-tier offloading has several key changes to this framework:

• It designates a single (CPU DRAM) backend as the primary tier but also allows multiple backends to serve as a secondary tier. However, these backends differ from the current backend definition in that the source for offloading (and target for loading) is the CPU DRAM tier rather than the GPU HBM.

• Introduce a new component -- the TieringManager that serves as an orchestrator for the communication with the Primary and Secondary tiers. A more detailed description of this appears in the TieringManager section below.

• The GPU-CPU offloading is managed by the scheduler side offloading manager yet executed by the worker side backend. Namely, operations are scheduled and invoked by the scheduler thread, whereas the actual data migration is executed by the worker thread (as it is today in the offloading connector). In contrast, all secondary tier operations are both managed and executed by the scheduler side only.

• No matter what the TP rank of a vLLM is, the KV data in CPU DRAM will be stored in canonical form, that of TP rank 1. This is the foundation for supporting cross TP rank KV offloading support. For example, if the secondary tier is storage then all KV values will be stored in a single TP1 canonical form, no matter what the TP rank on the vLLM.

Proposed Change.

In order to facilitate the design changes above we are implementing the changes described below to the current offloading connector.

The TieringManager

The native KV offloading in vLLM v1 currently supports offloading from GPU memory to an external location (like CPU memory). This RFC extends the design to allow offloading from CPU memory to additional tiers such as local storage, object storage, and remote nodes (P/D disaggregation).

The OffloadingConnector interface is unchanged, it holds a single OffloadingManager. The new TieringManager implements that interface and orchestrates the tier hierarchy internally.

Two tier types

Primary Tier: a single tier with exclusive access to GPU KV memory. The existing CPU Manager serves as the primary tier.

Secondary Tier(s): one or more tiers with read/write access to the Primary Tier's CPU memory. No direct GPU access. Each secondary tier implements the SecondaryTierManager interface.

SecondaryTierManager Interface

class SecondaryTierManager(ABC):
    def lookup(block_hashes) -> int | None
    def submit_store(job_metadata: JobMetadata) -> None
    def submit_load(job_metadata: JobMetadata) -> None
    def get_finished() -> Iterable[JobResult]
    def touch(block_hashes)

@dataclass
class JobMetadata:
    job_id: int                         # unique job identifier
    block_hashes: list[BlockHash]       # which blocks are being transferred
    spec: CPUMemoryViewLoadStoreSpec    # memory views into CPU tensors + block IDs

spec is a zero-copy memory view into the primary tier's CPU tensors. For submit_store it is read-only (secondary tier reads from CPU); for submit_load it is writable (secondary tier writes into CPU).

CPU Manager changes

Extend the CPU Manager to expose its worker's cpu_tensors so the TieringManager can pass zero-copy memory views to secondary tiers for direct reads and writes.

Key Design Principles

1. Always cascade to all tiers: When a block is confirmed in the primary tier, it is asynchronously pushed to every secondary tier.
2. Primary tier is the gateway: Only the primary tier accesses GPU memory. Secondary tiers read/write CPU memory via memory views.
3. Staged promotion: Blocks in secondary tiers must be promoted to the primary tier before the GPU can access them. lookup() returns None while promotion is in progress (scheduler retries).
4. Non-blocking scheduler methods: All SecondaryTierManager methods run in the Scheduler process. submit_store() / submit_load() submit async jobs; get_finished() polls for completion.
5. Secondary tiers own their evictions: Each secondary tier manages its own eviction policy independently.

Store Flow (Cascade)

GPU → Primary Tier (CPU) → [all] Secondary Tiers

When TieringManager.complete_store() is called, the KV data is confirmed in CPU memory. The TieringManager calls submit_store() on every secondary tier to cascade the data asynchronously.

Load Flow (Promotion)

GPU ← Primary Tier (CPU) ← Secondary Tier

When TieringManager.lookup() is invoked:

1. Check primary tier first.
2. For remaining blocks, check each secondary tier in order.
3. On a hit, call submit_load() to initiate async promotion to the primary tier, then return None (retry later).

The TieringManager calls get_finished() on all secondary tiers each scheduling cycle to finalize completed jobs.

Canonical CPU layout

The following Diagram depicts the general idea - the CPU memory layout is in canonical form, where each block holds all the KV data relevant to this block in consecutive memory layout. Loading and offloading from and to the secondary tier is always of a single memory buffer and is handled by a scheduler thread (rather than the GPU dedicated workers).

The main changes required to facilitate this are:

• During initialization one of the worker side connectors allocates all CPU memory for the KV cache and assigns the relevant parts of this memory to each of the other workers. This is done in shared memory between the scheduler and workers.

• Each worker is responsible for copying its relevant slice of the KV cache to the appropriate CPU memory locations.

• We mandate that the layout in CPU memory will be in canonical form. In a TP setting the division between workers is according to the heads count -- each worker gets an equal share of the attention heads. As in the drawing above the, we ask that the heads divide the KV CPU memory into contiguous regions and thus can be mapped easily to various TP ranks. There is no constraint on the GPU memory layout, but we achieve better performance and simpler code if that is the case in the GPU as well. This is explained in RFC [RFC]: KV cache layout combining all layers per block #27742 and implemented in PR [KVConnector][Core] Support cross-layer KV blocks #27743. This layout change enables coalescing copies between GPU and CPU to fewer large contiguous memory copies and thus enables efficient use of the DMA for this purpose (see discussion in the following blog https://vllm.ai/blog/kv-offloading-connector ).

Potential Secondary Tiers

• Storage -- This is the obvious secondary tier. Can include:

  • File System API

  • Object Storage

  • Key-Value Store

    • Can be either shared (remote) or local storage

• PD disaggregation -- The current PD connector ("NIXL connector") in vLLM is GPU to GPU communication. Having an alternative CPU to CPU implementation while introduces more hops and therefore more latency, has several benefits:

  • Quicker offloading on the P node, once moved to local CPU can release GPU memory.

  • Shorter time GPU memory required on the D node. Only need to allocate GPU buffers after KV data arrives on the D node CPU memory

  • Simple and clean cross TP handling. No need for complex grid of GPU to GPU chatter.

  • Potentially faster communication on the P to D leg as we will only need to communicate a single unified buffer (relevant to a TP setting).

• P2P -- a generalization of the PD setting is a general P2P communication between nodes.

Feedback Period.

No response

CC List.

@ronensc @orozery @robertgshaw2-redhat @tlrmchlsmth @WoosukKwon @njhill @NickLucche @omerpaz95

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[ruihong123]

Can the design principle 2, "Primary tier is the gateway" be relaxed? With this limitation, the GDS can not be applied to this multi-layered caching design.

[orozery]

Can the design principle 2, "Primary tier is the gateway" be relaxed? With this limitation, the GDS can not be applied to this multi-layered caching design.

GDS would be a separate connector.
You could use it together with multi-tier offloading using MultiConnector.

[dannyharnik]

Can the design principle 2, "Primary tier is the gateway" be relaxed? With this limitation, the GDS can not be applied to this multi-layered caching design.

GDS would be a separate connector. You could use it together with multi-tier offloading using MultiConnector.

Optimally, we'd like to consider an approach that can selectively do loads directly from storage using GDS.
This will still keep the offload path via the primary tier. In the case of TP>1 this will require the GDS to read a part of the file for each TP rank (doable but adds complexity).

[IgniteGo]

Thanks for the detailed RFC! The design for native multi-tier KV offloading looks very promising.

I have a quick conceptual question: What is the primary motivation for building this natively within vLLM, as opposed to relying on external connectors like LMCache? Are there specific architectural advantages, performance benefits, or core differences in this native implementation that external multi-tier cache solutions cannot achieve?

I’d love to understand the core distinction I might be missing. Looking forward to your insights!

[Change72]

The multi-tier offloading data plane has landed (#40020 framework, #41968 object-store tier), but the external KV-event contract is still unclear. Today secondary-tier state is internal to TieringOffloadingSpec, so an external KV-event consumer cannot reliably discover which blocks live below the primary CPU tier.

#43468 added self-describing BlockStored / BlockRemoved events for the single-tier CPUOffloadingSpec, but it intentionally rejects self_describing_kv_events under TieringOffloadingSpec. The reason is that tier promotions can create primary-tier store events that do not correspond to a live GPU store job, so the current request-time metadata tracker cannot describe them.

We would like to extend self-describing KV events to the multi-tier path, and are thinking about a phased plan:

1. Tier-presence events. Emit per-tier BlockStored / BlockRemoved events using the existing event shape: constituent block_hashes plus a stable medium value naming the tier. This is enough for consumers to maintain per-tier block presence during cascade-store, promotion, and eviction, even before every tier event carries the full self-describing payload.

2. Accurate secondary eviction. Emit per-block BlockRemoved events when a secondary tier evicts. Today secondary eviction is internal to the secondary tier, and JobResult only reports {job_id, success}, so TieringManager likely needs an eviction-reporting path.

3. Full self-describing payload across tiers. Persist the small self-describing metadata (token_ids, parent_hash, per-block block_size, cache-spec metadata, etc.) alongside the offloaded block so cascade, promotion, and secondary-tier events can all re-emit full payloads even after the original request is gone. This keeps the contract producer-side and consumer-agnostic: Dynamo, llm-d, and future consumers can all index the same event stream without consumer-specific hash-join logic.

Proposed API direction:

• Use medium as the stable per-tier identifier, with one event describing one tier.
• Extend SecondaryTierManager with an eviction-reporting path so TieringManager can emit accurate BlockRemoved events for secondary tiers.
• Use complete_store for cascade-store events, get_finished_jobs for promotion completion, and secondary eviction reporting for remove events.

Happy to prototype this once there is agreement on tier identity, eviction reporting, and payload shape.

cc @dannyharnik @orozery — would appreciate your guidance.

[IgniteGo]

Thank you for the detailed and clear explanation! It really helps me understand the current challenges and the proposed plan.

[orozery]

@Change72 I think that supporting simple BlockStored events (just hashes, not self-describing) can be straightforward.
Regarding simple BlockRemoved, it depends how eviction is managed for a secondary tier.
I think that for storage, especially for shared storage, but maybe even for local, eviction should be managed outside of vLLM. This means the we (at kv_offload managers) don't have to emit BlockRemoved for these secondary tiers.

I would try to think of ways to avoid self-describing metadata serialization in vLLM.
This complicates the data-path logically and adds performance penalty.
Ideally, this should be handled externally.

cc @vMaroon