[RFC]: Semantic KV Cache Reuse Interface

[zbennett10]

Motivation.

vLLM's exact prefix cache is the right primitive when requests share the same token prefix. Many long-context workloads have a nearby but different shape: the expensive reusable content is present, but the request starts with a different instruction, chat wrapper, retrieval order, or paraphrased lead-in. Exact prefix caching correctly misses in those cases. This potentially leaves a lot of potential optimization on the table.

This proposal is about exposing the engine-side controls needed for that class of reuse without putting a semantic search algorithm in vLLM. External systems can discover donors; vLLM should continue to own scheduling, paged KV allocation, block lifetime, failure recovery, and exact prefix-cache semantics.

• semantic lookup is an optional external policy decision,
• reporting external tokens to vLLM is a materialization promise,
• exact prefix-cache writes must remain exact,
• any approximate or request-only reuse needs explicit cache-commit semantics before it can be enabled safely with prefix caching.

Proposed Change.

Semantic KV Cache Connector Interface for vLLM

Summary

vLLM already has most of the control points needed for semantic KV cache reuse. KVConnectorBase_V1 separates scheduler-side external match discovery from worker-side KV materialization, the scheduler already has an explicit num_external_computed_tokens path, and the worker/model-runner path already lets connectors load KV into vLLM-owned paged KV slots before attention reads them.

This proposal defines a conservative semantic-KV connector strategy on top of that existing surface:

1. implement semantic donor discovery in an out-of-tree connector first,
2. use vLLM's existing external-token allocation path for materialized reuse,
3. keep vLLM's exact prefix cache semantics unchanged,
4. add only small generic upstream hooks after the connector proves out,
5. use SemBlend as an optional open-source example provider, not as a required vLLM dependency.

The first target is prefix-anchored, contiguous, block-aligned reuse: | exact local prefix | external prefix | tokens to compute |. With vLLM's current API, materialized external KV should be exact-equivalent to the target token/hash namespace or run in discovery-only mode. True non-identical semantic KV materialization needs one additional generic control: vLLM must be able to avoid committing approximate external KV as exact prefix-cache state. More general N:M, fragmented, or non-prefix semantic reuse is valuable, but it should not be the first upstream contract because it requires deeper changes in the scheduler, model runner, slot mapping, batching, and attention lifecycle.

Preliminary SemBlend and SGLang Findings

Early work has been underway within SGLang to provide the interface for semantic/fuzzy kv cache reuse. This includes the first, open source semantic kv cache provider SemBlend (https://github.com/WorldFlowAI/semblend). This works shows that semantic donor discovery can find reuse opportunities exact prefix matching misses, especially in cross-instruction long-context prompts.

Early SGLang/SemBlend experiments reported warm-path TTFT wins that are very promising. We have observed up to 32x speedup at 16K. This is with no degradation in quality. In fact it is not uncommon for the PPL to improve. Benchmarks are still early and we (WorldFlow AI) are working on a labeled dataset tailor built for semantic kv cache reuse measurements.

The start of this within vLLM should be conservative: keep semantic discovery out of core, start with prefix-anchored contiguous reuse, and add a generic cache-commit policy before non-identical semantic KV can be materialized with prefix caching enabled.

Non-goals for v1

• Do not merge a semantic search algorithm into vLLM core.
• Do not introduce a mandatory dependency on SemBlend -> https://github.com/worldflowai/semblend, embeddings, ANN libraries, or a remote service.
• Do not route semantic reuse through LMCache as a requirement.
• Do not change the meaning of vLLM's BlockHash, BlockPool, or exact prefix cache.
• Do not insert semantic donor KV into the exact prefix-cache hash table unless the token sequence and block hash are truly exact.
• Do not standardize arbitrary fragmented or multi-donor materialization in the first interface.
• Do not claim a semantic match is reused unless the connector can actually materialize the corresponding KV blocks into vLLM-owned slots.
• Do not materialize non-identical semantic KV through vLLM's exact external token path while exact prefix-cache writes remain enabled for that request or deployment.
• Do not require model-weight access in the first semantic connector patch.
• Do not support prompt embeddings, multimodal prompts, encoder-decoder models, LoRA-mixed batches, speculative edge cases, or Mamba/hybrid layouts until the connector has explicit compatibility gates for them.

Current vLLM and PR context

The relevant upstream surface is the vLLM V1 KV connector stack:

• vllm/distributed/kv_transfer/kv_connector/v1/base.py
• vllm/distributed/kv_transfer/kv_connector/factory.py
• vllm/config/kv_transfer.py
• vllm/v1/core/sched/scheduler.py
• vllm/v1/core/kv_cache_manager.py
• vllm/v1/core/block_pool.py
• vllm/v1/worker/kv_connector_model_runner_mixin.py
• vllm/v1/worker/gpu/kv_connector.py
• vllm/v1/worker/gpu/model_runner.py
• vllm/model_executor/layers/attention/kv_transfer_utils.py
• vllm/v1/worker/gpu_model_runner.py

vLLM supports out-of-tree connectors through kv_connector_module_path. External V1 connectors are loaded dynamically by the factory and must accept the three-argument constructor:

connector_cls(vllm_config, role, kv_cache_config)

The scheduler already calls:

• connector.on_new_request(request),
• connector.get_num_new_matched_tokens(request, num_local_computed_tokens),
• connector.update_state_after_alloc(request, blocks, num_external_tokens),
• connector.build_connector_meta(scheduler_output),
• connector.update_connector_output(connector_output),
• connector.take_events(),
• connector.reset_cache() when connector reset is requested,
• connector.request_finished(...) or connector.request_finished_all_groups(...).

The worker path already calls:

• register_kv_caches(...) or register_cross_layers_kv_cache(...),
• set_host_xfer_buffer_ops(...),
• handle_preemptions(...),
• bind_connector_metadata(...),
• start_load_kv(forward_context),
• wait_for_layer_load(layer_name),
• save_kv_layer(layer_name, kv_layer, attn_metadata),
• wait_for_save(),
• get_finished(...),
• get_block_ids_with_load_errors(),
• get_kv_connector_stats() and get_kv_connector_kv_cache_events(),
• build_connector_worker_meta().

This makes a direct vLLM connector technically feasible. LMCache remains an important exact-KV and layerwise retrieval implementation, and can remain an optional backend where it fits a deployment. This proposal does not require semantic reuse to route through LMCache; it keeps vLLM core provider-neutral.

Lessons from SGLang

The SGLang prototype is the closest implementation precedent. It is useful because it has exercised semantic donor discovery and materialized reuse in a real engine.

Lessons to preserve:

• Thin engine boundary: the engine owns the lifecycle and memory model; the semantic provider owns donor discovery.
• Opt-in and lazy imports: disabled semantic matching should cost nothing and should not import optional packages.
• Discovery-only mode: operators need hit-rate and quality telemetry before KV injection is enabled.
• Fail closed: provider errors, stale donors, low confidence, allocation failure, or transfer failure must fall back to normal prefill/recompute.
• Donor lifetime: a donor cannot be referenced after its KV slots or blocks have been evicted.
• Recipient-owned realization: donor KV should be copied or loaded into recipient-owned slots before the request state claims those tokens are computed.
• No false exact-cache state: semantic matches must not pollute exact prefix cache state.
• Namespace isolation: donor lookup must respect model, tokenizer, KV layout, dtype, adapter, and positional-encoding compatibility.
• Quality gates: speedup is not enough; validation needs negative controls and output-quality checks (ROUGE-L, F1, PPL, LLM-as-Judge, etc.).

The SGLang work is also a reminder not to over-generalize the first vLLM interface. More flexible non-prefix or multi-donor reuse may be valuable, but it brings engine-specific scheduling, batching, and correction questions. vLLM should start with one prefix-anchored contiguous span and keep broader realization modes out of the first contract.

vLLM Codebase Notes

KVConnector V1

KVConnectorBase_V1 already defines the right separation:

• Scheduler-side methods advertise whether external KV is available and build opaque metadata for workers.
• Worker-side methods use that metadata to load and save KV around the forward pass.
• Worker-side failures can be reported through get_block_ids_with_load_errors().
• Worker-to-scheduler status can be sent through KVConnectorOutput.
• Optional hooks already cover block-pool binding, required KV layout, connector stats reconstruction, Prometheus metrics, cache reset, KV events, and out-of-band transfer handshakes.

The most important method for semantic reuse is:

def get_num_new_matched_tokens(
    self,
    request: Request,
    num_computed_tokens: int,
) -> tuple[int | None, bool]:
    ...

The method must report the largest prefix of prompt tokens whose KV is actually available for loading beyond the local exact prefix. Returning None means the connector is still resolving the lookup and the scheduler should retry later. Returning (tokens, True) means vLLM should allocate blocks and put the request into WAITING_FOR_REMOTE_KVS while the connector completes the load.

This contract is strict. A semantic provider may find a similar donor, but the connector must not report matched tokens unless it can materialize those tokens into the allocated vLLM KV blocks.

Dynamic Connector Loading

KVConnectorFactory lets external connector modules take priority over the internal registry:

{
  "kv_connector": "SemBlendConnectorV1",
  "kv_role": "kv_both",
  "kv_connector_module_path": "semblend.integration.vllm.connector_v1"
}

This is the right starting point. A SemBlend connector can live outside vLLM, exercise the current API, and only propose upstream changes for gaps that cannot be solved out of tree.

Scheduler External-Token Path

In Scheduler.schedule(), vLLM performs local exact prefix lookup first:

new_computed_blocks, num_new_local_computed_tokens =
    kv_cache_manager.get_computed_blocks(request)

Only then does the scheduler ask the connector for external tokens:

ext_tokens, load_kv_async =
    connector.get_num_new_matched_tokens(request, num_new_local_computed_tokens)

The total computed prefix is:

num_computed_tokens =
    num_new_local_computed_tokens + num_external_computed_tokens

The scheduler then calls allocate_slots(...) with:

num_new_computed_tokens = num_new_local_computed_tokens
num_external_computed_tokens = ext_tokens
delay_cache_blocks = load_kv_async

This gives the connector exactly what semantic reuse needs: vLLM allocates recipient-owned KV blocks, and the connector receives the block table in update_state_after_alloc(...) so it can plan the load.

KVCacheManager and BlockPool

KVCacheManager.get_computed_blocks() is exact prefix-cache lookup. If the whole prompt hits, vLLM recomputes the last token to obtain logits.

KVCacheManager.allocate_slots() explicitly distinguishes:

| existing computed | new local exact computed | external computed | new tokens |

The external computed region is the semantic connector's target. vLLM allocates slots for it, but the connector owns the load.

BlockPool.cache_full_blocks() updates block hashes from the Request object. That is safe for exact computed KV. It is not automatically safe for semantic donor KV when the target tokens differ from the donor tokens. A semantic connector must therefore avoid making vLLM believe semantic donor blocks are exact prefix-cache blocks unless the donor and target token/hash namespace is identical.

This is the most important current API limitation. The scheduler's external token path was designed for exact external KV systems. With prefix caching enabled, synchronously loaded external tokens may be committed by allocate_slots(...); asynchronously loaded external tokens are committed when _update_waiting_for_remote_kv(...) calls cache_blocks(...) after the load finishes. That is correct for exact external KV. It is unsafe for approximate semantic KV because vLLM would store donor-derived KV under the target request's exact block hashes.

For v1, the simplest safe policy is:

• exact local prefix cache remains authoritative,
• discovery-only mode returns no external tokens and never loads KV,
• materialized mode without vLLM core changes is limited to exact-equivalent spans,
• non-identical semantic external blocks are used for the current request only after vLLM exposes a way to suppress exact prefix-cache writes for the affected request/span,
• semantic blocks are not inserted into the exact prefix-cache map as reusable exact hits unless the connector verifies token and namespace identity.

If vLLM later adds typed non-exact cache events, semantic reuse can become observable without overloading BlockHash.

Worker and Attention Lifecycle

The worker path already wraps model execution in a connector lifecycle:

bind metadata
start_load_kv(forward_context)
  attention layer:
    wait_for_layer_load(layer_name)
    attention reads paged KV
    save_kv_layer(layer_name, kv_layer, attn_metadata)
wait_for_save()
report finished loads/saves, invalid blocks, stats, events
clear metadata

Current vLLM has both the legacy KVConnectorModelRunnerMixin path and the newer vllm/v1/worker/gpu/kv_connector.py ActiveKVConnector path. The semantic connector should rely on the base KVConnectorBase_V1 lifecycle, not on private details of either model-runner implementation.

The attention decorator calls wait_for_layer_load() before attention and save_kv_layer() after attention. This supports both all-at-once KV loading and layerwise pipelining.

Destination addressing for externally loaded prefix KV should come from the scheduler load plan's block IDs plus KV layout and block-size metadata. The forward context's slot mapping describes tokens scheduled in the current forward pass; it is useful for save/current-token paths, but it is not the source of truth for locating skipped externally computed prefix blocks. Async load can also run through the no-forward connector path with no useful attention metadata.

The connector can register KV cache tensors through register_kv_caches(...) or, when it explicitly opts into a uniform layout, through register_cross_layers_kv_cache(...). Default semantic connectors should keep prefer_cross_layer_blocks=False until they have backend-stride-aware copy support. If a connector requires a specific KV cache layout, it should use get_required_kvcache_layout(...) rather than inferring layout from private runner details. If cross-layer layout is enabled, the load plan must record that layout and address by layer index, block ID, and backend stride order.

MultiConnector and HMA

MultiConnector queries child connectors in order and assigns a request to the first child that reports external matched tokens. It saves to all child connectors. This is useful for composition, but semantic reuse should start as a single connector until its lifecycle and metrics are stable.

vLLM now checks Hybrid Memory Allocator compatibility for configured connectors. If a connector does not support HMA, vLLM can auto-disable HMA; if HMA is explicitly enabled, the connector must support it. A semantic connector should implement SupportsHMA and request_finished_all_groups(...) as early as practical. However, current invalid-block recovery still assumes a single KV cache group in parts of the affected-request scan. Until group-aware load-failure recovery lands, HMA materialized semantic tests should be limited to no-failure/exact paths, or recompute experiments should document --disable-hybrid-kv-cache-manager.

Existing Connector Precedents

LMCache shows a layerwise retrieval path:

• lookup occurs on the scheduler side,
• load specs are carried in connector metadata,
• worker-side start_load_kv() retrieves tokens into vLLM KV caches,
• layerwise retrieval can be synchronized in wait_for_layer_load().

FlexKV shows that a connector can be mostly scheduler-driven and delegate to an external package while satisfying the vLLM connector surface.

Both precedents support the direct-connector strategy: vLLM should own the generic connector hooks, while specialized systems live outside vLLM or in thin optional wrappers.

Layered Architecture

Semantic KV reuse in vLLM should be split into five layers.

Layer 1: Semantic Donor Discovery

Provider-owned. SemBlend is an example of one such open source provider.

Input:

• token IDs,
• optional prompt text,
• exact local prefix length,
• cache namespace,
• provider configuration.

Output:

• donor ID,
• donor lease or generation token,
• donor token IDs or provider-owned donor handle,
• similarity and quality signals,
• candidate reusable prefix length,
• materialization mode,
• fallback reason if rejected.

The provider may use embeddings, lexical hashes, chunk alignment, ANN search, learned scoring, or hybrid logic. vLLM should not depend on the method.

A reported semantic match must include a donor lease, generation token, or equivalent connector-private validity token. The connector acquires it before reporting external tokens, validates it before materialization, and releases it on request finish, reset, stale donor, failed load, or cancellation. The vLLM interface should not expose raw provider slots or donor memory addresses.

Layer 2: Scheduler Admission and Block Allocation

vLLM-owned.

Input:

• local exact prefix length,
• connector-reported external prefix tokens,
• async/sync load decision.

Output:

• request status,
• allocated block table,
• connector metadata slot for worker execution.

The connector can defer lookup by returning (None, False). It can request async loading by returning (tokens, True). It can fail closed by returning (0, False).

For non-identical semantic matches, the connector should return (0, False) until vLLM exposes a per-request or per-span way to suppress exact prefix-cache writes, or the deployment explicitly runs without prefix caching. Current vLLM has no per-request "skip exact-cache write" control. Reporting non-identical semantic tokens as external computed tokens in the current exact-cache path can pollute future exact hits.

Layer 3: Worker-Side Materialization

Connector-owned, using vLLM-owned memory.

Input:

• scheduler metadata,
• vLLM KV cache tensors,
• slot mappings,
• attention metadata,
• donor KV source.

Output:

• loaded KV in allocated recipient slots,
• optional saved donor KV for future use,
• completion status,
• invalid block IDs if loading failed.

The connector must not leave a request in a state where vLLM believes tokens are computed but the corresponding KV slots were never populated.

For synchronous loads, the connector should avoid reporting external tokens unless it can deterministically build the load plan after allocation. In the current call order, allocate_slots(...) may have already committed exact-cache metadata before update_state_after_alloc(...) runs, so plan-construction failure after allocation is a recovery path, not a normal admission path.

For approximate semantic KV, the connector must also prevent the request's donor-derived prefix from becoming future exact-cache state. If vLLM cannot provide that control, the connector should stay in discovery-only or exact-equivalent materialization mode.

Layer 4: Lifecycle, Errors, and Metrics

Shared vLLM/connector responsibility.

The connector reports:

• lookup result,
• load result,
• save result,
• invalid block IDs,
• async completion,
• fallback reason,
• semantic quality signals.

The scheduler applies kv_load_failure_policy:

• recompute: truncate to the longest valid prefix and reschedule,
• fail: fail affected requests and evict cached sync invalid/downstream blocks. Async failed blocks are not exact-cached and are freed when the pending transfer is finalized.

Semantic connectors should default examples to recompute while the feature is experimental.

Layer 5: Optional Model-Aware Correction

Future extension.

Some semantic reuse modes need selective recomputation or model-aware correction to preserve output quality under non-identical tokens or shifted positions. That requires model context and likely a generic worker-initialization hook. It should be a separate, generic vLLM API discussion after the first connector proves:

• discovery hit rate,
• load correctness,
• block accounting,
• failure recovery,
• no exact-cache pollution.

Example Out-of-Tree Connector Shape

The first implementation should be an out-of-tree connector using the existing KVConnectorBase_V1 API. No vLLM base-class changes are required for the first prototype.

The connector can define its own metadata types:

from dataclasses import dataclass, field
from typing import Any, Mapping, Optional, Sequence

from vllm.distributed.kv_transfer.kv_connector.v1.base import (
    KVConnectorMetadata,
    KVConnectorWorkerMetadata,
)


@dataclass(frozen=True)
class SemanticKVMatch:
    request_id: str
    donor_id: str
    donor_lease_id: str
    similarity: float
    local_prefix_tokens: int
    external_prefix_tokens: int
    donor_prefix_tokens: int
    materialization: str  # "none" | "sync" | "async"
    quality_tier: str     # "exact" | "verified" | "semantic"
    fallback_reason: Optional[str] = None
    metadata: Mapping[str, Any] = field(default_factory=dict)


@dataclass(frozen=True)
class SemanticLoadPlan:
    request_id: str
    donor_id: str
    donor_lease_id: str
    local_prefix_tokens: int
    external_prefix_tokens: int
    block_ids_by_group: tuple[list[int], ...]
    donor_offset_tokens: int = 0
    require_block_aligned: bool = True


@dataclass
class SemanticConnectorMetadata(KVConnectorMetadata):
    loads: list[SemanticLoadPlan] = field(default_factory=list)
    saves: list[str] = field(default_factory=list)


@dataclass
class SemanticConnectorWorkerMetadata(KVConnectorWorkerMetadata):
    loaded: set[str] = field(default_factory=set)
    failed_block_ids: set[int] = field(default_factory=set)
    stats: Mapping[str, Any] = field(default_factory=dict)

    def aggregate(
        self,
        other: "SemanticConnectorWorkerMetadata",
    ) -> "SemanticConnectorWorkerMetadata":
        return SemanticConnectorWorkerMetadata(
            loaded=self.loaded | other.loaded,
            failed_block_ids=self.failed_block_ids | other.failed_block_ids,
            stats={**self.stats, **other.stats},
        )

These types are connector-private in v1. The only vLLM-facing contract proposed for v1 is provider-neutral external KV lifecycle behavior. If multiple semantic connectors emerge, vLLM can later standardize the small subset that needs to be shared.

Method Mapping

on_new_request(request)

Use this hook for cheap request registration and optional async lookup submission.

The reviewed upstream main snapshot has this hook. If a target vLLM revision does not, async discovery should be started lazily from get_num_new_matched_tokens(...) and return (None, False) while pending, unless a scheduler-side request-arrival hook is proposed separately.

Recommended behavior:

• derive a cache namespace from vLLM config,
• reject unsupported requests early,
• if async discovery is enabled, submit provider lookup to a bounded executor,
• store lookup state by request.request_id,
• do not mutate vLLM scheduler state.

get_num_new_matched_tokens(request, num_computed_tokens)

This is the scheduler's gate. It should return one of:

• (0, False) for miss, rejected match, unsupported request, or safe fallback,
• (None, False) while an async lookup is still pending,
• (n, False) for synchronous materialization during the next forward,
• (n, True) for async loading through WAITING_FOR_REMOTE_KVS.

Rules:

• n is additional tokens beyond num_computed_tokens, not total hit tokens.
• n must describe a prefix of the target prompt after the exact local prefix.
• n must not exceed what the connector can actually load.
• under current vLLM prefix-cache semantics, n must represent exact-equivalent KV unless an upstream cache-commit policy can prevent exact-cache publication for the affected request/span.
• if using the synchronous path, full-prompt hits should leave at least one token for vLLM to recompute for logits.
• if using the asynchronous path, full-prompt hits can be reported, but the scheduler will still reduce num_computed_tokens by one after load completion.
• low-confidence semantic matches return (0, False) and should be counted as rejected, not as errors.

update_state_after_alloc(request, blocks, num_external_tokens)

This is where the connector converts a semantic match into a concrete load plan.

Recommended behavior:

• if num_external_tokens == 0, clear any pending load state for the request,
• read the allocated block IDs from blocks,
• verify the allocated region covers the external token span,
• verify block and chunk alignment for v1,
• build a SemanticLoadPlan,
• store it for build_connector_meta(...).

If plan construction might fail for a normal, non-stale donor, the connector should avoid reporting the tokens in the first place. Invalid-block reporting is still required for late failures such as donor eviction, transfer failure, or post-allocation validation mismatch.

build_connector_meta(scheduler_output)

Return metadata for the current worker step and clear per-step scheduler state.

Metadata should include:

• load plans for newly scheduled semantic hits,
• save plans for requests whose KV should become donors,
• lookup IDs to clear or unpin,
• no raw provider objects that cannot serialize across process boundaries.

handle_preemptions(kv_connector_metadata)

Use this only for connector-owned async saves or loads whose source blocks may be overwritten after scheduler preemption. A semantic connector that registers donors in the background must either snapshot the needed KV before blocks are reused or cancel the donor registration and release the lease.

start_load_kv(forward_context)

Worker-side materialization begins here.

Recommended behavior:

• read SemanticConnectorMetadata,
• copy step-scoped load/save plans into connector-owned state,
• compute destination addresses from block_ids_by_group and KV layout metadata,
• load donor KV into recipient slots,
• support either all-at-once load or layerwise pipelining,
• record failed block IDs if donor load is incomplete.

Later async completion, invalid-block reporting, and get_finished() must not depend on _get_connector_metadata() after the model-runner context clears it.

wait_for_layer_load(layer_name)

If loading is layerwise, block until the requested layer is ready. If loading is all-at-once, this can be a cheap no-op after start_load_kv(...) completes. Layerwise transfer modes that perform real Python-side synchronization or copying must override requires_piecewise_for_cudagraph(extra_config) to force PIECEWISE CUDA graphs. All-at-once modes can leave the layer hooks as no-ops after start_load_kv(...).

save_kv_layer(layer_name, kv_layer, attn_metadata)

Register or save newly computed KV as future donor state. For v1, saving can be conservative:

• save only completed full blocks,
• skip unsupported request types,
• skip mixed namespaces,
• skip when donor registration would block the forward pass,
• offload embedding/index insertion to a background path when possible.

get_block_ids_with_load_errors()

Report failed recipient block IDs. This lets vLLM truncate to the longest valid prefix and recompute when kv_load_failure_policy="recompute".

Semantic connectors should use this path instead of silently accepting partial loads.

request_finished(...) and request_finished_all_groups(...)

Use request completion to:

• finish donor registration,
• release provider-side handles,
• cancel stale lookup futures,
• report async save status,
• return optional transfer params only if the connector has a real consumer.

Upstream-quality connectors should implement SupportsHMA and use request_finished_all_groups(...).

reset_cache(), shutdown(), and take_events()

Use reset and shutdown hooks to cancel pending lookups, release donor leases, clear provider-local indexes when requested, and stop background workers. Use take_events() only for vLLM-compatible KV events; semantic-specific diagnostics can stay in connector metrics until vLLM agrees on typed non-exact cache events.

Request Flow

Synchronous Load

new request
  |
  v
exact prefix lookup in KVCacheManager
  |
  v
connector semantic lookup
  |
  |-- miss / low confidence / unsupported --> normal prefill
  |
  v
connector returns (external_tokens, False)
  |
  v
vLLM allocates recipient KV blocks
  |
  v
connector builds SemanticLoadPlan from block table
  |
  v
worker start_load_kv loads donor KV into recipient blocks
  |
  v
attention reads recipient-owned KV
  |
  v
remaining prompt tokens are computed normally

Asynchronous Lookup or Load

new request
  |
  v
on_new_request submits lookup
  |
  v
get_num_new_matched_tokens returns (None, False)
  |
  v
scheduler retries request later
  |
  v
lookup complete, connector returns (external_tokens, True)
  |
  v
vLLM allocates recipient blocks and sets WAITING_FOR_REMOTE_KVS
  |
  v
worker or transfer path loads KV
  |
  v
connector reports finished_recving
  |
  v
vLLM caches valid blocks or truncates on failure
  |
  v
request re-enters normal scheduling

Save and Donor Registration

request computes prompt KV normally
  |
  v
save_kv_layer sees full blocks / eligible request
  |
  v
connector snapshots donor identity and namespace
  |
  v
background provider registration embeds and indexes donor
  |
  v
future requests can discover donor

Contract Rules

Namespace Compatibility

A semantic donor must match the target request on every dimension that affects KV correctness:

• model ID and revision,
• tokenizer ID and revision,
• tokenization settings and chat template,
• dtype and KV cache dtype,
• block size and KV layout,
• attention backend layout requirements,
• tensor and pipeline parallel configuration,
• LoRA or adapter identity,
• RoPE scaling and positional encoding settings,
• multimodal feature hashes when supported,
• cache salt or tenant namespace when supplied.

If any namespace field is unknown, the v1 connector should miss.

Tenant and Data-Boundary Safety

Semantic lookup may use prompt text, token IDs, embeddings, or remote ANN services. The connector must treat those as request data, not as generic engine metadata.

Recommended v1 rules:

• do not search or reuse donors across cache salts, tenants, adapters, or served-model namespaces,
• do not send prompt text or embeddings to a remote provider unless the operator explicitly configures that provider,
• make donor retention and embedding retention connector/provider configuration, not vLLM defaults,
• expose metrics for rejected cross-namespace candidates without logging prompt text,
• fail closed when namespace or tenant metadata is missing.

Prefix and Block Alignment

The first connector should only report semantic reuse that can be expressed as additional computed prefix tokens after the exact local prefix.

Recommended v1 gates:

• external_prefix_tokens >= min_reuse_tokens,
• external_prefix_tokens aligned to vLLM block size or provider chunk size,
• donor offset aligned to the same boundary,
• no gaps inside the reported prefix,
• no multi-donor composition inside the reported prefix,
• no target-position reordering.

These gates can be relaxed later with a new materialization design.

Full-Prompt Hits

vLLM needs at least one token recomputed to produce logits on a full prompt hit. LMCache already handles this by reducing the synchronous reported allocation when all prompt tokens are cached.

Semantic connectors should follow the same rule:

• synchronous path: cap external tokens so at least one token is computed,
• for block-aligned v1, this may mean recomputing the final block, not only the final token,
• asynchronous path: allow full prompt load only if the connector and scheduler agree on vLLM's existing full-prompt adjustment after load completion.

Exact Prefix Cache Non-Pollution

Semantic KV for non-identical tokens must not be registered as an exact prefix cache hit.

Safe v1 behavior:

• use semantic donor KV for the current request,
• record semantic metrics separately from exact prefix-cache events,
• register newly computed target KV as exact cache only after vLLM computes it under the target token sequence,
• only emit exact-style cache events when token hashes are exact.

The current vLLM external-token path does not yet expose this policy per request. That means a production-grade non-identical semantic materialization path needs a small upstream extension before it can be enabled with prefix caching. Until then, materialized mode should be exact-equivalent, or the whole deployment should run without prefix caching for workloads where non-identical semantic materialization is enabled.

Required Upstream Gap: Cache-Commit Policy

The first generic vLLM gap is not semantic search. It is cache-commit control for externally loaded KV.

Today, vLLM treats external computed tokens as valid exact KV for the target request. That is correct for P/D transfer, LMCache exact reuse, and external stores whose KV was computed for the same token/hash namespace. Semantic reuse adds a different case: the connector may intentionally load donor KV that is useful for the current request but should not be committed as exact reusable KV for future requests.

The minimal generic extension is a connector-controlled cache policy for external tokens:

class ExternalKVCachePolicy(Enum):
    EXACT_COMMIT = "exact_commit"      # default: current behavior
    REQUEST_ONLY = "request_only"      # current request only, no exact-cache write


def get_external_kv_cache_policy(
    self,
    request_id: str,
    num_external_tokens: int,
) -> ExternalKVCachePolicy:
    return ExternalKVCachePolicy.EXACT_COMMIT

The policy lookup should be side-effect-free and derived from the same connector state used by get_num_new_matched_tokens(...). vLLM should call it only after a connector reports num_external_tokens > 0 and before any allocation path can publish exact-cache metadata.

vLLM would consult this policy:

• before KVCacheManager.allocate_slots(...) can commit synchronous external blocks, either by passing the policy into allocation or by splitting allocation from exact-cache commit,
• in _update_waiting_for_remote_kv(...) for asynchronous external loads,
• before committing blocks through exact prefix-cache paths.

update_state_after_alloc(...) is too late for the synchronous path in the current call order because allocate_slots(...) may already have committed exact-cache state when delay_cache_blocks=False.

EXACT_COMMIT preserves current behavior for existing connectors. REQUEST_ONLY lets semantic connectors load KV into recipient-owned slots while preventing exact-cache pollution. If a request uses REQUEST_ONLY, vLLM should avoid committing the semantic-tainted region and any downstream KV computed from that approximate prefix as future exact cache. REQUEST_ONLY tokens should also be excluded from existing exact-style prefix-cache hit metrics or reported under an explicit external request-only outcome.

This hook is provider-agnostic. It benefits any connector that can provide KV valid for the active request but not valid as future exact prefix-cache state.

Error Handling

Provider and connector failures should map to normal vLLM behavior:

• lookup error: semantic miss,
• lookup timeout: semantic miss or deferred retry,
• donor stale before allocation: semantic miss,
• donor stale after allocation: invalid block report and recompute,
• partial load: invalid block report and recompute,
• unsupported request: semantic miss,
• allocation failure: scheduler tries another request or normal path,
• reset: clear donor state and cancel pending lookup futures.

kv_load_failure_policy="recompute" should be the recommended example setting for semantic experiments.

Why Direct vLLM Integration Makes Sense

Direct integration is technically feasible because vLLM already owns a generic KVConnector V1 path with dynamic external connector loading. A SemBlend-style connector does not need LMCache to reach the scheduler, block allocator, worker KV tensors, slot mappings, attention-layer synchronization, or failure policy.

Direct integration is also a cleaner open-source strategy:

• vLLM maintainers can review generic connector hooks without reviewing a full semantic algorithm,
• SemBlend can iterate out of tree at its own release cadence,
• LMCache remains optional rather than required,
• users can test the connector with kv_connector_module_path,
• successful gaps can be upstreamed as small, provider-agnostic PRs.

LMCache remains useful as a reference for layerwise retrieval and for users who already deploy it. This proposal simply avoids making LMCache a required semantic-KV integration layer.

Recommended Prototype

Build an out-of-tree SemBlendConnectorV1 first.

Phase 0: Discovery-Only Connector

Goal: prove request plumbing, namespace construction, lookup latency, and metrics without KV injection.

Behavior:

• implement on_new_request(...),
• run semantic lookup,
• return (0, False) from get_num_new_matched_tokens(...),
• emit semantic hit/miss/reject metrics,
• never load donor KV.

This gives safe production telemetry.

Phase 1: Exact-Equivalent External Load

Goal: verify vLLM block allocation, metadata, worker loading, and failure handling using only donors whose token span is exact-equivalent.

Behavior:

• find donors through the semantic provider,
• require exact block/chunk match for the materialized span,
• load donor KV into vLLM recipient blocks,
• report invalid block IDs on load mismatch,
• compare outputs to cold baseline.

This exercises the connector path without semantic approximation risk.

Phase 2: Cache-Commit Policy Hook

Goal: add the minimal generic vLLM control needed for non-identical semantic KV materialization without exact-cache pollution.

Behavior:

• default external KV cache policy preserves current exact behavior,
• semantic connector can mark a request or external span as request-only,
• vLLM skips exact prefix-cache writes for semantic-tainted KV,
• tests verify future exact requests cannot hit approximate semantic blocks.

Phase 3: Conservative Semantic Prefix Reuse

Goal: enable approximate semantic donor reuse for long, similar prompts under strict gates.

Behavior:

• require high similarity,
• require minimum reusable tokens,
• require block/chunk alignment,
• require namespace match,
• require quality tier from the provider,
• require REQUEST_ONLY cache policy for non-identical semantic hits,
• support discovery-only and materialized modes,
• default failures to recompute.

Only exact-equivalent spans should use EXACT_COMMIT. Provider quality tiers such as "verified" or "semantic" may justify current-request use under strict gates, but they are not exact prefix-cache eligibility signals unless the connector proves token/hash identity.

Phase 3 materialization is approximate semantic reuse. Correctness-preserving reuse for non-identical token spans requires a later model-aware correction or selective recomputation design.

Phase 4: Optional Model-Aware Correction

Goal: add selective recomputation or correction only after the connector proves the basic lifecycle.

Potential upstream hook:

@dataclass
class WorkerConnectorInitializationData:
    model: torch.nn.Module | None = None
    # Future optional fields:
    # model_runner: Any | None = None
    # kv_cache_config: KVCacheConfig | None = None
    # attention_backends: Mapping[str, type[AttentionBackend]] | None = None

def initialize_worker_connector(
    self,
    initialization_data: WorkerConnectorInitializationData,
) -> None:
    return None

This hook should remain generic. It should not mention SemBlend, semantic KV, or LMCache in the base interface.

Configuration Example

Example vLLM command shape:

vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ \
  --kv-transfer-config '{
    "kv_connector": "SemBlendConnectorV1",
    "kv_role": "kv_both",
    "kv_connector_module_path": "semblend.integration.vllm.connector_v1",
    "kv_load_failure_policy": "recompute",
    "kv_connector_extra_config": {
      "semantic_mode": "discovery_only",
      "min_similarity": 0.75,
      "min_reuse_tokens": 1024,
      "block_aligned_only": true,
      "lookup_timeout_ms": 5,
      "max_donors": 10000
    }
  }'

Materialized mode should be a separate explicit setting:

{
  "semantic_mode": "materialize_prefix",
  "quality_gate": "strict",
  "async_lookup": true,
  "async_load": false
}

Metrics and Logs

Semantic connectors should expose metrics that distinguish discovery, admission, materialization, and fallback.

Recommended counters:

• semantic_lookup_requests_total
• semantic_lookup_hits_total
• semantic_lookup_misses_total
• semantic_lookup_rejected_total
• semantic_lookup_errors_total
• semantic_lookup_deferred_total
• semantic_materialize_requests_total
• semantic_materialize_success_total
• semantic_materialize_recompute_total
• semantic_materialize_failed_total
• semantic_invalid_blocks_total
• semantic_donor_registrations_total
• semantic_donor_evictions_total

Recommended histograms:

• lookup latency,
• external tokens reported,
• tokens actually loaded,
• similarity,
• reuse ratio,
• TTFT cold vs warm in benchmarks,
• fallback stage latency.

Recommended structured fallback reasons:

• exact_prefix_sufficient
• provider_disabled
• provider_timeout
• provider_error
• provider_miss
• low_similarity
• low_reuse_tokens
• namespace_mismatch
• unsupported_request
• not_prefix_anchored
• not_block_aligned
• donor_stale
• allocation_failed
• load_failed
• invalid_blocks
• recompute_policy

Test Plan

vLLM-side tests should use deterministic fake connectors/providers and small synthetic block tables. They should not require SemBlend, embeddings, ANN libraries, a remote service, or model-quality benchmarks to validate the generic connector contract.

Unit Tests

Connector scheduler side:

• get_num_new_matched_tokens returns (0, False) for misses.
• returns (None, False) for pending async lookup.
• caps or rejects full-prompt synchronous hits correctly.
• rejects namespace mismatches.
• rejects unaligned spans.
• does not report tokens when materialization mode is disabled.
• clears state on reset.

Connector worker side:

• metadata round-trips through build_connector_meta.
• start_load_kv populates the expected recipient block IDs.
• wait_for_layer_load synchronizes layerwise load.
• partial load reports invalid block IDs.
• get_finished reports async loads.
• request_finished_all_groups handles HMA block groups.

Block accounting:

• no exact prefix-cache hash insertion for semantic-only blocks.
• sync and async tests for both EXACT_COMMIT and REQUEST_ONLY.
• for non-identical semantic hits, the semantic span and every downstream block computed from that approximate prefix has no BlockHash and cannot be hit by a later exact-prefix request.
• no donor lease or handle leak after request completion.
• no pending lookup leak after cancellation.
• no stale donor use after reset.

Scheduler Tests

Add tests modeled on existing KVConnector scheduler coverage:

• external sync hit schedules only remaining tokens,
• external async hit enters WAITING_FOR_REMOTE_KVS,
• async completion caches or recomputes valid prefix,
• invalid block with recompute truncates computed tokens,
• invalid block with fail fails the request,
• local exact prefix cache wins before semantic lookup,
• connector returning None defers request without failing it,
• allocation failure,
• preemption and resume,
• abort while WAITING_FOR_REMOTE_KVS,
• finished_recving arriving after abort,
• FCFS ordering with skipped waiting queues,
• duplicate concurrent donor lookups.

Integration Tests

Minimum integration matrix:

• small decoder-only model,
• one quantized 7B-class model,
• prefix cache enabled and disabled,
• HMA enabled if connector supports it,
• HMA disabled prototype mode,
• chunked prefill on/off,
• kv_load_failure_policy=recompute,
• discovery-only vs materialized mode,
• disabled, discovery-only, exact-equivalent, and semantic materialized modes at fixed QPS/concurrency and fixed prompt/output distributions.

Quality Tests

Use paired prompt clusters:

• exact control,
• partial overlap control,
• paraphrase / cross-instruction cases,
• reordered context cases as discovery-only until materialization supports them,
• diverse negative controls.

Metrics:

• TTFT cold vs warm,
• p50/p95/p99 TTFT, ITL, end-to-end latency, throughput, GPU memory, scheduler CPU time, lookup queue depth, and fallback-stage latency,
• generated answer similarity,
• ROUGE-L / token F1 for smoke tests,
• perplexity or likelihood under the target prompt where practical,
• LLM-as-judge for qualitative validation,
• false-positive rate on diverse controls.

Do not use speedup alone as a correctness signal.

Phase 1 exact-equivalent runs should match cold-baseline greedy output exactly and match prompt logprobs/logits within dtype tolerance. Semantic-quality metrics apply after approximate semantic materialization begins.

Each phase should define exit criteria and rollback triggers: p95/p99 lookup latency, scheduler-loop overhead, fallback/error rate, pending lookup depth, invalid-block rate, TTFT/ITL regression bounds, false-positive rate, and canary duration before advancing.

Upstream Patch Sequence

Patch 1: Documentation and External Connector Example

Goal: show how an external connector can use kv_connector_module_path for prefix-style external KV reuse.

Scope:

• docs,
• small deterministic example connector if maintainers want one,
• tests that exercise the generic connector lifecycle without SemBlend or an ANN dependency,
• no semantic algorithm in vLLM.

Patch 2: External KV Cache-Commit Policy

This is the required gate before non-identical semantic KV materialization is enabled with prefix caching.

Scope:

• default exact-commit behavior for existing connectors,
• request-only or non-committable external KV policy for connectors,
• scheduler and KVCacheManager tests proving no exact-cache pollution.

Patch 3: Connector Stats Improvements

If current connector stats are not expressive enough, add generic status labels for external-cache outcomes. Keep semantic-specific fields inside the connector unless multiple connectors need the same schema.

Patch 4: Generic Worker Initialization Hook

Only if needed after the out-of-tree connector proves model-aware correction is necessary.

Scope:

• WorkerConnectorInitializationData,
• initialize_worker_connector(...) no-op default,
• model-runner call after KV caches are registered,
• MultiConnector delegation,
• connector tests.

Rules:

• V1 only unless maintainers explicitly want broader scope,
• generic name,
• optional fields,
• no SemBlend or LMCache coupling in the base class.

Patch 5: Typed Non-Exact Cache Events

Only after semantic materialization is working and maintainers want observability in vLLM core.

Scope:

• new event type for approximate/semantic external reuse,
• no mutation of exact prefix-cache hash semantics,
• no scheduler behavior changes.

Patch 6: Fragmented Materialization Design

Do not include in the first interface. This needs its own proposal covering:

• non-prefix target positions,
• two-pass or split prefill,
• batch isolation,
• slot displacement,
• RoPE correction,
• selective recomputation,
• attention backend constraints,
• exact accounting under chunked prefill and speculative decoding.

Designs to Avoid

• Putting SemBlend's algorithm directly in vLLM core.
• Making LMCache a mandatory dependency for semantic vLLM reuse.
• Treating semantic donor token spans as exact BlockHash hits.
• Returning semantic matches from get_num_new_matched_tokens before the connector knows how to load them.
• Reporting non-prefix scattered spans as a prefix token count.
• Sharing donor physical KV slots with the recipient request without recipient-owned realization.
• Letting provider exceptions fail user requests.
• Holding global locks or doing embedding work on the scheduler critical path without a timeout.
• Adding model-runner hooks that name one provider or one algorithm.
• Upstreaming a broad fragmented-reuse API before the prefix path is stable.

SemBlend as an Example Connector

SemBlend can serve as the optional open-source connector/provider that exercises this interface: https://github.com/worldflowai/semblend

Recommended packaging:

• vLLM keeps the generic connector surface,
• SemBlend ships SemBlendConnectorV1 outside vLLM,
• users load it through kv_connector_module_path,
• imports stay lazy,
• discovery-only mode is the first documented mode,
• materialized mode is clearly experimental until quality and accounting tests are published.

The current SemBlend vLLM compatibility path can keep working while the engine-native connector is developed. New vLLM integration work should target the direct connector API rather than requiring LMCache changes.

Open Questions

• Should vLLM expose a generic connector reset hook beyond reset_cache() for scheduler and worker state together?
• Should connector stats support a standard fallback_reason field?
• Should exact prefix-cache stats and external semantic-cache stats remain completely separate in user-facing metrics?
• What is the minimum HMA support expected for new third-party connectors?
• Should async semantic lookup be driven from on_new_request or from the first get_num_new_matched_tokens call?
• Should REQUEST_ONLY be a request-level policy, an external-span policy, or a more general "non-committable prefix" taint tracked by KVCacheManager?
• Should vLLM provide a stable helper for deriving connector cache namespaces from VllmConfig?
• If model-aware correction is needed, should the generic worker-init hook pass only model, or a broader immutable context object?
• What are the maintainers' requirements for testing external connectors that are not vendored into vLLM?

References

• vLLM KVConnector V1 base: vllm/distributed/kv_transfer/kv_connector/v1/base.py
• vLLM KVConnector factory: vllm/distributed/kv_transfer/kv_connector/factory.py
• vLLM KV transfer config: vllm/config/kv_transfer.py
• vLLM scheduler: vllm/v1/core/sched/scheduler.py
• vLLM KV cache manager: vllm/v1/core/kv_cache_manager.py
• vLLM block pool: vllm/v1/core/block_pool.py
• vLLM worker connector lifecycle: vllm/v1/worker/kv_connector_model_runner_mixin.py
• vLLM active GPU connector lifecycle: vllm/v1/worker/gpu/kv_connector.py
• vLLM GPU model runner path: vllm/v1/worker/gpu/model_runner.py
• vLLM attention transfer decorator: vllm/model_executor/layers/attention/kv_transfer_utils.py
• SemBlend: https://github.com/worldflowai/semblend

Feedback Period.

Things are moving fast, I'd like to get things formalized within a couple weeks, if possible.

CC List.

@orozery (because you're the only person that I think I've worked with before when it comes to vLLM) :D

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[Nightwing-77]

i've heard of some Sparse Attention models utilizing semantic similarity of kv cache for the attention computation
(ps: love the explanation but keep it a bit shorter)

[NickLucche]

Thanks for elaborating this proposal @zbennett10 !
I don't have the bw today to review the proposal in depth sorry - I also don't understand how things like "Patch 1: Documentation and External Connector Example" help in keeping the RFC to the topic.. - so I guess what I would like to understand right now is what parts of the connector interface are keeping your proposal from being able to be implemented as an OOT connector?

[zbennett10]

@NickLucche
No worries!
I don’t think the initial version is blocked from being an OOT connector.. although it might not provide a huge benefit. Discovery-only, telemetry, and conservative prefix-shaped / exact-equivalent external KV loads should be implementable OOT today...

The main gap is for non-identical semantic KV materialization. Two interface limits matter here that I'm aware of.. although it might not be limited to these:

1. get_num_new_matched_tokens() is prefix-count shaped. It cannot represent segmented/non-prefix reuse like “reuse these spans, compute these gaps/halos.”
2. Once external tokens are reported, vLLM largely treats them as computed target-prefix KV.. I think.. For semantic donor KV, we need a request-scoped/cache-commit policy so approximate donor KV is not published into the exact prefix cache.

Basically, I am thinking about this body of work, roughly in these stages:

1. Discovery-only OOT connector -> run donor lookup and emit hit/miss/reject metrics...
2. Exact-equivalent/prefix-shaped materialization only -> return positive matched tokens only when the donor KV is valid for the target prefix shape
3. EXACT_COMMIT vs REQUEST_ONLY (or something like this) external KV cache policy -> let external KV be marked exact-cacheable or current-request-only, preventing semantic cache pollution
4. Single-donor exact-token multi-segment reuse -> add a segmented plan for reordered/repeated spans, but start with exact tokens before fuzzy semantics
5. Fuzzy/semantic multi-segment reuse -> enable after quality gates, RoPE/position handling, halo recompute, backend validation, sparse/partial attention support, and metrics prove it

I think numbers 1 and 2 can be OOT.. 3, 4, 5 entail updates I think.

[AasthaPJoshi]

One upstream piece that may be worth standardizing early is reuse provenance, even if the semantic provider itself stays fully out-of-tree.

The RFC already separates exact cache state from request-only semantic reuse, but from an operator / higher-layer perspective those outcomes are materially different. I’d want a small provider-neutral result vocabulary exposed through connector stats/events, e.g.:

• exact local prefix hit
• exact external KV hit
• request-only semantic reuse
• semantic candidate rejected
• fallback/recompute

That would let gateways, eval systems, and agent runtimes correlate latency/cost wins with the actual reuse path instead of seeing only aggregate “cache hit” numbers.

This feels especially important for agentic workloads: a semantic reuse path can be fast and still change downstream tool selection or trajectory. If provenance is explicit, quality regressions can be joined back to the cache outcome during evaluation.

I’d probably keep semantic-specific similarity/quality fields connector-private initially, but standardize the class of reuse + fallback reason at the vLLM boundary if multiple external-KV systems are expected to coexist.