LMCache-Ascend is a community maintained plugin for running LMCache on the Ascend NPU.

Maintenance notice: The stable in-process version (LMCacheAscendConnectorV1Dynamic) is maintained on the dsv4_support_045 branch and is tightly coupled with LMCache 0.4.5 — pair it with the LMCache 0.4.5 release line, not newer versions. The main branch moves to MP mode (LMCacheMPConnector) and will no longer maintain the in-process mode.

To use LMCache-Ascend on the NPU hardware, please make sure the following prerequisites are satisfied.

• Hardware: Atlas 800I A2 Inference series. (A3 Inference/Training and 300I Duo are experimental).
• OS: Linux-based.
• Software:
  • Python: >= 3.10
  • CANN Toolkit: >= 8.5.0
  • Ascend Driver: >= 25.5
  • PyTorch: >= 2.8.0
  • vLLM: >= v0.23.0 & vLLM-Ascend: >= v0.23.0
• Container preparation: see the official vLLM-Ascend tutorials for preparing the base environment (NPU driver, CANN toolkit, and container images).

Please ensure your environment matches the versions below.

The minimal deployment is three steps: install the two packages, start the lmcache server, then start vLLM with the connector.

git clone https://github.com/LMCache/LMCache.git
cd LMCache
pip install -v --no-build-isolation -e .
cd ..

git clone --recurse-submodules https://github.com/LMCache/LMCache-Ascend.git
cd LMCache-Ascend
pip install -v --no-build-isolation -e .

third_party/hcomm must match the container's CANN version — see docs/mp_mode/lmcache_driven.md for the full installation guide (environment preparation, container startup, source installation).

lmcache server \
    --chunk-size 128 \
    --l1-size-gb 50 \
    --eviction-policy LRU

Wait for LMCache INFO: LMCache zmq cache server is running on localhost:5555.

--chunk-size 128 matches the vLLM block size and suits single-KV-format models (e.g. Qwen3); DeepSeek-V4 requires 4096 — see docs/mp_mode/lmcache_driven.md for the full deployment guide.

vllm serve /path/to/Qwen3-32B \
    --served-model-name qwen3 \
    --max-model-len 40960 \
    --max-num-batched-tokens 8192 \
    --gpu-memory-utilization 0.9 \
    --max-num-seqs 32 \
    --tensor-parallel-size 8 \
    --no-enable-prefix-caching \
    --port 8900 \
    --kv-transfer-config '{
      "kv_connector": "LMCacheMPConnector",
      "kv_role": "kv_both"
    }'

The server address defaults to localhost:5555 on both sides; on NPU the transfer mode defaults to lmcache_driven, so no extra config is needed here. A successful start shows LMCache INFO: Creating transfer context (device_type=npu, mode=lmcache_driven) for every worker.

For the engine_driven transfer mode (workers copy KV from a named SHM pool), see docs/mp_mode/engine_driven.md.

• docs/mp_mode/README.md — MP mode overview.
• docs/mp_mode/lmcache_driven.md — lmcache_driven full deployment guide (environment preparation, container, installation, server, vLLM, verification).
• docs/mp_mode/engine_driven.md — engine_driven deployment guide (Qwen3 example).
• docs/mp_mode/qwen3_5.md — hybrid Mamba models (for example Qwen3.5, Qwen3.6, and Qwen3.8): same flags as the LMCache recipe, and where Ascend differs.