KV Cache Memory Estimation Error for GLM-4.7-Flash-AWQ on V100

[windreamer]

Description

When running GLM-4.7-Flash-AWQ on a single V100-32G-SXM2, the KV Cache memory estimation appears to be incorrect, causing premature context length truncation.

Environment

• GPU: NVIDIA V100-32G-SXM2
• CUDA: 12.9
• Model: GLM-4.7-Flash-AWQ (18.4GB weights)
• Framework: LMDeploy (TurboMind backend) with PR GLM-4.7-Flash Turbomind support #4362

Observed Behavior

• Context length initialized: Only 4928 tokens (severely truncated)
• Expected behavior: Should support much longer context with 32GB VRAM

Error Logs

[TM][WARNING] [TM] `max_context_token_num` is not set, default to 202752.
[TM][WARNING] [SegMgr] prefix caching is enabled
[TM][WARNING] `session_len` truncated to 4928 due to limited KV cache memory
[TM][ERROR] [Engine] Warm-up for 6144 tokens failed with status 6
[TM][ERROR] [Engine] Warm-up for 8192 tokens failed with status 6
[TM][ERROR] [Engine] Warm-up for 8320 tokens failed with status 6

Root Cause Analysis

The issue appears to be in the block manager's maximum block calculation logic. The session length is truncated based on:

const auto max_cached_tokens = seq_mgr_->max_block_count() * (size_t)cache_block_seq_len * param_.attn_cp_size;
session_len_trunc_ = std::min(max_cached_tokens, (size_t)param_.session_len);

Located at: lmdeploy/src/turbomind/engine/engine.cc (lines 248-253)

Questions

1. Is the 2.6MB/token KV Cache usage expected for this model configuration?
2. Why does the block manager underestimate available memory on V100?

@windreamer 我在单块V100-32G-SXM2上测试了这个PR，可以跑通，但是KV Cache太大了，GLM-4.7-Flash-AWQ权重18.4GB，启动后总共占用31GB，但是只初始化了4928的上下文，2.6MB/token的缓存占用是否过于恐怖

Active code page: 65001
Add dll path C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.9\bin, please note cuda version should >= 11.3 when compiled with cuda 11
The following generation flags are not valid and may be ignored: ['top_p']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
2026-02-24 15:35:59,808 - lmdeploy - WARNING - converter.py:67 - data type fallback to float16 since torch.cuda.is_bf16_supported is False
[TM][WARNING] [TM] `max_context_token_num` is not set, default to 202752.
2026-02-24 15:36:01,275 - lmdeploy - WARNING - turbomind.py:246 - get 27431 model params
[TM][WARNING] [SegMgr] prefix caching is enabled
[TM][WARNING] `session_len` truncated to 4928 due to limited KV cache memory
[TM][ERROR] [Engine] Warm-up for 6144 tokens failed with status 6
[TM][ERROR] [Engine] Warm-up for 8192 tokens failed with status 6
[TM][ERROR] [Engine] Warm-up for 8320 tokens failed with status 6
HINT:    Please open http://127.0.0.1:10002 in a browser for detailed api usage!!!
HINT:    Please open http://127.0.0.1:10002 in a browser for detailed api usage!!!
HINT:    Please open http://127.0.0.1:10002 in a browser for detailed api usage!!!
INFO:     Started server process [7100]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:10002 (Press CTRL+C to quit)

Originally posted by @lingyezhixing in #4283

[windreamer]

@lingyezhixing 我们可以在这里继续讨论

[lingyezhixing]

I've found the problem. The current MLA implementation is incorrect. After attempting to fix it using vibe coding (Claude code + GLM4.7), the model can now load a 100K context, although inference produces gibberish and logs show errors. But the issue should indeed be this one. I'm not very familiar with this project, so sorry I can't help fix it:

MLA Architecture (DeepSeek V2/V3, GLM-4 Flash)
In correct MLA:

• Input → kv_a_proj → kv_a latent representation (512 dimensions, shared across all heads)
• kv_a → kv_b_proj → projected KV (20 heads × 448 dimensions)
• KV cache should store: kv_a (512 dim, 1 "head")
• At inference time: Recompute kv_b from cached kv_a

Current LMDeploy Implementation

• kv_a (512 dim) → kv_b_proj → kv_b (20 × 448 dim)
• MLACopyQKV assembles q, kv_a, kv_b into QKV buffer
• Problem: The QKV buffer contains kv_b (projected per-head data)
• invokeProcessKV_v2 writes the projected data to KV cache
• KV cache stores: 20 heads × 576 dim × 2 (K+V) = 46,080 bytes/token/layer

@echo off
set CUDA_VISIBLE_DEVICES=1
chcp 65001

conda activate lmdeploy && lmdeploy serve api_server E:\models\LLM\GLM-4.7-Flash-AWQ --server-name 0.0.0.0 --server-port 8080 --model-name GLM-4.7-Flash --backend turbomind --model-format awq --cache-max-entry-count 0.8 --enable-prefix-caching

(lmdeploy) D:\LLM\LLM-Manager\Model_startup_script>GLM-4.7-Flash-AWQ.bat
Active code page: 65001
Add dll path C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.9\bin, please note cuda version should >= 11.3 when compiled with cuda 11
The following generation flags are not valid and may be ignored: ['top_p']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
2026-02-24 21:39:59,402 - lmdeploy - WARNING - converter.py:67 - data type fallback to float16 since torch.cuda.is_bf16_supported is False
[TM][WARNING] [TM] `max_context_token_num` is not set, default to 202752.
2026-02-24 21:40:00,926 - lmdeploy - WARNING - turbomind.py:246 - get 27431 model params
[TM][WARNING] [SegMgr] prefix caching is enabled
[TM][WARNING] `session_len` truncated to 94464 due to limited KV cache memory
HINT:    Please open http://0.0.0.0:8080 in a browser for detailed api usage!!!
HINT:    Please open http://0.0.0.0:8080 in a browser for detailed api usage!!!
HINT:    Please open http://0.0.0.0:8080 in a browser for detailed api usage!!!
INFO:     Started server process [35780]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
INFO:     192.168.50.10:41286 - "POST /v1/chat/completions HTTP/1.1" 404 Not Found
INFO:     192.168.50.10:48958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[TM][WARNING] [ProcessInferRequests] [1] total sequence length (6 + 202746) exceeds `session_len` (94464), `max_new_tokens` is truncated to 94458
[2026-02-24 21:41:15 DP0] Avg thr (in/out): 0.6 / 9.6 tokens/s, API server (completed/routed/waiting): 0 / 1 / 0, Engine (running/waiting): 1 / 0, KV cache: 40.4%,
2026-02-24 21:41:25,135 - lmdeploy - ERROR - async_engine.py:271 - [safe_run] session 1 exception caught: CancelledError Cancelled via cancel scope 2ecbb74a3f0 by <Task pending name='Task-10' coro=<RequestResponseCycle.run_asgi() running at E:\Programming\pycodes\miniconda3\envs\lmdeploy\Lib\site-packages\uvicorn\protocols\http\h11_impl.py:403> cb=[set.discard()]>
[2026-02-24 21:41:25 DP0] Avg thr (in/out): 0.0 / 32.0 tokens/s, API server (completed/routed/waiting): 0 / 0 / 1, Engine (running/waiting): 1 / 0, KV cache: 40.8%,
INFO:     192.168.50.10:53150 - "POST /v1/chat/completions HTTP/1.1" 404 Not Found
INFO:     192.168.50.10:36254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[TM][WARNING] [ProcessInferRequests] [2] total sequence length (6 + 202746) exceeds `session_len` (94464), `max_new_tokens` is truncated to 94458
2026-02-24 21:41:33,758 - lmdeploy - ERROR - async_engine.py:271 - [safe_run] session 2 exception caught: CancelledError Cancelled via cancel scope 2ecbb7c6210 by <Task pending name='Task-15' coro=<RequestResponseCycle.run_asgi() running at E:\Programming\pycodes\miniconda3\envs\lmdeploy\Lib\site-packages\uvicorn\protocols\http\h11_impl.py:403> cb=[set.discard()]>
[2026-02-24 21:41:35 DP0] Avg thr (in/out): 0.6 / 19.5 tokens/s, API server (completed/routed/waiting): 0 / 0 / 2, Engine (running/waiting): 1 / 0, KV cache: 41.1%,

[valentijnvenus]

Eager to see this addressed too! if you could make your code available @lingyezhixing happy to test this on 4x v100. Which I assume is as simple as:

set CUDA_VISIBLE_DEVICES=0,1,2,3

Any other lmdeploy arguments to add for multiple cuda devices?