[Misc]: Question about prefix cache

[LIULIU7450]

Anything you want to discuss about ucm.

Hi, if I disable VLLM's prefix caching (--no-enable-prefix-caching) and use UCM's NFS storage to offload the KV cache to an SSD, is there still kv/prefix cache in HBM?
And I noticed that when I enabled VLLM's prefix caching (--enable-prefix-caching) and tested offloading the KV cache to the SSD, the KV cache files also generated on the SSD at the start of the program, even though my available GPU memory was much larger than the model size.
At this time, is the KV cache in the SSD the same as the KV cache in HBM?

[ygwpz]

1. if you add --no-enable-prefix-caching param, no prefix cache in HBM
2. if --enable-prefix-caching, kvcache in hbm is subset of kvcache in ssd

Anything you want to discuss about ucm.

Hi, if I disable VLLM's prefix caching (--no-enable-prefix-caching) and use UCM's NFS storage to offload the KV cache to an SSD, is there still kv/prefix cache in HBM? And I noticed that when I enabled VLLM's prefix caching (--enable-prefix-caching) and tested offloading the KV cache to the SSD, the KV cache files also generated on the SSD at the start of the program, even though my available GPU memory was much larger than the model size. At this time, is the KV cache in the SSD the same as the KV cache in HBM?

[LIULIU7450]

@ygwpz Hi, Thank you for your prompt reply！And I have another question. When I use DRAM Connector to offload KV Cache, I want to limit it to using only 150GB for store KV Cache.My configuration is as follow:

vllm serve \
    ./model/Meta-Llama-3.1-8B-Instruct/ \
    --port 6001 \
    --tensor-parallel-size 2 \
    --trust-remote-code \
    --kv-transfer-config \
    '{
        "kv_connector": "UnifiedCacheConnectorV1",
        "kv_connector_module_path": "ucm.integration.vllm.uc_connector",
        "kv_role": "kv_both",
        "kv_connector_extra_config": {
            "ucm_connector_name": "UcmDramStore",
            "ucm_connector_config": {
                "max_cache_size": 161061273600, #161061273600/1024^3=150GB
                "kv_block_size": 262144
            }
        }
    }'

When the program starts, the log shows “Dram cache usage exceeds limit! No more kv cache offload! Try to increase your initial max_cache_size”.but at this time the kv cache data has not grown to 150GB.
Is there something wrong with my max_cache_size settings?

[LIULIU7450]

I ran PD disaggregated using NFS connector on a single node with a 1 prefiller + 1 decoder setup.
Decoder log report:
Failed(-50003) to new block(121916496162224340395277335252610861816). [2648612,2648612][space_manager.cc:59,NewBlock]
How to solve this problem？

[harrisonyhq]

@qyh111 Take a look at this :)

[qyh111]

-50003 typically indicates a duplicate creation. D nodes usually don't require a create operation. One possibility is that the request length is exactly divisible by the block size. Due to VLLM limitations, we might assume the last block was missed, causing the D node to attempt to create it and dump, resulting in a duplicate creation. This is a minor bug but doesn't affect functionality.

[LIULIU7450]

@qyh111 Thank you for your reply! I ran PD disaggregated using NFS connector on a single node with a 1 prefiller + 1 decoder setup. In multi-turn & multi-users chat scenarios, the TTFT of PD disaggregated is higher than that of --tensor-parallel-size 2. Is this result correct?
This is my PD setting:

vllm serve \
    ./Meta-Llama-3.1-8B-Instruct/ \
    --port 6000 \
    --tensor-parallel-size 1 \
    --enforce-eager \
    --block_size 128 \
    --trust-remote-code \
    --kv-transfer-config \
    '{
        "kv_connector": "UnifiedCacheConnectorV1",
        "kv_connector_module_path": "ucm.integration.vllm.uc_connector",
        "kv_role": "kv_producer",
        "kv_connector_extra_config": {
            "ucm_connector_name": "UcmNfsStore",
            "ucm_connector_config": {
                "storage_backends": "./local_disk/",
                "transferStreamNumber":32
            }
        }
    }'

vllm serve \
    ./Meta-Llama-3.1-8B-Instruct/ \
    --port 6001 \
    --tensor-parallel-size 1 \
    --enforce-eager \
    --block_size 128 \
    --trust-remote-code \
    --kv-transfer-config \
    '{
        "kv_connector": "UnifiedCacheConnectorV1",
        "kv_connector_module_path": "ucm.integration.vllm.uc_connector",
        "kv_role": "kv_consumer",
        "kv_connector_extra_config": {
            "ucm_connector_name": "UcmNfsStore",
            "ucm_connector_config": {
                "storage_backends": "./local_disk/",
                "transferStreamNumber":32
            }
        }
    }'

This is my tensor-parallel setting:

vllm serve \
   ./Meta-Llama-3.1-8B-Instruct/ \
    --port 6001 \
    --block_size 128 \
    --trust-remote-code \
    --kv-transfer-config \
    '{
        "kv_connector": "UnifiedCacheConnectorV1",
        "kv_connector_module_path": "ucm.integration.vllm.uc_connector",
        "kv_role": "kv_both",
        "kv_connector_extra_config": {
            "ucm_connector_name": "UcmNfsStore",
            "ucm_connector_config": {
                "storage_backends": "./local_disk/",
                "transferStreamNumber":32
            }
        }
    }'

[LIULIU7450]

-50003通常表示重复创建。D节点通常不需要创建操作。一种可能性是请求长度正好可以被块大小整除。由于VLLM的限制，我们可能会假设最后一个块丢失，导致D节点尝试创建它并转储，从而导致重复创建。这是一个小错误，但不影响功能。

I found that Prefiller node also report a large number of -50003 error. and I want to know if this error will affect performance?

[ygwpz]

no effect to performance, and we will fix this error later