[Performance]: lmcache cannot work！

[tensorflowt]

Your current environment

My test environment is as follow：

Environment: 2x H100 with NVLink

My testing method：

# P instance
UCX_TLS=cuda_ipc,cuda_copy,tcp \
        LMCACHE_CONFIG_FILE=/work/vllm-0.8.5/examples/lmcache/disagg_prefill_lmcache_v1/configs/lmcache-prefiller-config.yaml \
        LMCACHE_USE_EXPERIMENTAL=True \
        VLLM_ENABLE_V1_MULTIPROCESSING=1 \
        VLLM_WORKER_MULTIPROC_METHOD=spawn \
        CUDA_VISIBLE_DEVICES=0 \
        vllm serve /work/Qwen/Qwen2.5-0.5B-Instruct \
        --port 8100 \
        --disable-log-requests \
        --enforce-eager \
        --kv-transfer-config \
        '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_producer","kv_connector_extra_config": {"discard_partial_chunks": false, "lmcache_rpc_port": "producer1"}}'

# D instance
 UCX_TLS=cuda_ipc,cuda_copy,tcp \
        LMCACHE_CONFIG_FILE=/work/vllm-0.8.5/examples/lmcache/disagg_prefill_lmcache_v1/configs/lmcache-decoder-config.yaml \
        LMCACHE_USE_EXPERIMENTAL=True \
        VLLM_ENABLE_V1_MULTIPROCESSING=1 \
        VLLM_WORKER_MULTIPROC_METHOD=spawn \
        CUDA_VISIBLE_DEVICES=1 \
        vllm serve /work/Qwen/Qwen2.5-0.5B-Instruct \
        --port 8200 \
        --disable-log-requests \
        --enforce-eager \
        --kv-transfer-config \
        '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_consumer","kv_connector_extra_config": {"discard_partial_chunks": false, "lmcache_rpc_port": "consumer1"}}'

# proxy
python3 /work/vllm-0.8.5/examples/lmcache/disagg_prefill_lmcache_v1/disagg_proxy_server.py \
        --host localhost \
        --port 9000 \
        --prefiller-host localhost \
        --prefiller-port 8100 \
        --decoder-host localhost \
        --decoder-port 8200

Errors encountered during operation：

UCX_TLS=cuda_ipc,cuda_copy,tcp \
>         LMCACHE_CONFIG_FILE=/work/vllm-0.8.5/examples/lmcache/disagg_prefill_lmcache_v1/configs/lmcache-prefiller-config.yaml \
>         LMCACHE_USE_EXPERIMENTAL=True \
>         VLLM_ENABLE_V1_MULTIPROCESSING=1 \
>         VLLM_WORKER_MULTIPROC_METHOD=spawn \
>         CUDA_VISIBLE_DEVICES=0 \
>         vllm serve /work/Qwen/Qwen2.5-0.5B-Instruct \
>         --port 8100 \
>         --disable-log-requests \
>         --enforce-eager \
>         --kv-transfer-config \
>         '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_producer","kv_connector_extra_config": {"discard_partial_chunks": false, "lmcache_rpc_port": "producer1"}}'

INFO 05-14 02:41:16 [__init__.py:239] Automatically detected platform cuda.
INFO 05-14 02:41:20 [api_server.py:1043] vLLM API server version 0.8.5
INFO 05-14 02:41:20 [api_server.py:1044] args: Namespace(subparser='serve', model_tag='/work/Qwen/Qwen2.5-0.5B-Instruct', config='', host=None, port=8100, uvicorn_log_level='info', disable_uvicorn_access_log=False, allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, enable_ssl_refresh=False, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='/work/Qwen/Qwen2.5-0.5B-Instruct', task='auto', tokenizer=None, hf_config_path=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, load_format='auto', download_dir=None, model_loader_extra_config={}, use_tqdm_on_load=True, config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', max_model_len=None, guided_decoding_backend='auto', reasoning_parser=None, logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=1, data_parallel_size=1, enable_expert_parallel=False, max_parallel_loading_workers=None, ray_workers_use_nsight=False, disable_custom_all_reduce=False, block_size=None, gpu_memory_utilization=0.9, swap_space=4, kv_cache_dtype='auto', num_gpu_blocks_override=None, enable_prefix_caching=None, prefix_caching_hash_algo='builtin', cpu_offload_gb=0, calculate_kv_scales=False, disable_sliding_window=False, use_v2_block_manager=True, seed=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_token=None, hf_overrides=None, enforce_eager=True, max_seq_len_to_capture=8192, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config={}, limit_mm_per_prompt={}, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=None, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=None, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', speculative_config=None, ignore_patterns=[], served_model_name=None, qlora_adapter_name_or_path=None, show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, max_num_batched_tokens=None, max_num_seqs=None, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, num_lookahead_slots=0, scheduler_delay_factor=0.0, preemption_mode=None, num_scheduler_steps=1, multi_step_stream_outputs=True, scheduling_policy='fcfs', enable_chunked_prefill=None, disable_chunked_mm_input=False, scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=KVTransferConfig(kv_connector='LMCacheConnectorV1', kv_buffer_device='cuda', kv_buffer_size=1000000000.0, kv_role='kv_producer', kv_rank=None, kv_parallel_size=1, kv_ip='127.0.0.1', kv_port=14579, kv_connector_extra_config={'discard_partial_chunks': False, 'lmcache_rpc_port': 'producer1'}), worker_cls='auto', worker_extension_cls='', generation_config='auto', override_generation_config=None, enable_sleep_mode=False, additional_config=None, enable_reasoning=False, disable_cascade_attn=False, disable_log_requests=True, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, enable_server_load_tracking=False, dispatch_function=<function ServeSubcommand.cmd at 0x7f5910656e60>)
WARNING 05-14 02:41:20 [utils.py:2267] Found ulimit of 1024 and failed to automatically increase with error current limit exceeds maximum limit. This can cause fd limit errors like `OSError: [Errno 24] Too many open files`. Consider increasing with ulimit -n
INFO 05-14 02:41:28 [config.py:717] This model supports multiple tasks: {'reward', 'generate', 'embed', 'classify', 'score'}. Defaulting to 'generate'.
INFO 05-14 02:41:28 [config.py:2003] Chunked prefill is enabled with max_num_batched_tokens=8192.
WARNING 05-14 02:41:28 [cuda.py:93] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used
INFO 05-14 02:41:32 [__init__.py:239] Automatically detected platform cuda.
INFO 05-14 02:41:35 [core.py:58] Initializing a V1 LLM engine (v0.8.5) with config: model='/work/Qwen/Qwen2.5-0.5B-Instruct', speculative_config=None, tokenizer='/work/Qwen/Qwen2.5-0.5B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='auto', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=None, served_model_name=/work/Qwen/Qwen2.5-0.5B-Instruct, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=False, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[],"max_capture_size":0}
WARNING 05-14 02:41:35 [utils.py:2522] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7f79a4660d60>
[rank0]:[W514 02:41:36.836683514 ProcessGroupGloo.cpp:727] Warning: Unable to resolve hostname to a (local) address. Using the loopback address as fallback. Manually set the network interface to bind to with GLOO_SOCKET_IFNAME. (function operator())
INFO 05-14 02:41:36 [parallel_state.py:1004] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0
INFO 05-14 02:41:36 [factory.py:64] Creating v1 connector with name: LMCacheConnectorV1
WARNING 05-14 02:41:36 [base.py:58] Initializing KVConnectorBase_V1. This API is experimental and subject to change in the future as we iterate the design.
[2025-05-14 02:41:36,425] LMCache INFO: Loading LMCache config file /work/vllm-0.8.5/examples/lmcache/disagg_prefill_lmcache_v1/configs/lmcache-prefiller-config.yaml (utils.py:32:lmcache.integration.vllm.utils)
[2025-05-14 02:41:36,428] LMCache INFO: Creating LMCacheEngine instance vllm-instance (cache_engine.py:444:lmcache.experimental.cache_engine)
[2025-05-14 02:41:36,428] LMCache INFO: Creating LMCacheEngine with config: LMCacheEngineConfig(chunk_size=256, local_cpu=False, max_local_cpu_size=0, local_disk=None, max_local_disk_size=0, remote_url=None, remote_serde=None, save_decode_cache=False, enable_blending=False, blend_recompute_ratio=0.15, blend_min_tokens=256, blend_special_str=' # # ', enable_p2p=False, lookup_url=None, distributed_url=None, error_handling=False, enable_controller=False, lmcache_instance_id='lmcache_default_instance', controller_url=None, lmcache_worker_url=None, enable_nixl=True, nixl_role='sender', nixl_peer_host='localhost', nixl_peer_port=55555, nixl_buffer_size=1073741824, nixl_buffer_device='cuda', nixl_enable_gc=True) (cache_engine.py:73:lmcache.experimental.cache_engine)
[1747215696.528332] [b3ab5fdd703b:1615 :0]     ucp_context.c:1268 UCX  WARN  transports 'cuda_ipc','cuda_copy' are not available, please use one or more of: mm, posix, self, shm, sm, sysv, tcp
Backend UCX was instantiated
Initialized NIXL agent: NixlRole.SENDER
ERROR 05-14 02:41:36 [core.py:396] EngineCore failed to start.
ERROR 05-14 02:41:36 [core.py:396] Traceback (most recent call last):
ERROR 05-14 02:41:36 [core.py:396]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 387, in run_engine_core
ERROR 05-14 02:41:36 [core.py:396]     engine_core = EngineCoreProc(*args, **kwargs)
ERROR 05-14 02:41:36 [core.py:396]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 329, in __init__
ERROR 05-14 02:41:36 [core.py:396]     super().__init__(vllm_config, executor_class, log_stats,
ERROR 05-14 02:41:36 [core.py:396]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 64, in __init__
ERROR 05-14 02:41:36 [core.py:396]     self.model_executor = executor_class(vllm_config)
ERROR 05-14 02:41:36 [core.py:396]   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/executor_base.py", line 52, in __init__
ERROR 05-14 02:41:36 [core.py:396]     self._init_executor()
ERROR 05-14 02:41:36 [core.py:396]   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/uniproc_executor.py", line 46, in _init_executor
ERROR 05-14 02:41:36 [core.py:396]     self.collective_rpc("init_device")
ERROR 05-14 02:41:36 [core.py:396]   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
ERROR 05-14 02:41:36 [core.py:396]     answer = run_method(self.driver_worker, method, args, kwargs)
ERROR 05-14 02:41:36 [core.py:396]   File "/usr/local/lib/python3.10/dist-packages/vllm/utils.py", line 2456, in run_method
ERROR 05-14 02:41:36 [core.py:396]     return func(*args, **kwargs)
ERROR 05-14 02:41:36 [core.py:396]   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker_base.py", line 604, in init_device
ERROR 05-14 02:41:36 [core.py:396]     self.worker.init_device()  # type: ignore
ERROR 05-14 02:41:36 [core.py:396]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_worker.py", line 135, in init_device
ERROR 05-14 02:41:36 [core.py:396]     init_worker_distributed_environment(self.vllm_config, self.rank,
ERROR 05-14 02:41:36 [core.py:396]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_worker.py", line 329, in init_worker_distributed_environment
ERROR 05-14 02:41:36 [core.py:396]     ensure_kv_transfer_initialized(vllm_config)
ERROR 05-14 02:41:36 [core.py:396]   File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/kv_transfer/kv_transfer_state.py", line 63, in ensure_kv_transfer_initialized
ERROR 05-14 02:41:36 [core.py:396]     _KV_CONNECTOR_AGENT = KVConnectorFactory.create_connector_v1(
ERROR 05-14 02:41:36 [core.py:396]   File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/kv_transfer/kv_connector/factory.py", line 73, in create_connector_v1
ERROR 05-14 02:41:36 [core.py:396]     return connector_cls(config, role)
ERROR 05-14 02:41:36 [core.py:396]   File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/kv_transfer/kv_connector/v1/lmcache_connector.py", line 25, in __init__
ERROR 05-14 02:41:36 [core.py:396]     self._lmcache_engine = LMCacheConnectorV1Impl(vllm_config, role, self)
ERROR 05-14 02:41:36 [core.py:396]   File "/usr/local/lib/python3.10/dist-packages/lmcache/integration/vllm/vllm_v1_adapter.py", line 314, in __init__
ERROR 05-14 02:41:36 [core.py:396]     self.lmcache_engine = init_lmcache_engine(
ERROR 05-14 02:41:36 [core.py:396]   File "/usr/local/lib/python3.10/dist-packages/lmcache/integration/vllm/vllm_adapter.py", line 111, in init_lmcache_engine
ERROR 05-14 02:41:36 [core.py:396]     engine = LMCacheEngineBuilder.get_or_create(ENGINE_NAME, config, metadata,
ERROR 05-14 02:41:36 [core.py:396]   File "/usr/local/lib/python3.10/dist-packages/lmcache/experimental/cache_engine.py", line 449, in get_or_create
ERROR 05-14 02:41:36 [core.py:396]     engine = LMCacheEngine(config, metadata, memory_allocator,
ERROR 05-14 02:41:36 [core.py:396]   File "/usr/local/lib/python3.10/dist-packages/lmcache/experimental/cache_engine.py", line 98, in __init__
ERROR 05-14 02:41:36 [core.py:396]     self.storage_manager = DistributedStorageManager(
ERROR 05-14 02:41:36 [core.py:396]   File "/usr/local/lib/python3.10/dist-packages/lmcache/experimental/storage_backend/storage_manager.py", line 535, in __init__
ERROR 05-14 02:41:36 [core.py:396]     self.storage_backend = NixlBackend.CreateNixlBackend(config, metadata)
ERROR 05-14 02:41:36 [core.py:396]   File "/usr/local/lib/python3.10/dist-packages/lmcache/experimental/storage_backend/nixl_backend.py", line 412, in CreateNixlBackend
ERROR 05-14 02:41:36 [core.py:396]     backend = NixlBackend(nixl_config)
ERROR 05-14 02:41:36 [core.py:396]   File "/usr/local/lib/python3.10/dist-packages/lmcache/experimental/storage_backend/nixl_backend.py", line 249, in __init__
ERROR 05-14 02:41:36 [core.py:396]     self._nixl_channel = NixlChannel(nixl_config)
ERROR 05-14 02:41:36 [core.py:396]   File "/usr/local/lib/python3.10/dist-packages/lmcache/experimental/storage_backend/connector/nixl_connector_v2.py", line 454, in __init__
ERROR 05-14 02:41:36 [core.py:396]     self._pipe = NixlPipe(nixl_config, self._side_channel)
ERROR 05-14 02:41:36 [core.py:396]   File "/usr/local/lib/python3.10/dist-packages/lmcache/experimental/storage_backend/connector/nixl_connector_v2.py", line 190, in __init__
ERROR 05-14 02:41:36 [core.py:396]     self._reg_descs = self._agent.register_memory(self._transfer_buffers)
ERROR 05-14 02:41:36 [core.py:396]   File "/usr/local/lib/python3.10/dist-packages/nixl/_api.py", line 265, in register_memory
ERROR 05-14 02:41:36 [core.py:396]     self.agent.registerMem(reg_descs, handle_list)
ERROR 05-14 02:41:36 [core.py:396] nixl._bindings.nixlBackendError: NIXL_ERR_BACKEND
Process EngineCore_0:
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 400, in run_engine_core
    raise e
  File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 387, in run_engine_core
    engine_core = EngineCoreProc(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 329, in __init__
    super().__init__(vllm_config, executor_class, log_stats,
  File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 64, in __init__
    self.model_executor = executor_class(vllm_config)
  File "/usr/local/lib/python3.10/dist-packages/vllm/executor/executor_base.py", line 52, in __init__
    self._init_executor()
  File "/usr/local/lib/python3.10/dist-packages/vllm/executor/uniproc_executor.py", line 46, in _init_executor
    self.collective_rpc("init_device")
  File "/usr/local/lib/python3.10/dist-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
    answer = run_method(self.driver_worker, method, args, kwargs)
  File "/usr/local/lib/python3.10/dist-packages/vllm/utils.py", line 2456, in run_method
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker_base.py", line 604, in init_device
    self.worker.init_device()  # type: ignore
  File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_worker.py", line 135, in init_device
    init_worker_distributed_environment(self.vllm_config, self.rank,
  File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_worker.py", line 329, in init_worker_distributed_environment
    ensure_kv_transfer_initialized(vllm_config)
  File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/kv_transfer/kv_transfer_state.py", line 63, in ensure_kv_transfer_initialized
    _KV_CONNECTOR_AGENT = KVConnectorFactory.create_connector_v1(
  File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/kv_transfer/kv_connector/factory.py", line 73, in create_connector_v1
    return connector_cls(config, role)
  File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/kv_transfer/kv_connector/v1/lmcache_connector.py", line 25, in __init__
    self._lmcache_engine = LMCacheConnectorV1Impl(vllm_config, role, self)
  File "/usr/local/lib/python3.10/dist-packages/lmcache/integration/vllm/vllm_v1_adapter.py", line 314, in __init__
    self.lmcache_engine = init_lmcache_engine(
  File "/usr/local/lib/python3.10/dist-packages/lmcache/integration/vllm/vllm_adapter.py", line 111, in init_lmcache_engine
    engine = LMCacheEngineBuilder.get_or_create(ENGINE_NAME, config, metadata,
  File "/usr/local/lib/python3.10/dist-packages/lmcache/experimental/cache_engine.py", line 449, in get_or_create
    engine = LMCacheEngine(config, metadata, memory_allocator,
  File "/usr/local/lib/python3.10/dist-packages/lmcache/experimental/cache_engine.py", line 98, in __init__
    self.storage_manager = DistributedStorageManager(
  File "/usr/local/lib/python3.10/dist-packages/lmcache/experimental/storage_backend/storage_manager.py", line 535, in __init__
    self.storage_backend = NixlBackend.CreateNixlBackend(config, metadata)
  File "/usr/local/lib/python3.10/dist-packages/lmcache/experimental/storage_backend/nixl_backend.py", line 412, in CreateNixlBackend
    backend = NixlBackend(nixl_config)
  File "/usr/local/lib/python3.10/dist-packages/lmcache/experimental/storage_backend/nixl_backend.py", line 249, in __init__
    self._nixl_channel = NixlChannel(nixl_config)
  File "/usr/local/lib/python3.10/dist-packages/lmcache/experimental/storage_backend/connector/nixl_connector_v2.py", line 454, in __init__
    self._pipe = NixlPipe(nixl_config, self._side_channel)
  File "/usr/local/lib/python3.10/dist-packages/lmcache/experimental/storage_backend/connector/nixl_connector_v2.py", line 190, in __init__
    self._reg_descs = self._agent.register_memory(self._transfer_buffers)
  File "/usr/local/lib/python3.10/dist-packages/nixl/_api.py", line 265, in register_memory
    self.agent.registerMem(reg_descs, handle_list)
nixl._bindings.nixlBackendError: NIXL_ERR_BACKEND
^CException ignored in: <function Context.__del__ at 0x7f7a84a380d0>
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/zmq/sugar/context.py", line 142, in __del__
    self.destroy()
  File "/usr/local/lib/python3.10/dist-packages/zmq/sugar/context.py", line 324, in destroy
    self.term()
  File "/usr/local/lib/python3.10/dist-packages/zmq/sugar/context.py", line 266, in term
    super().term()
  File "_zmq.py", line 564, in zmq.backend.cython._zmq.Context.term
  File "_zmq.py", line 160, in zmq.backend.cython._zmq._check_rc
  File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 371, in signal_handler
    raise SystemExit()
SystemExit:
[rank0]:[W514 02:57:14.981244545 ProcessGroupNCCL.cpp:1496] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
^CTraceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 398, in __init__
    self._wait_for_engine_startup()
  File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 423, in _wait_for_engine_startup
    events = poller.poll(STARTUP_POLL_PERIOD_MS)
  File "/usr/local/lib/python3.10/dist-packages/zmq/sugar/poll.py", line 106, in poll
    return zmq_poll(self.sockets, timeout=timeout)
  File "_zmq.py", line 1555, in zmq.backend.cython._zmq.zmq_poll
  File "_zmq.py", line 160, in zmq.backend.cython._zmq._check_rc
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/vllm", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/cli/main.py", line 53, in main
    args.dispatch_function(args)
  File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/cli/serve.py", line 27, in cmd
    uvloop.run(run_server(args))
  File "/usr/local/lib/python3.10/dist-packages/uvloop/__init__.py", line 82, in run
    return loop.run_until_complete(wrapper())
  File "uvloop/loop.pyx", line 1511, in uvloop.loop.Loop.run_until_complete
  File "uvloop/loop.pyx", line 1504, in uvloop.loop.Loop.run_until_complete
  File "uvloop/loop.pyx", line 1377, in uvloop.loop.Loop.run_forever
  File "uvloop/loop.pyx", line 555, in uvloop.loop.Loop._run
  File "uvloop/loop.pyx", line 474, in uvloop.loop.Loop._on_idle
  File "uvloop/cbhandles.pyx", line 83, in uvloop.loop.Handle._run
  File "uvloop/cbhandles.pyx", line 61, in uvloop.loop.Handle._run
  File "/usr/local/lib/python3.10/dist-packages/uvloop/__init__.py", line 61, in wrapper
    return await main
  File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 1078, in run_server
    async with build_async_engine_client(args) as engine_client:
  File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 146, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
  File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 178, in build_async_engine_client_from_engine_args
    async_llm = AsyncLLM.from_vllm_config(
  File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/async_llm.py", line 150, in from_vllm_config
    return cls(
  File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/async_llm.py", line 118, in __init__
    self.engine_core = core_client_class(
  File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 642, in __init__
    super().__init__(
  File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 410, in __init__
    self._finalizer()
  File "/usr/lib/python3.10/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 319, in __call__
    core_engine.close()
  File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 295, in close
    proc_handle.shutdown()
  File "/usr/local/lib/python3.10/dist-packages/vllm/v1/utils.py", line 128, in shutdown
    self._finalizer()
  File "/usr/lib/python3.10/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/usr/local/lib/python3.10/dist-packages/vllm/v1/utils.py", line 137, in shutdown
    proc.join(5)
  File "/usr/lib/python3.10/multiprocessing/process.py", line 149, in join
    res = self._popen.wait(timeout)
  File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 40, in wait
    if not wait([self.sentinel], timeout):
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 931, in wait
    ready = selector.select(timeout)
  File "/usr/lib/python3.10/selectors.py", line 416, in select
    fd_event_list = self._selector.poll(timeout)

pip list

Package                                  Version
---------------------------------------- --------------------
accelerate                               0.34.0
aiofiles                                 23.2.1
aiohappyeyeballs                         2.4.0
aiohttp                                  3.10.5
aiohttp-cors                             0.7.0
aioprometheus                            23.12.0
aiosignal                                1.3.1
airportsdata                             20241001
aliyun-python-sdk-core                   2.16.0
aliyun-python-sdk-kms                    2.16.5
altair                                   5.5.0
annotated-types                          0.7.0
antlr4-python3-runtime                   4.9.3
anyascii                                 0.3.2
anyio                                    4.4.0
argcomplete                              3.5.3
astor                                    0.8.1
async-timeout                            4.0.3
attrdict                                 2.0.1
attrs                                    24.2.0
audioread                                3.0.1
auto_gptq                                0.7.1
autoawq                                  0.2.5
autoawq_kernels                          0.0.9
av                                       14.0.1
babel                                    2.16.0
bcrypt                                   4.2.1
beautifulsoup4                           4.12.3
bitsandbytes                             0.45.1
black                                    24.10.0
blake3                                   1.0.4
blinker                                  1.9.0
boto3                                    1.28.64
botocore                                 1.31.85
cached_path                              1.6.7
cachetools                               5.5.1
cdifflib                                 1.2.9
certifi                                  2019.11.28
cffi                                     1.17.1
chardet                                  3.0.4
charset-normalizer                       3.3.2
chattts                                  0.2.1
click                                    8.1.7
cloudpickle                              3.0.0
cn2an                                    0.5.23
colorama                                 0.4.6
coloredlogs                              15.0.1
colorful                                 0.5.6
compressed-tensors                       0.9.3
conformer                                0.3.2
contourpy                                1.3.1
controlnet-aux                           0.0.7
crcmod                                   1.7
cryptography                             44.0.0
cuda-python                              12.6.2.post1
cupy-cuda12x                             13.4.0
cycler                                   0.12.1
Cython                                   3.0.11
dataclass-wizard                         0.35.0
datamodel-code-generator                 0.26.5
datasets                                 2.21.0
dateparser                               1.1.8
dbus-python                              1.2.16
decorator                                5.1.1
decord                                   0.6.0
DeepCache                                0.1.1
Deprecated                               1.2.15
depyf                                    0.18.0
diffusers                                0.32.2
dill                                     0.3.8
diskcache                                5.6.3
Distance                                 0.1.3
distlib                                  0.3.9
distro                                   1.9.0
distro-info                              0.23+ubuntu1.1
dnspython                                2.7.0
docopt                                   0.6.2
ecdsa                                    0.19.0
editdistance                             0.8.1
einops                                   0.8.0
einx                                     0.3.0
email_validator                          2.2.0
encodec                                  0.1.1
eva-decord                               0.6.1
exceptiongroup                           1.2.2
fastapi                                  0.115.11
fastapi-cli                              0.0.7
fastrlock                                0.8.3
ffmpy                                    0.5.0
filelock                                 3.17.0
FlagEmbedding                            1.2.11
flashinfer                               0.1.6+cu124torch2.4
Flask                                    3.1.0
flatbuffers                              25.1.21
fonttools                                4.55.5
frozendict                               2.4.6
frozenlist                               1.4.1
fsspec                                   2024.6.1
fugashi                                  1.4.0
funasr                                   1.1.16
fvcore                                   0.1.5.post20221221
g2p-en                                   2.1.0
gdown                                    5.2.0
gekko                                    1.2.1
genson                                   1.3.0
gguf                                     0.16.3
google-api-core                          2.24.0
google-auth                              2.38.0
google-cloud-core                        2.4.1
google-cloud-storage                     2.19.0
google-crc32c                            1.6.0
google-resumable-media                   2.7.2
googleapis-common-protos                 1.66.0
gradio                                   4.26.0
gradio_client                            0.15.1
grpcio                                   1.70.0
gruut                                    2.4.0
gruut-ipa                                0.13.0
gruut-lang-de                            2.0.1
gruut-lang-en                            2.0.1
gruut-lang-es                            2.0.1
gruut-lang-fr                            2.0.2
h11                                      0.14.0
h2                                       4.2.0
hf_transfer                              0.1.8
hf-xet                                   1.0.2
hiredis                                  3.1.0
hpack                                    4.1.0
httpcore                                 1.0.5
httptools                                0.6.1
httpx                                    0.27.2
huggingface-hub                          0.30.1
humanfriendly                            10.0
hydra-core                               1.3.2
Hypercorn                                0.17.3
hyperframe                               6.1.0
HyperPyYAML                              1.2.2
idna                                     2.8
imageio                                  2.37.0
imageio-ffmpeg                           0.6.0
importlib_metadata                       8.0.0
importlib_resources                      6.5.2
infinistore                              0.2.35
inflect                                  5.6.2
iniconfig                                2.0.0
interegular                              0.3.3
iopath                                   0.1.10
isort                                    5.13.2
itsdangerous                             2.2.0
jaconv                                   0.4.0
jamo                                     0.4.1
jieba                                    0.42.1
Jinja2                                   3.1.5
jiter                                    0.5.0
jj-pytorchvideo                          0.1.5
jmespath                                 0.10.0
joblib                                   1.4.2
jsonlines                                1.2.0
jsonschema                               4.23.0
jsonschema-specifications                2023.12.1
kaldiio                                  2.18.0
kiwisolver                               1.4.8
lark                                     1.2.2
lazy_loader                              0.4
libnacl                                  2.1.0
librosa                                  0.10.2.post1
lightning                                2.5.0.post0
lightning-utilities                      0.11.9
linkify-it-py                            2.0.3
llama_cpp_python                         0.3.4
llguidance                               0.7.13
llvmlite                                 0.44.0
lm-format-enforcer                       0.10.11
lmcache                                  0.2.1
loguru                                   0.7.3
loralib                                  0.1.2
markdown-it-py                           3.0.0
MarkupSafe                               2.1.5
matplotlib                               3.10.0
mdit-py-plugins                          0.4.2
mdurl                                    0.1.2
mecab-python3                            1.0.10
memray                                   1.15.0
mistral_common                           1.5.4
modelscope                               1.18.1
mpmath                                   1.3.0
msgpack                                  1.0.8
msgspec                                  0.18.6
multidict                                6.0.5
multiprocess                             0.70.16
mypy-extensions                          1.0.0
nanobind                                 2.6.1
narwhals                                 1.23.0
natsort                                  8.4.0
nemo_text_processing                     1.0.2
nest-asyncio                             1.6.0
networkx                                 3.3
ninja                                    1.11.1.3
nixl                                     0.2.0
nltk                                     3.9.1
num2words                                0.5.14
numba                                    0.61.2
numpy                                    1.26.4
nvidia-cublas-cu12                       12.4.5.8
nvidia-cuda-cupti-cu12                   12.4.127
nvidia-cuda-nvrtc-cu12                   12.4.127
nvidia-cuda-runtime-cu12                 12.4.127
nvidia-cudnn-cu12                        9.1.0.70
nvidia-cufft-cu12                        11.2.1.3
nvidia-curand-cu12                       10.3.5.147
nvidia-cusolver-cu12                     11.6.1.9
nvidia-cusparse-cu12                     12.3.1.170
nvidia-cusparselt-cu12                   0.6.2
nvidia-ml-py                             12.560.30
nvidia-nccl-cu12                         2.21.5
nvidia-nvjitlink-cu12                    12.4.127
nvidia-nvtx-cu12                         12.4.127
nvtx                                     0.2.11
omegaconf                                2.3.0
onnxruntime                              1.20.1
onnxruntime-gpu                          1.16.0
openai                                   1.60.0
opencensus                               0.11.4
opencensus-context                       0.1.3
opencv-contrib-python-headless           4.11.0.86
opencv-python                            4.11.0.86
opencv-python-headless                   4.11.0.86
opentelemetry-api                        1.26.0
opentelemetry-exporter-otlp              1.26.0
opentelemetry-exporter-otlp-proto-common 1.26.0
opentelemetry-exporter-otlp-proto-grpc   1.26.0
opentelemetry-exporter-otlp-proto-http   1.26.0
opentelemetry-proto                      1.26.0
opentelemetry-sdk                        1.26.0
opentelemetry-semantic-conventions       0.47b0
opentelemetry-semantic-conventions-ai    0.4.6
optimum                                  1.23.3
orjson                                   3.10.15
ormsgpack                                1.7.0
oss2                                     2.19.1
outlines                                 0.1.11
outlines_core                            0.1.26
packaging                                24.1
pandas                                   2.2.2
parameterized                            0.9.0
partial-json-parser                      0.2.1.1.post4
passlib                                  1.7.4
pathspec                                 0.12.1
peft                                     0.14.0
pillow                                   10.4.0
pip                                      24.3.1
platformdirs                             4.3.6
pluggy                                   1.5.0
pooch                                    1.8.2
portalocker                              3.1.1
prettytable                              3.14.0
priority                                 2.0.0
proces                                   0.1.7
prometheus_client                        0.20.0
prometheus-fastapi-instrumentator        7.0.0
proto-plus                               1.25.0
protobuf                                 4.25.7
psutil                                   6.0.0
py-cpuinfo                               9.0.0
py-spy                                   0.4.0
pyairports                               2.1.1
pyarrow                                  17.0.0
pyasn1                                   0.6.1
pyasn1_modules                           0.4.1
pybase16384                              0.3.7
pybind11                                 2.13.6
pycountry                                24.6.1
pycparser                                2.22
pycryptodome                             3.21.0
pydantic                                 2.10.6
pydantic_core                            2.27.2
pydub                                    0.25.1
Pygments                                 2.19.1
PyGObject                                3.36.0
pykakasi                                 2.3.0
pynini                                   2.1.5
pynndescent                              0.5.13
pyparsing                                3.2.1
pypinyin                                 0.53.0
PySocks                                  1.7.1
pytest                                   8.3.4
python-apt                               2.0.1+ubuntu0.20.4.1
python-crfsuite                          0.9.11
python-dateutil                          2.9.0.post0
python-dotenv                            1.0.1
python-jose                              3.3.0
python-json-logger                       3.2.1
python-multipart                         0.0.20
python-rapidjson                         1.20
pytorch-lightning                        2.5.0.post0
pytorch-wpe                              0.0.1
pytz                                     2024.1
PyYAML                                   6.0.2
pyzmq                                    26.2.0
quantile-python                          1.1
Quart                                    0.20.0
qwen-vl-utils                            0.0.8
ray                                      2.43.0
redis                                    5.2.1
referencing                              0.35.1
regex                                    2024.7.24
requests                                 2.32.3
requests-unixsocket                      0.2.0
rich                                     13.9.4
rich-toolkit                             0.13.2
rouge                                    1.0.1
rpds-py                                  0.20.0
rsa                                      4.9
ruamel.yaml                              0.18.10
ruamel.yaml.clib                         0.2.12
ruff                                     0.9.3
s3transfer                               0.7.0
sacremoses                               0.1.1
safetensors                              0.4.4
scikit-image                             0.25.0
scikit-learn                             1.6.1
scipy                                    1.15.1
semantic-version                         2.10.0
sentence-transformers                    3.4.0
sentencepiece                            0.2.0
setproctitle                             1.3.4
setuptools                               75.8.0
sglang                                   0.4.1.post7
shellingham                              1.5.4
silero-vad                               5.1.2
six                                      1.17.0
smart-open                               7.1.0
sniffio                                  1.3.1
sortedcontainers                         2.4.0
soundfile                                0.13.0
soupsieve                                2.6
soxr                                     0.5.0.post1
sse-starlette                            2.1.3
starlette                                0.46.0
sympy                                    1.13.1
tabulate                                 0.9.0
taskgroup                                0.2.2
tblib                                    3.0.0
tensorboardX                             2.6.2.2
tensorizer                               2.9.1
termcolor                                2.5.0
text-generation                          0.7.0
textual                                  1.0.0
threadpoolctl                            3.5.0
tifffile                                 2025.1.10
tiktoken                                 0.7.0
timm                                     1.0.14
tokenizers                               0.21.1
toml                                     0.10.2
tomli                                    2.2.1
tomlkit                                  0.12.0
torch                                    2.6.0
torch-complex                            0.4.4
torchao                                  0.10.0
torchaudio                               2.6.0
torchdiffeq                              0.2.5
torchmetrics                             1.6.1
torchvision                              0.21.0
tqdm                                     4.66.5
transformers                             4.51.3
transformers-stream-generator            0.0.5
triton                                   3.2.0
tritonclient                             2.54.0
typer                                    0.15.2
typing_extensions                        4.12.2
tzdata                                   2024.1
tzlocal                                  5.2
uc-micro-py                              1.0.3
umap-learn                               0.5.7
unattended-upgrades                      0.1
unidic-lite                              1.0.8
urllib3                                  2.0.7
uvicorn                                  0.30.6
uvloop                                   0.20.0
vector-quantize-pytorch                  1.17.3
verovio                                  4.5.1
virtualenv                               20.29.2
vllm                                     0.8.5
vllm-flash-attn                          2.6.1
vocos                                    0.1.0
watchfiles                               0.24.0
wcwidth                                  0.2.13
websockets                               11.0.3
Werkzeug                                 3.1.3
WeTextProcessing                         1.0.3
wget                                     3.2
wheel                                    0.34.2
wrapt                                    1.17.2
wsproto                                  1.2.0
x-transformers                           1.44.6
xformers                                 0.0.29.post2
xgrammar                                 0.1.18
xinference                               1.2.2
xoscar                                   0.4.6
xxhash                                   3.5.0
yacs                                     0.1.8
yarl                                     1.9.9
zipp                                     3.20.1
zstandard                                0.23.0

How would you like to use vllm

I want to run inference of a [specific model](put link here). I don't know how to integrate it with vllm.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[Unprincess17]

It looks like ucx is not installed or configured correctly.

[1747215696.528332] [b3ab5fdd703b:1615 :0] ucp_context.c:1268 UCX WARN transports 'cuda_ipc','cuda_copy' are not available, please use one or more of: mm, posix, self, shm, sm, sysv, tcp

I ran into the same issue. And walkarounded by reinstalling ucx and nixl from source.

You can first test with lmcache nixl test.

[GGBond8488]

I met the same issue and when I changed "cuda_ipc','cuda_copy·" to "tcp" I met

  warnings.warn('resource_tracker: There appear to be %d '
(VllmWorker rank=0 pid=36042) ERROR 05-15 09:31:29 [multiproc_executor.py:435]   File "/.local/lib/python3.10/site-packages/lmcache/experimental/storage_backend/nixl_backend.py", line 249, in __init__
(VllmWorker rank=0 pid=36042) ERROR 05-15 09:31:29 [multiproc_executor.py:435]     self._nixl_channel = NixlChannel(nixl_config)
(VllmWorker rank=0 pid=36042) ERROR 05-15 09:31:29 [multiproc_executor.py:435]   File "/.local/lib/python3.10/site-packages/lmcache/experimental/storage_backend/connector/nixl_connector_v2.py", line 454, in __init__
(VllmWorker rank=0 pid=36042) ERROR 05-15 09:31:29 [multiproc_executor.py:435]     self._pipe = NixlPipe(nixl_config, self._side_channel)
(VllmWorker rank=0 pid=36042) ERROR 05-15 09:31:29 [multiproc_executor.py:435]   File "/local/lib/python3.10/site-packages/lmcache/experimental/storage_backend/connector/nixl_connector_v2.py", line 190, in __init__
(VllmWorker rank=0 pid=36042) ERROR 05-15 09:31:29 [multiproc_executor.py:435]     self._reg_descs = self._agent.register_memory(self._transfer_buffers)
(VllmWorker rank=0 pid=36042) ERROR 05-15 09:31:29 [multiproc_executor.py:435]   File "/.local/lib/python3.10/site-packages/nixl/_api.py", line 265, in register_memory
(VllmWorker rank=0 pid=36042) ERROR 05-15 09:31:29 [multiproc_executor.py:435]     self.agent.registerMem(reg_descs, handle_list)
(VllmWorker rank=0 pid=36042) ERROR 05-15 09:31:29 [multiproc_executor.py:435] nixl._bindings.nixlBackendError: NIXL_ERR_BACKEND

[tensorflowt]

It looks like ucx is not installed or configured correctly.

[1747215696.528332] [b3ab5fdd703b:1615 :0] ucp_context.c:1268 UCX WARN transports 'cuda_ipc','cuda_copy' are not available, please use one or more of: mm, posix, self, shm, sm, sysv, tcp

I ran into the same issue. And walkarounded by reinstalling ucx and nixl from source.

You can first test with lmcache nixl test.

Can you provide a mirror? I have configured the environment but still can't run it.

[Unprincess17]

@tensorflowt Please try to import this: https://drive.google.com/file/d/19BVc7RLSCt0Z-IStwSdN3p47bxY5fsZs/view?usp=sharing. The vllm and other repos are inside the /root.
Import the docker image

sudo docker import lmcache.tar.gz vllm:lmcache

I start the container with

sudo docker run -dit --runtime nvidia --gpus all --security-opt=label=disabled \
  -v /sda/cache/modelscope:/root/.cache/modelscope \
  --device=/dev/infiniband:/dev/infiniband \
  --volume=/sys/class/net:/sys/class/net \
  --volume=/sys/devices:/sys/devices \
  --volume=/sys/bus/pci:/sys/bus/pci \
  --cap-add=NET_RAW \
  --network=host \
 --privileged --cap-add CAP_SYS_PTRACE --shm-size="8g" vllm:lmcache

You might need to recompile and reinstall the /root/ucx-1.18.0, in case the cuda or nvidia driver versions are different.

[tensorflowt]

@tensorflowt Please try to import this: https://drive.google.com/file/d/19BVc7RLSCt0Z-IStwSdN3p47bxY5fsZs/view?usp=sharing. The vllm and other repos are inside the /root. Import the docker image

sudo docker import lmcache.tar.gz vllm:lmcache

I start the container with

sudo docker run -dit --runtime nvidia --gpus all --security-opt=label=disabled \
  -v /sda/cache/modelscope:/root/.cache/modelscope \
  --device=/dev/infiniband:/dev/infiniband \
  --volume=/sys/class/net:/sys/class/net \
  --volume=/sys/devices:/sys/devices \
  --volume=/sys/bus/pci:/sys/bus/pci \
  --cap-add=NET_RAW \
  --network=host \
 --privileged --cap-add CAP_SYS_PTRACE --shm-size="8g" vllm:lmcache

You might need to recompile and reinstall the /root/ucx-1.18.0, in case the cuda or nvidia driver versions are different.

I want to confirm whether the command to enter the container is to run vllm serve directly? Because I can't see it here, I'm not sure whether I just need to put the model file in the /root/.cache/modelscope path?

In addition, how can I enter the container directly without performing any operations? I just want to confirm the environment configuration, but I found that the entry failed and bash was missing.

[Unprincess17]

I want to confirm whether the command to enter the container is to run vllm serve directly? Because I can't see it here, I'm not sure whether I just need to put the model file in the /root/.cache/modelscope path?

I enter the container by docker exec -it name_of_container bash. It is build up on ac2-registry.cn-hangzhou.cr.aliyuncs.com/ac2/cuda:12.6.0-devel-cudnn9-ubuntu22.04.

In addition, how can I enter the container directly without performing any operations? I just want to confirm the environment configuration, but I found that the entry failed and bash was missing.

Sorry for misleading you. You are right, I didn't check the exported tarfile.
This is a quick check:

shufan@gpusvr01:/nfs/home/shufan$ sudo sha256sum lmcache.tar                                   
2b7d8c4efd597e4e83eff8bb518324320ecec794a055413078ca8862901bef55  lmcache.tar  
shufan@gpusvr01:/nfs/home/shufan$ sudo docker import lmcache.tar lmcache:test                                                                                                
sha256:74322ade2642715a6db19471eb1d5507cc77b891f2d42065fa402fd8750370bb
shufan@gpusvr01:/nfs/home/shufan$ sudo docker run -dit --runtime nvidia --gpus all --security-opt=label=disabled   -v /sda/cache /modelscope:/root/.cache/modelscope   --device=/dev/infiniband:/dev/infiniband   --volume=/sys/class/net:/sys/class/net   --volume=/sys/devices:/sys/devices   --volume=/sys/bus/pci:/sys/bus/pci   --cap-add=NET_RAW   --network=host  --privileged --cap-add CAP_SYS_PTRACE --shm-size="8g" lmcache:test
docker: Error response from daemon: no command specified.
See 'docker run --help'.
shufan@gpusvr01:/nfs/home/shufan$ sudo docker run -dit --runtime nvidia --gpus all --security-opt=label=disabled   -v /sda/cache
/modelscope:/root/.cache/modelscope   --device=/dev/infiniband:/dev/infiniband   --volume=/sys/class/net:/sys/class/net   --volu
me=/sys/devices:/sys/devices   --volume=/sys/bus/pci:/sys/bus/pci   --cap-add=NET_RAW   --network=host  --privileged --cap-add C
AP_SYS_PTRACE --shm-size="8g" lmcache:test bash
adeda5e45e5a282e99eb577795dc32121dc8ad49d763eeba5517bfc3894d69ae
shufan@gpusvr01:/nfs/home/shufan$ sudo docker exec -it aded bash 
root@gpusvr01:/# ls ~/
LMCache  Mooncake  nixl  ucx-1.18.0  vllm
root@gpusvr01:/# cd ~/LMCache/tests/disagg/
root@gpusvr01:~/LMCache/tests/disagg# UCX_TLS=cuda_ipc,cuda_copy,tcp CUDA_VISIBLE_DEVICES=0 python3 test_nixl_channel_v2.py --role receiver --num-objs 500 --batch-size 30 --simulate-workload --num-expected-senders 1 --port 5556                             

On another terminal, I ran UCX_TLS=cuda_ipc,cuda_copy,tcp CUDA_VISIBLE_DEVICES=0 python3 test_nixl_channel_v2.py --role sender --num-objs 500 --batch-size 30 --simulate-workload --num-expected-senders 1 --port 5556

Below are the output of receiver:

[2025-05-20 11:22:18,398] LMCache INFO: Generated 500 objects with total size 16000.00 MB (test_nixl_channel_v2.py:347:__main__)
[2025-05-20 11:22:18,401] LMCache INFO: Waiting to receive data... (test_nixl_channel_v2.py:201:__main__)                       
[2025-05-20 11:22:18,401] LMCache INFO: Received 0/500 tensors so far... (test_nixl_channel_v2.py:213:__main__)                 
[2025-05-20 11:22:22,272] LMCache INFO: Initializing NixlPipe for sender: sender-21305bdc49d245bebd7d1caed3190e3a (nixl_connecto
r_v2.py:634:lmcache.experimental.storage_backend.connector.nixl_connector_v2)
[2025-05-20 11:22:22,480] LMCache INFO: Received 0/500 tensors so far... (test_nixl_channel_v2.py:213:__main__)
Backend UCX was instantiated
Initialized NIXL agent: NixlRole.RECEIVER
[2025-05-20 11:22:22,489] LMCache INFO: Sent local transfer descriptors to sender (nixl_connector_v2.py:244:lmcache.experimental
.storage_backend.connector.nixl_connector_v2)
[2025-05-20 11:22:22,489] LMCache INFO: NixlPipe initialized successfully for sender: sender-21305bdc49d245bebd7d1caed3190e3a (n
ixl_connector_v2.py:645:lmcache.experimental.storage_backend.connector.nixl_connector_v2)
[2025-05-20 11:22:22,490] LMCache INFO: New sender connected with ID: sender-21305bdc49d245bebd7d1caed3190e3a (nixl_connector_v2
.py:716:lmcache.experimental.storage_backend.connector.nixl_connector_v2)
[2025-05-20 11:22:22,506] LMCache INFO: Observer received 30 keys and 30 objects (test_nixl_channel_v2.py:66:__main__)
...
[2025-05-20 11:22:23,398] LMCache INFO: Observer received 20 keys and 20 objects (test_nixl_channel_v2.py:66:__main__)
[2025-05-20 11:22:23,481] LMCache INFO: Received all 500 unique keys and 500 total tensors (test_nixl_channel_v2.py:222:__main__
)
[2025-05-20 11:22:23,855] LMCache INFO: Received 500 unique keys and 500 total tensors (test_nixl_channel_v2.py:98:__main__)
[2025-05-20 11:22:26,411] LMCache INFO: Closing pipe for sender: sender-21305bdc49d245bebd7d1caed3190e3a (nixl_connector_v2.py:7
58:lmcache.experimental.storage_backend.connector.nixl_connector_v2)
[2025-05-20 11:22:26,413] LMCache INFO: Test completed (test_nixl_channel_v2.py:385:__main__)

[tensorflowt]

I want to confirm whether the command to enter the container is to run vllm serve directly? Because I can't see it here, I'm not sure whether I just need to put the model file in the /root/.cache/modelscope path?

I enter the container by docker exec -it name_of_container bash. It is build up on ac2-registry.cn-hangzhou.cr.aliyuncs.com/ac2/cuda:12.6.0-devel-cudnn9-ubuntu22.04.

In addition, how can I enter the container directly without performing any operations? I just want to confirm the environment configuration, but I found that the entry failed and bash was missing.

Sorry for misleading you. You are right, I didn't check the exported tarfile. This is a quick check:

shufan@gpusvr01:/nfs/home/shufan$ sudo sha256sum lmcache.tar                                   
2b7d8c4efd597e4e83eff8bb518324320ecec794a055413078ca8862901bef55  lmcache.tar  
shufan@gpusvr01:/nfs/home/shufan$ sudo docker import lmcache.tar lmcache:test                                                                                                
sha256:74322ade2642715a6db19471eb1d5507cc77b891f2d42065fa402fd8750370bb
shufan@gpusvr01:/nfs/home/shufan$ sudo docker run -dit --runtime nvidia --gpus all --security-opt=label=disabled   -v /sda/cache /modelscope:/root/.cache/modelscope   --device=/dev/infiniband:/dev/infiniband   --volume=/sys/class/net:/sys/class/net   --volume=/sys/devices:/sys/devices   --volume=/sys/bus/pci:/sys/bus/pci   --cap-add=NET_RAW   --network=host  --privileged --cap-add CAP_SYS_PTRACE --shm-size="8g" lmcache:test
docker: Error response from daemon: no command specified.
See 'docker run --help'.
shufan@gpusvr01:/nfs/home/shufan$ sudo docker run -dit --runtime nvidia --gpus all --security-opt=label=disabled   -v /sda/cache
/modelscope:/root/.cache/modelscope   --device=/dev/infiniband:/dev/infiniband   --volume=/sys/class/net:/sys/class/net   --volu
me=/sys/devices:/sys/devices   --volume=/sys/bus/pci:/sys/bus/pci   --cap-add=NET_RAW   --network=host  --privileged --cap-add C
AP_SYS_PTRACE --shm-size="8g" lmcache:test bash
adeda5e45e5a282e99eb577795dc32121dc8ad49d763eeba5517bfc3894d69ae
shufan@gpusvr01:/nfs/home/shufan$ sudo docker exec -it aded bash 
root@gpusvr01:/# ls ~/
LMCache  Mooncake  nixl  ucx-1.18.0  vllm
root@gpusvr01:/# cd ~/LMCache/tests/disagg/
root@gpusvr01:~/LMCache/tests/disagg# UCX_TLS=cuda_ipc,cuda_copy,tcp CUDA_VISIBLE_DEVICES=0 python3 test_nixl_channel_v2.py --role receiver --num-objs 500 --batch-size 30 --simulate-workload --num-expected-senders 1 --port 5556                             

On another terminal, I ran UCX_TLS=cuda_ipc,cuda_copy,tcp CUDA_VISIBLE_DEVICES=0 python3 test_nixl_channel_v2.py --role sender --num-objs 500 --batch-size 30 --simulate-workload --num-expected-senders 1 --port 5556

Below are the output of receiver:

[2025-05-20 11:22:18,398] LMCache INFO: Generated 500 objects with total size 16000.00 MB (test_nixl_channel_v2.py:347:__main__)
[2025-05-20 11:22:18,401] LMCache INFO: Waiting to receive data... (test_nixl_channel_v2.py:201:__main__)                       
[2025-05-20 11:22:18,401] LMCache INFO: Received 0/500 tensors so far... (test_nixl_channel_v2.py:213:__main__)                 
[2025-05-20 11:22:22,272] LMCache INFO: Initializing NixlPipe for sender: sender-21305bdc49d245bebd7d1caed3190e3a (nixl_connecto
r_v2.py:634:lmcache.experimental.storage_backend.connector.nixl_connector_v2)
[2025-05-20 11:22:22,480] LMCache INFO: Received 0/500 tensors so far... (test_nixl_channel_v2.py:213:__main__)
Backend UCX was instantiated
Initialized NIXL agent: NixlRole.RECEIVER
[2025-05-20 11:22:22,489] LMCache INFO: Sent local transfer descriptors to sender (nixl_connector_v2.py:244:lmcache.experimental
.storage_backend.connector.nixl_connector_v2)
[2025-05-20 11:22:22,489] LMCache INFO: NixlPipe initialized successfully for sender: sender-21305bdc49d245bebd7d1caed3190e3a (n
ixl_connector_v2.py:645:lmcache.experimental.storage_backend.connector.nixl_connector_v2)
[2025-05-20 11:22:22,490] LMCache INFO: New sender connected with ID: sender-21305bdc49d245bebd7d1caed3190e3a (nixl_connector_v2
.py:716:lmcache.experimental.storage_backend.connector.nixl_connector_v2)
[2025-05-20 11:22:22,506] LMCache INFO: Observer received 30 keys and 30 objects (test_nixl_channel_v2.py:66:__main__)
...
[2025-05-20 11:22:23,398] LMCache INFO: Observer received 20 keys and 20 objects (test_nixl_channel_v2.py:66:__main__)
[2025-05-20 11:22:23,481] LMCache INFO: Received all 500 unique keys and 500 total tensors (test_nixl_channel_v2.py:222:__main__
)
[2025-05-20 11:22:23,855] LMCache INFO: Received 500 unique keys and 500 total tensors (test_nixl_channel_v2.py:98:__main__)
[2025-05-20 11:22:26,411] LMCache INFO: Closing pipe for sender: sender-21305bdc49d245bebd7d1caed3190e3a (nixl_connector_v2.py:7
58:lmcache.experimental.storage_backend.connector.nixl_connector_v2)
[2025-05-20 11:22:26,413] LMCache INFO: Test completed (test_nixl_channel_v2.py:385:__main__)

Thank you very much for your detailed reply. I have solved the problem of lmcache environment configuration and can run the lmcache test command normally, but I have encountered another problem. It is found that the torch version of lmcache main branch is 0.7.0, which is incompatible with the corresponding vllm version 0.8.5 (corresponding to torch version 0.6.0). So I want to confirm, what is the vllm version you are using?

[Unprincess17]

I am using vllm version 0.8.5, and at this commit f25e0d1.

[tensorflowt]

I am using vllm version 0.8.5, and at this commit f25e0d1.

This PR has been merged into the main branch. I see that the 0.8.5 branch has not merged this PR, but I failed to compile the main branch locally. Can you provide the pip list information? I want to confirm the version information of vllm and lmcache.

[Unprincess17]

# pip list
Package                                  Version                             Editable project location
---------------------------------------- ----------------------------------- -------------------------
aiofiles                                 24.1.0
aiohappyeyeballs                         2.6.1
aiohttp                                  3.11.18
aiosignal                                1.3.2
airportsdata                             20250224
annotated-types                          0.7.0
anyio                                    4.9.0
appdirs                                  1.4.4
astor                                    0.8.1
async-timeout                            5.0.1
attrs                                    25.3.0
beautifulsoup4                           4.10.0
beniget                                  0.4.1
blake3                                   1.0.4
blinker                                  1.9.0
bottle                                   0.12.19
Brotli                                   1.0.9
cachetools                               5.5.2
certifi                                  2025.4.26
chardet                                  4.0.0
charset-normalizer                       3.4.2
click                                    8.1.8
cloudpickle                              3.1.1
compressed-tensors                       0.9.4
cryptography                             3.4.8
cupy-cuda12x                             13.4.1
cycler                                   0.11.0
datasets                                 3.5.1
dbus-python                              1.2.18
decorator                                4.4.2
defusedxml                               0.7.1
Deprecated                               1.2.18
depyf                                    0.18.0
dill                                     0.3.8
diskcache                                5.6.3
distro                                   1.9.0
distro-info                              1.1+ubuntu0.2
dnspython                                2.7.0
docker                                   5.0.3
einops                                   0.8.1
email_validator                          2.2.0
exceptiongroup                           1.2.2
fastapi                                  0.115.12
fastapi-cli                              0.0.7
fastrlock                                0.8.3
filelock                                 3.18.0
Flask                                    3.1.0
fonttools                                4.29.1
frozenlist                               1.6.0
fs                                       2.4.12
fsspec                                   2025.3.0
future                                   0.18.2
gast                                     0.5.2
gguf                                     0.16.3
Glances                                  3.2.4.2
googleapis-common-protos                 1.70.0
grpcio                                   1.71.0
h11                                      0.16.0
h2                                       4.2.0
hpack                                    4.1.0
html5lib                                 1.1
httpcore                                 1.0.9
httplib2                                 0.20.2
httptools                                0.6.4
httpx                                    0.28.1
huggingface-hub                          0.30.2
Hypercorn                                0.17.3
hyperframe                               6.1.0
idna                                     3.10
importlib_metadata                       8.0.0
infinistore                              0.2.35
influxdb                                 5.3.1
iniconfig                                2.1.0
interegular                              0.3.3
itsdangerous                             2.2.0
jeepney                                  0.7.1
Jinja2                                   3.1.6
jiter                                    0.9.0
jsonschema                               4.23.0
jsonschema-specifications                2025.4.1
keyring                                  23.5.0
kiwisolver                               1.3.2
lark                                     1.2.2
launchpadlib                             1.10.16
lazr.restfulclient                       0.14.4
lazr.uri                                 1.0.6
llguidance                               0.7.19
llvmlite                                 0.44.0
lm-format-enforcer                       0.10.11
lmcache                                  0.2.2.dev58                         /root/LMCache
lxml                                     4.8.0
lz4                                      3.1.3+dfsg
markdown-it-py                           3.0.0
MarkupSafe                               3.0.2
matplotlib                               3.5.1
mdurl                                    0.1.2
meson                                    1.8.0
mistral_common                           1.5.4
modelscope                               1.25.0
more-itertools                           8.10.0
mpmath                                   1.3.0
msgpack                                  1.1.0
msgspec                                  0.19.0
multidict                                6.4.3
multiprocess                             0.70.16
nest-asyncio                             1.6.0
netifaces                                0.11.0
networkx                                 3.4.2
ninja                                    1.11.1.4
nixl                                     0.2.0
numba                                    0.61.2
numpy                                    2.2.5
nvidia-cublas-cu12                       12.6.4.1
nvidia-cuda-cupti-cu12                   12.6.80
nvidia-cuda-nvrtc-cu12                   12.6.77
nvidia-cuda-runtime-cu12                 12.6.77
nvidia-cudnn-cu12                        9.5.1.17
nvidia-cufft-cu12                        11.3.0.4
nvidia-cufile-cu12                       1.11.1.6
nvidia-curand-cu12                       10.3.7.77
nvidia-cusolver-cu12                     11.7.1.2
nvidia-cusparse-cu12                     12.5.4.2
nvidia-cusparselt-cu12                   0.6.3
nvidia-nccl-cu12                         2.26.2
nvidia-nvjitlink-cu12                    12.6.85
nvidia-nvtx-cu12                         12.6.77
nvtx                                     0.2.11
oauthlib                                 3.2.0
olefile                                  0.46
openai                                   1.77.0
opencv-python-headless                   4.11.0.86
opentelemetry-api                        1.26.0
opentelemetry-exporter-otlp              1.26.0
opentelemetry-exporter-otlp-proto-common 1.26.0
opentelemetry-exporter-otlp-proto-grpc   1.26.0
opentelemetry-exporter-otlp-proto-http   1.26.0
opentelemetry-proto                      1.26.0
opentelemetry-sdk                        1.26.0
opentelemetry-semantic-conventions       0.47b0
opentelemetry-semantic-conventions-ai    0.4.6
outlines                                 0.1.11
outlines_core                            0.1.26
packaging                                25.0
pandas                                   2.2.3
partial-json-parser                      0.2.1.1.post5
pillow                                   11.2.1
pip                                      22.0.2
pluggy                                   1.5.0
ply                                      3.11
priority                                 2.0.0
prometheus_client                        0.21.1
prometheus-fastapi-instrumentator        7.1.0
propcache                                0.3.1
protobuf                                 4.25.7
psutil                                   7.0.0
py-cpuinfo                               9.0.0
pyarrow                                  20.0.0
pyasn1                                   0.4.8
pybind11                                 2.14.0.dev1
pybind11_global                          2.14.0.dev1
pycountry                                24.6.1
pycryptodomex                            3.11.0
pydantic                                 2.11.4
pydantic_core                            2.33.2
Pygments                                 2.19.1
PyGObject                                3.42.1
PyJWT                                    2.3.0
pyparsing                                2.4.7
pysmi                                    0.3.2
pysnmp                                   4.4.12
pystache                                 0.6.0
pytest                                   8.3.5
python-apt                               2.4.0+ubuntu4
python-dateutil                          2.9.0.post0
python-dotenv                            1.1.0
python-json-logger                       3.3.0
python-multipart                         0.0.20
pythran                                  0.10.0
pytz                                     2025.2
PyYAML                                   6.0.2
pyzmq                                    26.4.0
Quart                                    0.20.0
ray                                      2.45.0
redis                                    6.0.0
referencing                              0.36.2
regex                                    2024.11.6
requests                                 2.32.3
rich                                     14.0.0
rich-toolkit                             0.14.5
rpds-py                                  0.24.0
safetensors                              0.5.3
scipy                                    1.15.2
SecretStorage                            3.3.1
sentencepiece                            0.2.0
setuptools                               79.0.1
setuptools-scm                           8.3.1
shellingham                              1.5.4
six                                      1.17.0
sniffio                                  1.3.1
sortedcontainers                         2.4.0
soupsieve                                2.3.1
starlette                                0.46.2
sympy                                    1.14.0
taskgroup                                0.2.2
tiktoken                                 0.9.0
tokenizers                               0.21.1
tomli                                    2.2.1
torch                                    2.7.0
torchaudio                               2.7.0
torchvision                              0.22.0
tqdm                                     4.67.1
transformers                             4.51.3
triton                                   3.3.0
typer                                    0.15.3
typing_extensions                        4.13.2
typing-inspection                        0.4.0
tzdata                                   2025.2
ufoLib2                                  0.13.1
unattended-upgrades                      0.1
unicodedata2                             14.0.0
urllib3                                  2.4.0
uvicorn                                  0.34.2
uvloop                                   0.21.0
vllm                                     0.8.5.dev475+gf9bc5a069.precompiled /root/vllm
wadllib                                  1.3.6
watchfiles                               1.0.5
webencodings                             0.5.1
websocket-client                         1.2.3
websockets                               15.0.1
Werkzeug                                 3.1.3
wheel                                    0.37.1
wrapt                                    1.17.2
wsproto                                  1.2.0
xformers                                 0.0.30
xgrammar                                 0.1.18
xxhash                                   3.5.0
yarl                                     1.20.0
zipp                                     3.21.0

[tensorflowt]

# pip list
Package                                  Version                             Editable project location
---------------------------------------- ----------------------------------- -------------------------
aiofiles                                 24.1.0
aiohappyeyeballs                         2.6.1
aiohttp                                  3.11.18
aiosignal                                1.3.2
airportsdata                             20250224
annotated-types                          0.7.0
anyio                                    4.9.0
appdirs                                  1.4.4
astor                                    0.8.1
async-timeout                            5.0.1
attrs                                    25.3.0
beautifulsoup4                           4.10.0
beniget                                  0.4.1
blake3                                   1.0.4
blinker                                  1.9.0
bottle                                   0.12.19
Brotli                                   1.0.9
cachetools                               5.5.2
certifi                                  2025.4.26
chardet                                  4.0.0
charset-normalizer                       3.4.2
click                                    8.1.8
cloudpickle                              3.1.1
compressed-tensors                       0.9.4
cryptography                             3.4.8
cupy-cuda12x                             13.4.1
cycler                                   0.11.0
datasets                                 3.5.1
dbus-python                              1.2.18
decorator                                4.4.2
defusedxml                               0.7.1
Deprecated                               1.2.18
depyf                                    0.18.0
dill                                     0.3.8
diskcache                                5.6.3
distro                                   1.9.0
distro-info                              1.1+ubuntu0.2
dnspython                                2.7.0
docker                                   5.0.3
einops                                   0.8.1
email_validator                          2.2.0
exceptiongroup                           1.2.2
fastapi                                  0.115.12
fastapi-cli                              0.0.7
fastrlock                                0.8.3
filelock                                 3.18.0
Flask                                    3.1.0
fonttools                                4.29.1
frozenlist                               1.6.0
fs                                       2.4.12
fsspec                                   2025.3.0
future                                   0.18.2
gast                                     0.5.2
gguf                                     0.16.3
Glances                                  3.2.4.2
googleapis-common-protos                 1.70.0
grpcio                                   1.71.0
h11                                      0.16.0
h2                                       4.2.0
hpack                                    4.1.0
html5lib                                 1.1
httpcore                                 1.0.9
httplib2                                 0.20.2
httptools                                0.6.4
httpx                                    0.28.1
huggingface-hub                          0.30.2
Hypercorn                                0.17.3
hyperframe                               6.1.0
idna                                     3.10
importlib_metadata                       8.0.0
infinistore                              0.2.35
influxdb                                 5.3.1
iniconfig                                2.1.0
interegular                              0.3.3
itsdangerous                             2.2.0
jeepney                                  0.7.1
Jinja2                                   3.1.6
jiter                                    0.9.0
jsonschema                               4.23.0
jsonschema-specifications                2025.4.1
keyring                                  23.5.0
kiwisolver                               1.3.2
lark                                     1.2.2
launchpadlib                             1.10.16
lazr.restfulclient                       0.14.4
lazr.uri                                 1.0.6
llguidance                               0.7.19
llvmlite                                 0.44.0
lm-format-enforcer                       0.10.11
lmcache                                  0.2.2.dev58                         /root/LMCache
lxml                                     4.8.0
lz4                                      3.1.3+dfsg
markdown-it-py                           3.0.0
MarkupSafe                               3.0.2
matplotlib                               3.5.1
mdurl                                    0.1.2
meson                                    1.8.0
mistral_common                           1.5.4
modelscope                               1.25.0
more-itertools                           8.10.0
mpmath                                   1.3.0
msgpack                                  1.1.0
msgspec                                  0.19.0
multidict                                6.4.3
multiprocess                             0.70.16
nest-asyncio                             1.6.0
netifaces                                0.11.0
networkx                                 3.4.2
ninja                                    1.11.1.4
nixl                                     0.2.0
numba                                    0.61.2
numpy                                    2.2.5
nvidia-cublas-cu12                       12.6.4.1
nvidia-cuda-cupti-cu12                   12.6.80
nvidia-cuda-nvrtc-cu12                   12.6.77
nvidia-cuda-runtime-cu12                 12.6.77
nvidia-cudnn-cu12                        9.5.1.17
nvidia-cufft-cu12                        11.3.0.4
nvidia-cufile-cu12                       1.11.1.6
nvidia-curand-cu12                       10.3.7.77
nvidia-cusolver-cu12                     11.7.1.2
nvidia-cusparse-cu12                     12.5.4.2
nvidia-cusparselt-cu12                   0.6.3
nvidia-nccl-cu12                         2.26.2
nvidia-nvjitlink-cu12                    12.6.85
nvidia-nvtx-cu12                         12.6.77
nvtx                                     0.2.11
oauthlib                                 3.2.0
olefile                                  0.46
openai                                   1.77.0
opencv-python-headless                   4.11.0.86
opentelemetry-api                        1.26.0
opentelemetry-exporter-otlp              1.26.0
opentelemetry-exporter-otlp-proto-common 1.26.0
opentelemetry-exporter-otlp-proto-grpc   1.26.0
opentelemetry-exporter-otlp-proto-http   1.26.0
opentelemetry-proto                      1.26.0
opentelemetry-sdk                        1.26.0
opentelemetry-semantic-conventions       0.47b0
opentelemetry-semantic-conventions-ai    0.4.6
outlines                                 0.1.11
outlines_core                            0.1.26
packaging                                25.0
pandas                                   2.2.3
partial-json-parser                      0.2.1.1.post5
pillow                                   11.2.1
pip                                      22.0.2
pluggy                                   1.5.0
ply                                      3.11
priority                                 2.0.0
prometheus_client                        0.21.1
prometheus-fastapi-instrumentator        7.1.0
propcache                                0.3.1
protobuf                                 4.25.7
psutil                                   7.0.0
py-cpuinfo                               9.0.0
pyarrow                                  20.0.0
pyasn1                                   0.4.8
pybind11                                 2.14.0.dev1
pybind11_global                          2.14.0.dev1
pycountry                                24.6.1
pycryptodomex                            3.11.0
pydantic                                 2.11.4
pydantic_core                            2.33.2
Pygments                                 2.19.1
PyGObject                                3.42.1
PyJWT                                    2.3.0
pyparsing                                2.4.7
pysmi                                    0.3.2
pysnmp                                   4.4.12
pystache                                 0.6.0
pytest                                   8.3.5
python-apt                               2.4.0+ubuntu4
python-dateutil                          2.9.0.post0
python-dotenv                            1.1.0
python-json-logger                       3.3.0
python-multipart                         0.0.20
pythran                                  0.10.0
pytz                                     2025.2
PyYAML                                   6.0.2
pyzmq                                    26.4.0
Quart                                    0.20.0
ray                                      2.45.0
redis                                    6.0.0
referencing                              0.36.2
regex                                    2024.11.6
requests                                 2.32.3
rich                                     14.0.0
rich-toolkit                             0.14.5
rpds-py                                  0.24.0
safetensors                              0.5.3
scipy                                    1.15.2
SecretStorage                            3.3.1
sentencepiece                            0.2.0
setuptools                               79.0.1
setuptools-scm                           8.3.1
shellingham                              1.5.4
six                                      1.17.0
sniffio                                  1.3.1
sortedcontainers                         2.4.0
soupsieve                                2.3.1
starlette                                0.46.2
sympy                                    1.14.0
taskgroup                                0.2.2
tiktoken                                 0.9.0
tokenizers                               0.21.1
tomli                                    2.2.1
torch                                    2.7.0
torchaudio                               2.7.0
torchvision                              0.22.0
tqdm                                     4.67.1
transformers                             4.51.3
triton                                   3.3.0
typer                                    0.15.3
typing_extensions                        4.13.2
typing-inspection                        0.4.0
tzdata                                   2025.2
ufoLib2                                  0.13.1
unattended-upgrades                      0.1
unicodedata2                             14.0.0
urllib3                                  2.4.0
uvicorn                                  0.34.2
uvloop                                   0.21.0
vllm                                     0.8.5.dev475+gf9bc5a069.precompiled /root/vllm
wadllib                                  1.3.6
watchfiles                               1.0.5
webencodings                             0.5.1
websocket-client                         1.2.3
websockets                               15.0.1
Werkzeug                                 3.1.3
wheel                                    0.37.1
wrapt                                    1.17.2
wsproto                                  1.2.0
xformers                                 0.0.30
xgrammar                                 0.1.18
xxhash                                   3.5.0
yarl                                     1.20.0
zipp                                     3.21.0

I tried to compile the source code according to the vllm version you provided, but it failed. I want to use the image you provided, but I found that the lmcache.tar.gz file (md5:cdbd0bd5ea1361c75de6218b30b4f077) you provided before still cannot enter the container successfully. The lmcache.tar file is displayed successfully. Can you provide the download address of this file? Thanks

[Unprincess17]

It is strange :(
The lmcache.tar.gz looks good:

shufan@gpusvr01:/nfs/home/shufan/tmp$ md5sum lmcache.tar.gz
cdbd0bd5ea1361c75de6218b30b4f077  lmcache.tar.gz
shufan@gpusvr01:/nfs/home/shufan/tmp$ sudo md5sum lmcache.tar
5a10b32fd04af8f8281793c6ee1ba2bb  lmcache.tar

This lmcache.tar is from tar -zxvf lmcache.tar.gzz. And it's ~23G, beyond google drive's limit.

[tensorflowt]

It is strange :( The lmcache.tar.gz looks good:

shufan@gpusvr01:/nfs/home/shufan/tmp$ md5sum lmcache.tar.gz
cdbd0bd5ea1361c75de6218b30b4f077  lmcache.tar.gz
shufan@gpusvr01:/nfs/home/shufan/tmp$ sudo md5sum lmcache.tar
5a10b32fd04af8f8281793c6ee1ba2bb  lmcache.tar

This lmcache.tar is from tar -zxvf lmcache.tar.gzz. And it's ~23G, beyond google drive's limit.

Following your process, I can successfully enter the container, but my cuda is 12.4. I tested lmcache and it failed. It should be recompiled.
In addition, I checked the lmcache community and found that there is indeed an incompatibility issue with the torch version.
https://github.com/LMCache/LMCache/pull/583/files#diff-50c86b7ed8ac2cf95bd48334961bf0530cdc77b5a56f852c5c61b89d735fd711
This PR says that it has been fixed, but I still failed to test the lmcache example.