INFO 05-12 13:45:42 [__init__.py:239] Automatically detected platform cuda.
INFO 05-12 13:45:43 [api_server.py:1034] vLLM API server version 0.8.3rc2.dev179+gdc1b4a6f1.d20250423
INFO 05-12 13:45:43 [api_server.py:1035] args: Namespace(subparser='serve', model_tag='/home/models/DeepSeek-V2-Lite', config='', host=None, port=8000, uvicorn_log_level='info', disable_uvicorn_access_log=False, allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, enable_ssl_refresh=False, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='/home/models/DeepSeek-V2-Lite', task='auto', tokenizer=None, hf_config_path=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=True, allowed_local_media_path=None, load_format='auto', download_dir=None, model_loader_extra_config=None, use_tqdm_on_load=True, config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', max_model_len=512, guided_decoding_backend='auto', logits_processor_pattern=None, model_impl='auto', distributed_executor_backend='ray', pipeline_parallel_size=1, tensor_parallel_size=8, data_parallel_size=2, enable_expert_parallel=True, max_parallel_loading_workers=None, ray_workers_use_nsight=False, disable_custom_all_reduce=False, block_size=None, enable_prefix_caching=None, prefix_caching_hash_algo='builtin', disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=None, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.97, num_gpu_blocks_override=None, max_num_batched_tokens=512, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, max_num_seqs=1, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_token=None, hf_overrides=None, enforce_eager=True, max_seq_len_to_capture=8192, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', worker_extension_cls='', generation_config='auto', override_generation_config=None, enable_sleep_mode=False, calculate_kv_scales=False, additional_config=None, enable_reasoning=False, reasoning_parser=None, disable_cascade_attn=False, disable_chunked_mm_input=False, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, enable_server_load_tracking=False, dispatch_function=<function ServeSubcommand.cmd at 0x7fedf08062a0>)
INFO 05-12 13:45:43 [config.py:209] Replacing legacy 'type' key with 'rope_type'
INFO 05-12 13:45:51 [config.py:689] This model supports multiple tasks: {'classify', 'score', 'generate', 'reward', 'embed'}. Defaulting to 'generate'.
INFO 05-12 13:45:51 [config.py:1901] Chunked prefill is enabled with max_num_batched_tokens=512.
WARNING 05-12 13:45:51 [cuda.py:97] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used
INFO 05-12 13:45:56 [__init__.py:239] Automatically detected platform cuda.
INFO 05-12 13:45:56 [__init__.py:239] Automatically detected platform cuda.
(EngineCore_0 pid=2325124) INFO 05-12 13:45:58 [core.py:61] Initializing a V1 LLM engine (v0.8.3rc2.dev179+gdc1b4a6f1.d20250423) with config: model='/home/models/DeepSeek-V2-Lite', speculative_config=None, tokenizer='/home/models/DeepSeek-V2-Lite', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=512, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=8, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='auto', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=None, served_model_name=/home/models/DeepSeek-V2-Lite, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=False, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[],"max_capture_size":0}
(EngineCore_1 pid=2325125) INFO 05-12 13:45:58 [core.py:61] Initializing a V1 LLM engine (v0.8.3rc2.dev179+gdc1b4a6f1.d20250423) with config: model='/home/models/DeepSeek-V2-Lite', speculative_config=None, tokenizer='/home/models/DeepSeek-V2-Lite', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=512, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=8, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='auto', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=None, served_model_name=/home/models/DeepSeek-V2-Lite, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=False, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[],"max_capture_size":0}
(EngineCore_1 pid=2325125) 2025-05-12 13:45:59,016 INFO worker.py:1694 -- Connecting to existing Ray cluster at address: 10.30.1.16:6379...
(EngineCore_0 pid=2325124) 2025-05-12 13:45:59,016 INFO worker.py:1694 -- Connecting to existing Ray cluster at address: 10.30.1.16:6379...
(EngineCore_1 pid=2325125) 2025-05-12 13:45:59,028 INFO worker.py:1888 -- Connected to Ray cluster.
(EngineCore_0 pid=2325124) 2025-05-12 13:45:59,029 INFO worker.py:1888 -- Connected to Ray cluster.
(EngineCore_0 pid=2325124) INFO 05-12 13:45:59 [ray_utils.py:335] No current placement group found. Creating a new placement group.
(EngineCore_0 pid=2325124) INFO 05-12 13:45:59 [ray_distributed_executor.py:176] use_ray_spmd_worker: True
(EngineCore_1 pid=2325125) INFO 05-12 13:46:00 [ray_utils.py:335] No current placement group found. Creating a new placement group.
(EngineCore_1 pid=2325125) ERROR 05-12 13:46:00 [core.py:387] EngineCore hit an exception: Traceback (most recent call last):
(EngineCore_1 pid=2325125) ERROR 05-12 13:46:00 [core.py:387] File "/home/wangyxbh/vllm/vllm/v1/engine/core.py", line 376, in run_engine_core
(EngineCore_1 pid=2325125) ERROR 05-12 13:46:00 [core.py:387] engine_core = DPEngineCoreProc(*args, **kwargs)
(EngineCore_1 pid=2325125) ERROR 05-12 13:46:00 [core.py:387] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_1 pid=2325125) ERROR 05-12 13:46:00 [core.py:387] File "/home/wangyxbh/vllm/vllm/v1/engine/core.py", line 564, in __init__
(EngineCore_1 pid=2325125) ERROR 05-12 13:46:00 [core.py:387] super().__init__(input_path, output_path, vllm_config, executor_class,
(EngineCore_1 pid=2325125) ERROR 05-12 13:46:00 [core.py:387] File "/home/wangyxbh/vllm/vllm/v1/engine/core.py", line 320, in __init__
(EngineCore_1 pid=2325125) ERROR 05-12 13:46:00 [core.py:387] super().__init__(vllm_config, executor_class, log_stats)
(EngineCore_1 pid=2325125) ERROR 05-12 13:46:00 [core.py:387] File "/home/wangyxbh/vllm/vllm/v1/engine/core.py", line 67, in __init__
(EngineCore_1 pid=2325125) ERROR 05-12 13:46:00 [core.py:387] self.model_executor = executor_class(vllm_config)
(EngineCore_1 pid=2325125) ERROR 05-12 13:46:00 [core.py:387] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_1 pid=2325125) ERROR 05-12 13:46:00 [core.py:387] File "/home/wangyxbh/vllm/vllm/executor/executor_base.py", line 286, in __init__
(EngineCore_1 pid=2325125) ERROR 05-12 13:46:00 [core.py:387] super().__init__(*args, **kwargs)
(EngineCore_1 pid=2325125) ERROR 05-12 13:46:00 [core.py:387] File "/home/wangyxbh/vllm/vllm/executor/executor_base.py", line 52, in __init__
(EngineCore_1 pid=2325125) ERROR 05-12 13:46:00 [core.py:387] self._init_executor()
(EngineCore_1 pid=2325125) ERROR 05-12 13:46:00 [core.py:387] File "/home/wangyxbh/vllm/vllm/executor/ray_distributed_executor.py", line 105, in _init_executor
(EngineCore_1 pid=2325125) ERROR 05-12 13:46:00 [core.py:387] initialize_ray_cluster(self.parallel_config)
(EngineCore_1 pid=2325125) ERROR 05-12 13:46:00 [core.py:387] File "/home/wangyxbh/vllm/vllm/executor/ray_utils.py", line 358, in initialize_ray_cluster
(EngineCore_1 pid=2325125) ERROR 05-12 13:46:00 [core.py:387] raise ValueError(
(EngineCore_1 pid=2325125) ERROR 05-12 13:46:00 [core.py:387] ValueError: Current node has no GPU available. current_node_resource={'bundle_group_2_dde7c82b3e1573af25e1cb38b27c02000000': 999.999, 'bundle_group_0_dde7c82b3e1573af25e1cb38b27c02000000': 999.999, 'node:10.30.1.16': 0.999, 'accelerator_type:L40S': 1.0, 'bundle_group_6_dde7c82b3e1573af25e1cb38b27c02000000': 999.999, 'bundle_group_dde7c82b3e1573af25e1cb38b27c02000000': 7999.992, 'node:10.30.1.16_group_dde7c82b3e1573af25e1cb38b27c02000000': 0.001, 'node:10.30.1.16_group_0_dde7c82b3e1573af25e1cb38b27c02000000': 0.001, 'bundle_group_5_dde7c82b3e1573af25e1cb38b27c02000000': 999.999, 'bundle_group_4_dde7c82b3e1573af25e1cb38b27c02000000': 999.999, 'node:__internal_head__': 1.0, 'CPU': 64.0, 'object_store_memory': 200000000000.0, 'bundle_group_1_dde7c82b3e1573af25e1cb38b27c02000000': 999.999, 'bundle_group_7_dde7c82b3e1573af25e1cb38b27c02000000': 999.999, 'bundle_group_3_dde7c82b3e1573af25e1cb38b27c02000000': 999.999, 'memory': 1952192348160.0}. vLLM engine cannot start without GPU. Make sure you have at least 1 GPU available in a node current_node_id='841b8f4a6a5122881cce5f9cbe246bad6af8c4675e4368e8195faa31' current_ip='10.30.1.16'.
(EngineCore_1 pid=2325125) ERROR 05-12 13:46:00 [core.py:387]
(EngineCore_1 pid=2325125) INFO 05-12 13:46:00 [ray_distributed_executor.py:127] Shutting down Ray distributed executor. If you see error log from logging.cc regarding SIGTERM received, please ignore because this is the expected termination process in Ray.
CRITICAL 05-12 13:46:00 [core_client.py:359] Got fatal signal from worker processes, shutting down. See stack trace above for root cause issue.
Killed
