[Feature]: deepseek v4 support

[liudonghua123]

🚀 The feature, motivation and pitch

The DeepSeek v4 has been released.

https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro
https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[wushiqi03]

Same question, what is the earliest DeepSeek V4 will be supported?

[aidendle94]

Please refer to:

https://vllm.ai/blog/deepseek-v4

[wushiqi03]

Please refer to:

https://vllm.ai/blog/deepseek-v4

I truly appreciate it. Thank you so much!

[linchpinlin]

vllm/vllm-openai:deepseekv4-cu129
8 * L20 46G

VLLM_LOGGING_LEVEL=DEBUG vllm serve /data/pretrained/DeepSeek-V4-Flash
--served-model-name DeepSeek-V4-Flash
--trust-remote-code
--kv-cache-dtype fp8
--block-size 256
--enable-expert-parallel
--data-parallel-size 8
--gpu-memory-utilization 0.85
--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE", "custom_ops":["all"]}'
--attention_config.use_fp4_indexer_cache=True
--tokenizer-mode deepseek_v4
--tool-call-parser deepseek_v4
--enable-auto-tool-choice
--reasoning-parser deepseek_v4

(EngineCore_DP1 pid=28757) RuntimeError: Worker failed with error 'AssertionError: auto_functionalized was not removed', please check the stack trace above for the root cause
(EngineCore_DP1 pid=28757) DEBUG 04-24 07:16:23 [distributed/device_communicators/shm_broadcast.py:168] Canceling waiting reads on SHM Buffer
terminate called without an active exception
(EngineCore_DP4 pid=28760) Process EngineCore_DP4:
(EngineCore_DP4 pid=28760) Traceback (most recent call last):
(EngineCore_DP4 pid=28760) File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP4 pid=28760) self.run()
(EngineCore_DP4 pid=28760) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore_DP4 pid=28760) self._target(*self._args, **self._kwargs)
(EngineCore_DP4 pid=28760) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1114, in run_engine_core
(EngineCore_DP4 pid=28760) raise e
(EngineCore_DP4 pid=28760) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1076, in run_engine_core
(EngineCore_DP4 pid=28760) engine_core = DPEngineCoreProc(*args, **kwargs)
(EngineCore_DP4 pid=28760) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP4 pid=28760) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1618, in init
(EngineCore_DP4 pid=28760) super().init(
(EngineCore_DP4 pid=28760) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore_DP4 pid=28760) return func(*args, **kwargs)
(EngineCore_DP4 pid=28760) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP4 pid=28760) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 850, in init
(EngineCore_DP4 pid=28760) super().init(
(EngineCore_DP4 pid=28760) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 127, in init
(EngineCore_DP4 pid=28760) kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore_DP4 pid=28760) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP4 pid=28760) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore_DP4 pid=28760) return func(*args, **kwargs)
(EngineCore_DP4 pid=28760) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP4 pid=28760) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 249, in _initialize_kv_caches
(EngineCore_DP4 pid=28760) available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore_DP4 pid=28760) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP4 pid=28760) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 142, in determine_available_memory
(EngineCore_DP4 pid=28760) return self.collective_rpc("determine_available_memory")
(EngineCore_DP4 pid=28760) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP4 pid=28760) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 412, in collective_rpc
(EngineCore_DP4 pid=28760) return future if non_block else future.result()
(EngineCore_DP4 pid=28760) ^^^^^^^^^^^^^^^
(EngineCore_DP4 pid=28760) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 89, in result
(EngineCore_DP4 pid=28760) return super().result()
(EngineCore_DP4 pid=28760) ^^^^^^^^^^^^^^^^
(EngineCore_DP4 pid=28760) File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
(EngineCore_DP4 pid=28760) return self.__get_result()
(EngineCore_DP4 pid=28760) ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP4 pid=28760) File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(EngineCore_DP4 pid=28760) raise self._exception
(EngineCore_DP4 pid=28760) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 93, in _wait_for_response
(EngineCore_DP4 pid=28760) response = self.aggregate(self.get_response())
(EngineCore_DP4 pid=28760) ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP4 pid=28760) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 399, in get_response
(EngineCore_DP4 pid=28760) raise RuntimeError(
(EngineCore_DP4 pid=28760) RuntimeError: Worker failed with error 'AssertionError: auto_functionalized was not removed', please check the stack trace above for the root cause
(EngineCore_DP4 pid=28760) DEBUG 04-24 07:16:23 [distributed/device_communicators/shm_broadcast.py:168] Canceling waiting reads on SHM Buffer
(EngineCore_DP7 pid=28763) ERROR 04-24 07:16:24 [v1/executor/multiproc_executor.py:292] Worker proc VllmWorker-0 died unexpectedly, shutting down executor.
(EngineCore_DP7 pid=28763) DEBUG 04-24 07:16:24 [v1/executor/multiproc_executor.py:453] Triggering shutdown of workers
(EngineCore_DP7 pid=28763) DEBUG 04-24 07:16:24 [v1/executor/multiproc_executor.py:434] Worker Termination: allow workers to gracefully shutdown
(EngineCore_DP7 pid=28763) Process EngineCore_DP7:
(EngineCore_DP7 pid=28763) Traceback (most recent call last):
(EngineCore_DP7 pid=28763) File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP7 pid=28763) self.run()
(EngineCore_DP7 pid=28763) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore_DP7 pid=28763) self._target(*self._args, **self._kwargs)
(EngineCore_DP7 pid=28763) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1114, in run_engine_core
(EngineCore_DP7 pid=28763) raise e
(EngineCore_DP7 pid=28763) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1076, in run_engine_core
(EngineCore_DP7 pid=28763) engine_core = DPEngineCoreProc(*args, **kwargs)
(EngineCore_DP7 pid=28763) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP7 pid=28763) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1618, in init
(EngineCore_DP7 pid=28763) super().init(
(EngineCore_DP7 pid=28763) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore_DP7 pid=28763) return func(*args, **kwargs)
(EngineCore_DP7 pid=28763) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP7 pid=28763) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 850, in init
(EngineCore_DP7 pid=28763) super().init(
(EngineCore_DP7 pid=28763) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 127, in init
(EngineCore_DP7 pid=28763) kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore_DP7 pid=28763) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP7 pid=28763) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore_DP7 pid=28763) return func(*args, **kwargs)
(EngineCore_DP7 pid=28763) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP7 pid=28763) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 249, in _initialize_kv_caches
(EngineCore_DP7 pid=28763) available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore_DP7 pid=28763) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP7 pid=28763) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 142, in determine_available_memory
(EngineCore_DP7 pid=28763) return self.collective_rpc("determine_available_memory")
(EngineCore_DP7 pid=28763) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP7 pid=28763) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 412, in collective_rpc
(EngineCore_DP7 pid=28763) return future if non_block else future.result()
(EngineCore_DP7 pid=28763) ^^^^^^^^^^^^^^^
(EngineCore_DP7 pid=28763) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 89, in result
(EngineCore_DP7 pid=28763) return super().result()
(EngineCore_DP7 pid=28763) ^^^^^^^^^^^^^^^^
(EngineCore_DP7 pid=28763) File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
(EngineCore_DP7 pid=28763) return self.__get_result()
(EngineCore_DP7 pid=28763) ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP7 pid=28763) File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(EngineCore_DP7 pid=28763) raise self._exception
(EngineCore_DP7 pid=28763) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 93, in _wait_for_response
(EngineCore_DP7 pid=28763) response = self.aggregate(self.get_response())
(EngineCore_DP7 pid=28763) ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP7 pid=28763) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 399, in get_response
(EngineCore_DP7 pid=28763) raise RuntimeError(
(EngineCore_DP7 pid=28763) RuntimeError: Worker failed with error 'AssertionError: auto_functionalized was not removed', please check the stack trace above for the root cause
(EngineCore_DP7 pid=28763) DEBUG 04-24 07:16:24 [distributed/device_communicators/shm_broadcast.py:168] Canceling waiting reads on SHM Buffer
Traceback (most recent call last):
File "/usr/local/bin/vllm", line 10, in
sys.exit(main())
^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 75, in main
args.dispatch_function(args)
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/serve.py", line 118, in cmd
run_multi_api_server(args)
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/serve.py", line 279, in run_multi_api_server
with launch_core_engines(
^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/contextlib.py", line 144, in exit
next(self.gen)
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 1094, in launch_core_engines
wait_for_engine_startup(
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 1153, in wait_for_engine_startup
raise RuntimeError(
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
/usr/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 8 leaked shared_memory objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
root@jf-deepseek-v4-flash-58b55ffc47-qxhzp:/vllm-workspace#

(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/utils.py", line 3401, in run^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] out = model(new_inputs)^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] File "/tmp/torchinductor_root/aj/cajpzp3f6mkmtg2ehq3z7xvu6ohrrdmu2nuzy3kvm776geg3wbd2.py", line 374, in call^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] buf1 = torch.ops.vllm.mhc_pre.default(buf0, arg3_1, arg4_1, arg5_1, 1e-06, 1e-06, 1e-06, 2.0, 20, n_splits=1)^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 865, in call^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] return self._op(*args, **kwargs)^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_compile.py", line 54, in inner^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] return disable_fn(*args, **kwargs)^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py", line 1263, in _fn^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] return fn(*args, **kwargs)^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 409, in torch_dispatch^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] res = func(*args, **kwargs)^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 865, in call^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] return self._op(*args, **kwargs)^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/mhc.py", line 263, in mhc_pre^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] tf32_hc_prenorm_gemm(^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/deep_gemm.py", line 479, in tf32_hc_prenorm_gemm^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] return _tf32_hc_prenorm_gemm_impl(^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] RuntimeError: Assertion error (/workspace/.deps/deepgemm-src/csrc/apis/hyperconnection.hpp:56): Unsupported architecture^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971]
(Worker_DP2_EP2 pid=62380) DEBUG 04-24 07:46:42 [distributed/device_communicators/shm_broadcast.py:191] Poller received notify event
(EngineCore_DP2 pid=59205) DEBUG 04-24 07:46:42 [distributed/device_communicators/shm_broadcast.py:191] Poller received notify event

[liuyibo]

Base model deployment isn't working out.

Here is my serving argument: --trust-remote-code --tensor-parallel-size=4 --kv-cache-dtype=fp8

And got the following error:

  File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/default_loader.py", line 381, in load_weights
    loaded_weights = model.load_weights(self.get_all_weights(model_config, model))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/deepseek_v4.py", line 846, in load_weights
    return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/reload/torchao_decorator.py", line 50, in patched_model_load_weights
    return original_load_weights(self, weights, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 355, in load_weights
    autoloaded_weights = set(self._load_module("", self.module, weights))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 302, in _load_module
    yield from self._load_module(
  File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 275, in _load_module
    loaded_params = module_load_weights(weights)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/deepseek_v4.py", line 711, in load_weights
    success = weight_loader(
              ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fused_moe/layer.py", line 1336, in weight_loader
    self._load_model_weight_or_group_weight_scale(
  File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fused_moe/layer.py", line 830, in _load_model_weight_or_group_weight_scale
    self._load_w13(
  File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fused_moe/layer.py", line 956, in _load_w13
    expert_data.copy_(loaded_weight)
RuntimeError: The size of tensor a (2048) must match the size of tensor b (4096) at non-singleton dimension 1

[huangmengwei-cmss]

Any plans for sm_80 support?

[pakjoeng]

HGX 8*H20-3e 141GB NOT WORK.

      containers:
        - args:
            - deepseek-ai/DeepSeek-V4-Pro
            - --host
            - "0.0.0.0"
            - --port
            - "80"
            - --trust-remote-code
            - --kv-cache-dtype
            - fp8
            - --block-size
            - '256'
            - --enable-expert-parallel
            - --data-parallel-size
            - '8'
            - --compilation-config
            - '{"cudagraph_mode":"FULL_AND_PIECEWISE", "custom_ops":["all"]}'
            - --max-model-len
            - '800000'
            - --tokenizer-mode
            - deepseek_v4
            - --tool-call-parser
            - deepseek_v4
            - --enable-auto-tool-choice 
            - --reasoning-parser
            - deepseek_v4
            - --served-model-name
            - ds-v4-pro
          image: vllm/vllm-openai:deepseekv4-cu129
          imagePullPolicy: IfNotPresent
          name: vllm-openai
          ports:
            - containerPort: 80
              protocol: TCP
          resources:
            limits:
              nvidia.com/gpu: "8"
            requests:
              nvidia.com/gpu: "8"
          env:
            - name: VLLM_USE_MODELSCOPE
              value: "true"
            - name: VLLM_ENGINE_READY_TIMEOUT_S
              value: "3600"

ERROR Logs:
ds-v4-pro-6d68d899b9-ccr5v_vllm-openai (1).log

And I tried

          env:
            - name: PYTORCH_CUDA_ALLOC_CONF
              value: "expandable_segments:True"
            - name: VLLM_DISABLE_COMPILE_CACHE
              value: '1'

and

            - --compilation-config
            - '{"cudagraph_mode":"NONE"}'
            - --gpu-memory-utilization
            - "0.96"
            - --max-model-len
            - '8192'

both of these not work.

[LZHgrla]

@linchpinlin
Same error #40778 (comment), in sm_120 GPU.
In my view, the root cause is that DeepGEMM only supports SM90 and SM100.

[Zha-Miku]

vllm/vllm-openai:deepseekv4-cu129VLLM/VLLM-openai：DeepSeekv4-CU129 8 * L20 46G

VLLM_LOGGING_LEVEL=DEBUG vllm serve /data/pretrained/DeepSeek-V4-Flash --served-model-name DeepSeek-V4-Flash --trust-remote-code --kv-cache-dtype fp8 --block-size 256 --enable-expert-parallel --data-parallel-size 8 --gpu-memory-utilization 0.85 --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE", "custom_ops":["all"]}' --attention_config.use_fp4_indexer_cache=True --tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 --enable-auto-tool-choice --reasoning-parser deepseek_v4

(EngineCore_DP1 pid=28757) RuntimeError: Worker failed with error 'AssertionError: auto_functionalized was not removed', please check the stack trace above for the root cause
(EngineCore_DP1 pid=28757) DEBUG 04-24 07:16:23 [distributed/device_communicators/shm_broadcast.py:168] Canceling waiting reads on SHM Buffer
terminate called without an active exception
(EngineCore_DP4 pid=28760) Process EngineCore_DP4:
(EngineCore_DP4 pid=28760) Traceback (most recent call last):
(EngineCore_DP4 pid=28760) File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP4 pid=28760) self.run()
(EngineCore_DP4 pid=28760) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore_DP4 pid=28760) self._target(*self._args, **self._kwargs)
(EngineCore_DP4 pid=28760) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1114, in run_engine_core
(EngineCore_DP4 pid=28760) raise e
(EngineCore_DP4 pid=28760) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1076, in run_engine_core
(EngineCore_DP4 pid=28760) engine_core = DPEngineCoreProc(*args, **kwargs)
(EngineCore_DP4 pid=28760) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP4 pid=28760) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1618, in init
(EngineCore_DP4 pid=28760) super().init(
(EngineCore_DP4 pid=28760) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore_DP4 pid=28760) return func(*args, **kwargs)
(EngineCore_DP4 pid=28760) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP4 pid=28760) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 850, in init
(EngineCore_DP4 pid=28760) super().init(
(EngineCore_DP4 pid=28760) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 127, in init
(EngineCore_DP4 pid=28760) kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore_DP4 pid=28760) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP4 pid=28760) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore_DP4 pid=28760) return func(*args, **kwargs)
(EngineCore_DP4 pid=28760) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP4 pid=28760) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 249, in _initialize_kv_caches
(EngineCore_DP4 pid=28760) available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore_DP4 pid=28760) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP4 pid=28760) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 142, in determine_available_memory
(EngineCore_DP4 pid=28760) return self.collective_rpc("determine_available_memory")
(EngineCore_DP4 pid=28760) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP4 pid=28760) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 412, in collective_rpc
(EngineCore_DP4 pid=28760) return future if non_block else future.result()
(EngineCore_DP4 pid=28760) ^^^^^^^^^^^^^^^
(EngineCore_DP4 pid=28760) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 89, in result
(EngineCore_DP4 pid=28760) return super().result()
(EngineCore_DP4 pid=28760) ^^^^^^^^^^^^^^^^
(EngineCore_DP4 pid=28760) File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
(EngineCore_DP4 pid=28760) return self.__get_result()
(EngineCore_DP4 pid=28760) ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP4 pid=28760) File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(EngineCore_DP4 pid=28760) raise self._exception
(EngineCore_DP4 pid=28760) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 93, in _wait_for_response
(EngineCore_DP4 pid=28760) response = self.aggregate(self.get_response())
(EngineCore_DP4 pid=28760) ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP4 pid=28760) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 399, in get_response
(EngineCore_DP4 pid=28760) raise RuntimeError(
(EngineCore_DP4 pid=28760) RuntimeError: Worker failed with error 'AssertionError: auto_functionalized was not removed', please check the stack trace above for the root cause
(EngineCore_DP4 pid=28760) DEBUG 04-24 07:16:23 [distributed/device_communicators/shm_broadcast.py:168] Canceling waiting reads on SHM Buffer
(EngineCore_DP7 pid=28763) ERROR 04-24 07:16:24 [v1/executor/multiproc_executor.py:292] Worker proc VllmWorker-0 died unexpectedly, shutting down executor.
(EngineCore_DP7 pid=28763) DEBUG 04-24 07:16:24 [v1/executor/multiproc_executor.py:453] Triggering shutdown of workers
(EngineCore_DP7 pid=28763) DEBUG 04-24 07:16:24 [v1/executor/multiproc_executor.py:434] Worker Termination: allow workers to gracefully shutdown
(EngineCore_DP7 pid=28763) Process EngineCore_DP7:
(EngineCore_DP7 pid=28763) Traceback (most recent call last):
(EngineCore_DP7 pid=28763) File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP7 pid=28763) self.run()
(EngineCore_DP7 pid=28763) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore_DP7 pid=28763) self._target(*self._args, **self._kwargs)
(EngineCore_DP7 pid=28763) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1114, in run_engine_core
(EngineCore_DP7 pid=28763) raise e
(EngineCore_DP7 pid=28763) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1076, in run_engine_core
(EngineCore_DP7 pid=28763) engine_core = DPEngineCoreProc(*args, **kwargs)
(EngineCore_DP7 pid=28763) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP7 pid=28763) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1618, in init
(EngineCore_DP7 pid=28763) super().init(
(EngineCore_DP7 pid=28763) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore_DP7 pid=28763) return func(*args, **kwargs)
(EngineCore_DP7 pid=28763) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP7 pid=28763) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 850, in init
(EngineCore_DP7 pid=28763) super().init(
(EngineCore_DP7 pid=28763) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 127, in init
(EngineCore_DP7 pid=28763) kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore_DP7 pid=28763) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP7 pid=28763) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore_DP7 pid=28763) return func(*args, **kwargs)
(EngineCore_DP7 pid=28763) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP7 pid=28763) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 249, in _initialize_kv_caches
(EngineCore_DP7 pid=28763) available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore_DP7 pid=28763) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP7 pid=28763) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 142, in determine_available_memory
(EngineCore_DP7 pid=28763) return self.collective_rpc("determine_available_memory")
(EngineCore_DP7 pid=28763) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP7 pid=28763) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 412, in collective_rpc
(EngineCore_DP7 pid=28763) return future if non_block else future.result()
(EngineCore_DP7 pid=28763) ^^^^^^^^^^^^^^^
(EngineCore_DP7 pid=28763) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 89, in result
(EngineCore_DP7 pid=28763) return super().result()
(EngineCore_DP7 pid=28763) ^^^^^^^^^^^^^^^^
(EngineCore_DP7 pid=28763) File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
(EngineCore_DP7 pid=28763) return self.__get_result()
(EngineCore_DP7 pid=28763) ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP7 pid=28763) File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(EngineCore_DP7 pid=28763) raise self._exception
(EngineCore_DP7 pid=28763) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 93, in _wait_for_response
(EngineCore_DP7 pid=28763) response = self.aggregate(self.get_response())
(EngineCore_DP7 pid=28763) ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP7 pid=28763) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 399, in get_response
(EngineCore_DP7 pid=28763) raise RuntimeError(
(EngineCore_DP7 pid=28763) RuntimeError: Worker failed with error 'AssertionError: auto_functionalized was not removed', please check the stack trace above for the root cause
(EngineCore_DP7 pid=28763) DEBUG 04-24 07:16:24 [distributed/device_communicators/shm_broadcast.py:168] Canceling waiting reads on SHM Buffer
Traceback (most recent call last):
File "/usr/local/bin/vllm", line 10, in
sys.exit(main())
^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 75, in main
args.dispatch_function(args)
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/serve.py", line 118, in cmd
run_multi_api_server(args)
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/serve.py", line 279, in run_multi_api_server
with launch_core_engines(
^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/contextlib.py", line 144, in exit
next(self.gen)
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 1094, in launch_core_engines
wait_for_engine_startup(
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 1153, in wait_for_engine_startup
raise RuntimeError(
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
/usr/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 8 leaked shared_memory objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
root@jf-deepseek-v4-flash-58b55ffc47-qxhzp:/vllm-workspace#

(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/utils.py", line 3401, in run^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] out = model(new_inputs)^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] File "/tmp/torchinductor_root/aj/cajpzp3f6mkmtg2ehq3z7xvu6ohrrdmu2nuzy3kvm776geg3wbd2.py", line 374, in call^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] buf1 = torch.ops.vllm.mhc_pre.default(buf0, arg3_1, arg4_1, arg5_1, 1e-06, 1e-06, 1e-06, 2.0, 20, n_splits=1)^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 865, in call^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] return self._op(*args, **kwargs)^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_compile.py", line 54, in inner^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] return disable_fn(*args, **kwargs)^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py", line 1263, in _fn^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] return fn(*args, **kwargs)^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 409, in torch_dispatch^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] res = func(*args, **kwargs)^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 865, in call^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] return self._op(*args, **kwargs)^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/mhc.py", line 263, in mhc_pre^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] tf32_hc_prenorm_gemm(^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/deep_gemm.py", line 479, in tf32_hc_prenorm_gemm^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] return _tf32_hc_prenorm_gemm_impl(^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971] RuntimeError: Assertion error (/workspace/.deps/deepgemm-src/csrc/apis/hyperconnection.hpp:56): Unsupported architecture^M
(Worker_DP2_EP2 pid=62380) ERROR 04-24 07:46:42 [v1/executor/multiproc_executor.py:971]
(Worker_DP2_EP2 pid=62380) DEBUG 04-24 07:46:42 [distributed/device_communicators/shm_broadcast.py:191] Poller received notify event
(EngineCore_DP2 pid=59205) DEBUG 04-24 07:46:42 [distributed/device_communicators/shm_broadcast.py:191] Poller received notify event

the same problem

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971] RuntimeError: Assertion error (/workspace/.deps/deepgemm-src/csrc/apis/hyperconnection.hpp:56): Unsupported architecture
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971] Traceback (most recent call last):
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 966, in worker_busy_loop
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]     output = func(*args, **kwargs)
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]              ^^^^^^^^^^^^^^^^^^^^^
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]     return func(*args, **kwargs)
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]            ^^^^^^^^^^^^^^^^^^^^^
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 370, in determine_available_memory
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]     self.model_runner.profile_run()
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5824, in profile_run
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]     hidden_states, last_hidden_states = self._dummy_run(
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]                                         ^^^^^^^^^^^^^^^^
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]     return func(*args, **kwargs)
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]            ^^^^^^^^^^^^^^^^^^^^^
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5512, in _dummy_run
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]     outputs = self.model(
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]               ^^^^^^^^^^^
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]     return self._call_impl(*args, **kwargs)
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1790, in _call_impl
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]     return forward_call(*args, **kwargs)
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/deepseek_v4.py", line 833, in forward
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]     hidden_states = self.model(
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]                     ^^^^^^^^^^^
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/decorators.py", line 467, in __call__
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]     return self.forward(*args, **kwargs)
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/deepseek_v4.py", line 629, in forward
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]     hidden_states = layer(
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]                     ^^^^^^
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]     return self._call_impl(*args, **kwargs)
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1790, in _call_impl
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]     return forward_call(*args, **kwargs)
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/deepseek_v4.py", line 519, in forward
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]     x, post, comb = self.hc_pre(
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]                     ^^^^^^^^^^^^
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/deepseek_v4.py", line 490, in hc_pre
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]     post_mix, res_mix, layer_input = torch.ops.vllm.mhc_pre(
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]                                      ^^^^^^^^^^^^^^^^^^^^^^^
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 1269, in __call__
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]     return self._op(*args, **kwargs)
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]            ^^^^^^^^^^^^^^^^^^^^^^^^^
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/mhc.py", line 263, in mhc_pre
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]     tf32_hc_prenorm_gemm(
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/deep_gemm.py", line 479, in tf32_hc_prenorm_gemm
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]     return _tf32_hc_prenorm_gemm_impl(
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971] RuntimeError: Assertion error (/workspace/.deps/deepgemm-src/csrc/apis/hyperconnection.hpp:56): Unsupported architecture
vllm  | (Worker_TP0_EP0 pid=407) ERROR 04-24 18:37:45 [multiproc_executor.py:971] 
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110] EngineCore failed to start.
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110] Traceback (most recent call last):
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1084, in run_engine_core
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110]     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110]   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110]     return func(*args, **kwargs)
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110]            ^^^^^^^^^^^^^^^^^^^^^
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 850, in __init__
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110]     super().__init__(
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 127, in __init__
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110]     kv_cache_config = self._initialize_kv_caches(vllm_config)
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110]                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110]   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110]     return func(*args, **kwargs)
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110]            ^^^^^^^^^^^^^^^^^^^^^
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 249, in _initialize_kv_caches
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110]     available_gpu_memory = self.model_executor.determine_available_memory()
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 142, in determine_available_memory
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110]     return self.collective_rpc("determine_available_memory")
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 412, in collective_rpc
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110]     return future if non_block else future.result()
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110]                                     ^^^^^^^^^^^^^^^
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 89, in result
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110]     return super().result()
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110]            ^^^^^^^^^^^^^^^^
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110]   File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110]     return self.__get_result()
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110]            ^^^^^^^^^^^^^^^^^^^
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110]   File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110]     raise self._exception
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 93, in _wait_for_response
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110]     response = self.aggregate(self.get_response())
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110]                               ^^^^^^^^^^^^^^^^^^^
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 399, in get_response
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110]     raise RuntimeError(
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:45 [core.py:1110] RuntimeError: Worker failed with error 'Assertion error (/workspace/.deps/deepgemm-src/csrc/apis/hyperconnection.hpp:56): Unsupported architecture', please check the stack trace above for the root cause
vllm  | (Worker_TP0_EP0 pid=407) WARNING 04-24 18:37:45 [multiproc_executor.py:893] WorkerProc was terminated
vllm  | (Worker_TP6_EP6 pid=413) WARNING 04-24 18:37:45 [multiproc_executor.py:893] WorkerProc was terminated
vllm  | (Worker_TP1_EP1 pid=408) WARNING 04-24 18:37:45 [multiproc_executor.py:893] WorkerProc was terminated
vllm  | (Worker_TP7_EP7 pid=414) WARNING 04-24 18:37:45 [multiproc_executor.py:893] WorkerProc was terminated
vllm  | (Worker_TP3_EP3 pid=410) WARNING 04-24 18:37:45 [multiproc_executor.py:893] WorkerProc was terminated
vllm  | (Worker_TP2_EP2 pid=409) WARNING 04-24 18:37:45 [multiproc_executor.py:893] WorkerProc was terminated
vllm  | (Worker_TP4_EP4 pid=411) WARNING 04-24 18:37:45 [multiproc_executor.py:893] WorkerProc was terminated
vllm  | (Worker_TP5_EP5 pid=412) WARNING 04-24 18:37:45 [multiproc_executor.py:893] WorkerProc was terminated
vllm  | (EngineCore pid=208) ERROR 04-24 18:37:47 [multiproc_executor.py:292] Worker proc VllmWorker-3 died unexpectedly, shutting down executor.
vllm  | (EngineCore pid=208) Process EngineCore:
vllm  | (EngineCore pid=208) Traceback (most recent call last):
vllm  | (EngineCore pid=208)   File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
vllm  | (EngineCore pid=208)     self.run()
vllm  | (EngineCore pid=208)   File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
vllm  | (EngineCore pid=208)     self._target(*self._args, **self._kwargs)
vllm  | (EngineCore pid=208)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1114, in run_engine_core
vllm  | (EngineCore pid=208)     raise e
vllm  | (EngineCore pid=208)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1084, in run_engine_core
vllm  | (EngineCore pid=208)     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
vllm  | (EngineCore pid=208)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm  | (EngineCore pid=208)   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
vllm  | (EngineCore pid=208)     return func(*args, **kwargs)
vllm  | (EngineCore pid=208)            ^^^^^^^^^^^^^^^^^^^^^
vllm  | (EngineCore pid=208)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 850, in __init__
vllm  | (EngineCore pid=208)     super().__init__(
vllm  | (EngineCore pid=208)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 127, in __init__
vllm  | (EngineCore pid=208)     kv_cache_config = self._initialize_kv_caches(vllm_config)
vllm  | (EngineCore pid=208)                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm  | (EngineCore pid=208)   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
vllm  | (EngineCore pid=208)     return func(*args, **kwargs)
vllm  | (EngineCore pid=208)            ^^^^^^^^^^^^^^^^^^^^^
vllm  | (EngineCore pid=208)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 249, in _initialize_kv_caches
vllm  | (EngineCore pid=208)     available_gpu_memory = self.model_executor.determine_available_memory()
vllm  | (EngineCore pid=208)                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm  | (EngineCore pid=208)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 142, in determine_available_memory
vllm  | (EngineCore pid=208)     return self.collective_rpc("determine_available_memory")
vllm  | (EngineCore pid=208)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm  | (EngineCore pid=208)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 412, in collective_rpc
vllm  | (EngineCore pid=208)     return future if non_block else future.result()
vllm  | (EngineCore pid=208)                                     ^^^^^^^^^^^^^^^
vllm  | (EngineCore pid=208)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 89, in result
vllm  | (EngineCore pid=208)     return super().result()
vllm  | (EngineCore pid=208)            ^^^^^^^^^^^^^^^^
vllm  | (EngineCore pid=208)   File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
vllm  | (EngineCore pid=208)     return self.__get_result()
vllm  | (EngineCore pid=208)            ^^^^^^^^^^^^^^^^^^^
vllm  | (EngineCore pid=208)   File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
vllm  | (EngineCore pid=208)     raise self._exception
vllm  | (EngineCore pid=208)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 93, in _wait_for_response
vllm  | (EngineCore pid=208)     response = self.aggregate(self.get_response())
vllm  | (EngineCore pid=208)                               ^^^^^^^^^^^^^^^^^^^
vllm  | (EngineCore pid=208)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 399, in get_response
vllm  | (EngineCore pid=208)     raise RuntimeError(
vllm  | (EngineCore pid=208) RuntimeError: Worker failed with error 'Assertion error (/workspace/.deps/deepgemm-src/csrc/apis/hyperconnection.hpp:56): Unsupported architecture', please check the stack trace above for the root cause
vllm  | (APIServer pid=1) Traceback (most recent call last):
vllm  | (APIServer pid=1)   File "/usr/local/bin/vllm", line 10, in <module>
vllm  | (APIServer pid=1)     sys.exit(main())
vllm  | (APIServer pid=1)              ^^^^^^
vllm  | (APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 75, in main
vllm  | (APIServer pid=1)     args.dispatch_function(args)
vllm  | (APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/serve.py", line 122, in cmd
vllm  | (APIServer pid=1)     uvloop.run(run_server(args))
vllm  | (APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 96, in run
vllm  | (APIServer pid=1)     return __asyncio.run(
vllm  | (APIServer pid=1)            ^^^^^^^^^^^^^^
vllm  | (APIServer pid=1)   File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run
vllm  | (APIServer pid=1)     return runner.run(main)
vllm  | (APIServer pid=1)            ^^^^^^^^^^^^^^^^
vllm  | (APIServer pid=1)   File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
vllm  | (APIServer pid=1)     return self._loop.run_until_complete(task)
vllm  | (APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm  | (APIServer pid=1)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
vllm  | (APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 48, in wrapper
vllm  | (APIServer pid=1)     return await main
vllm  | (APIServer pid=1)            ^^^^^^^^^^
vllm  | (APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 680, in run_server
vllm  | (APIServer pid=1)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
vllm  | (APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 694, in run_server_worker
vllm  | (APIServer pid=1)     async with build_async_engine_client(
vllm  | (APIServer pid=1)                ^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm  | (APIServer pid=1)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
vllm  | (APIServer pid=1)     return await anext(self.gen)
vllm  | (APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
vllm  | (APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 100, in build_async_engine_client
vllm  | (APIServer pid=1)     async with build_async_engine_client_from_engine_args(
vllm  | (APIServer pid=1)                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm  | (APIServer pid=1)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
vllm  | (APIServer pid=1)     return await anext(self.gen)
vllm  | (APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
vllm  | (APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 136, in build_async_engine_client_from_engine_args
vllm  | (APIServer pid=1)     async_llm = AsyncLLM.from_vllm_config(
vllm  | (APIServer pid=1)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm  | (APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 219, in from_vllm_config
vllm  | (APIServer pid=1)     return cls(
vllm  | (APIServer pid=1)            ^^^^
vllm  | (APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 148, in __init__
vllm  | (APIServer pid=1)     self.engine_core = EngineCoreClient.make_async_mp_client(
vllm  | (APIServer pid=1)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm  | (APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
vllm  | (APIServer pid=1)     return func(*args, **kwargs)
vllm  | (APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
vllm  | (APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 130, in make_async_mp_client
vllm  | (APIServer pid=1)     return AsyncMPClient(*client_args)
vllm  | (APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm  | (APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
vllm  | (APIServer pid=1)     return func(*args, **kwargs)
vllm  | (APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
vllm  | (APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 900, in __init__
vllm  | (APIServer pid=1)     super().__init__(
vllm  | (APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 535, in __init__
vllm  | (APIServer pid=1)     with launch_core_engines(
vllm  | (APIServer pid=1)          ^^^^^^^^^^^^^^^^^^^^
vllm  | (APIServer pid=1)   File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
vllm  | (APIServer pid=1)     next(self.gen)
vllm  | (APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 1094, in launch_core_engines
vllm  | (APIServer pid=1)     wait_for_engine_startup(
vllm  | (APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 1153, in wait_for_engine_startup
vllm  | (APIServer pid=1)     raise RuntimeError(
vllm  | (APIServer pid=1) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
vllm  | /usr/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 8 leaked shared_memory objects to clean up at shutdown
vllm  |   warnings.warn('resource_tracker: There appear to be %d '
vllm exited with code 0

env

(base) root@user-Rack-Server:/data/vllm# cat dsv4f.yml 
services:
  vllm:
    image: vllm/vllm-openai:deepseekv4-cu129
    #image: vllm/vllm-openai:cu130-nightly-7291d1b288558d48508e1a17c37b0aa170332264
    container_name: vllm
    volumes:
      #- ${HOME}/.cache/huggingface:/root/.cache/huggingface
      # If you use modelscope, you need mount this directory
      - /data/models:/models
    restart: always
    ports:
      - "23002:30000" # Un-comment and adjust if you want to publish a specific port
    environment:
      - TZ=Asia/Shanghai
      - HF_ENDPOINT=https://hf-mirror.com
      - VLLM_USE_MODELSCOPE=True
      - VLLM_TORCH_COMPILE_LEVEL=0
      - VLLM_USE_V1=0 

      - CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
    entrypoint: vllm serve
    command: 
      - /models/deepseek-ai/DeepSeek-V4-Flash/ 
      - --trust-remote-code
      - --tensor-parallel-size=8
        #- --enable-reasoning
      - --max-model-len=64000
        #- --max-seq-len-to-capture=200000
        #- --swap-space=16
      - --max-num-seqs=128
      - --enforce-eager
      - --enable-expert-parallel
      - --enable-auto-tool-choice
        #- --tool-call-parser
      - --tool-call-parser=hermes
      - --kv-cache-dtype=fp8
      - --served-model-name=qwen3
      - --host=0.0.0.0
      - --port=30000
    ipc: host
    privileged: true
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["0","1","2","3","4","5","6","7"]
              capabilities: [gpu]

[wuwenthink]

rtxpro6000*2 canot run it

PS C:\Users\wuwen> docker run --gpus all --privileged --ipc=host -p 8005:8005 -e CUDA_DEVICE_ORDER=PCI_BUS_ID -e CUDA_VISIBLE_DEVICES=0,1 -e NCCL_CUMEM_ENABLE=0 -v I:\AI-Chat\models\Deepseek:/models vllm/vllm-openai:deepseekv4-cu130 /models/DeepSeek-V4-Flash --trust-remote-code --kv-cache-dtype fp8 --block-size 256 --enable-expert-parallel --tensor-parallel-size 2 --attention_config.use_fp4_indexer_cache=True --tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 --enable-auto-tool-choice --reasoning-parser deepseek_v4 --host 0.0.0.0 --served-model-name VLLM-MODEL --gpu-memory-utilization 0.95 --async-scheduling --enable-prefix-caching --max-num-seqs 2
(APIServer pid=1) INFO 04-24 10:27:00 [utils.py:299]
(APIServer pid=1) INFO 04-24 10:27:00 [utils.py:299] █ █ █▄ ▄█
(APIServer pid=1) INFO 04-24 10:27:00 [utils.py:299] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.1.dev15830+g8d599d76a
(APIServer pid=1) INFO 04-24 10:27:00 [utils.py:299] █▄█▀ █ █ █ █ model /models/DeepSeek-V4-Flash
(APIServer pid=1) INFO 04-24 10:27:00 [utils.py:299] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=1) INFO 04-24 10:27:00 [utils.py:299]
(APIServer pid=1) INFO 04-24 10:27:00 [utils.py:233] non-default args: {'model_tag': '/models/DeepSeek-V4-Flash', 'enable_auto_tool_choice': True, 'tool_call_parser': 'deepseek_v4', 'host': '0.0.0.0', 'model': '/models/DeepSeek-V4-Flash', 'tokenizer_mode': 'deepseek_v4', 'trust_remote_code': True, 'served_model_name': ['VLLM-MODEL'], 'reasoning_parser': 'deepseek_v4', 'tensor_parallel_size': 2, 'enable_expert_parallel': True, 'block_size': 256, 'gpu_memory_utilization': 0.95, 'kv_cache_dtype': 'fp8', 'enable_prefix_caching': True, 'max_num_seqs': 2, 'async_scheduling': True, 'attention_config': AttentionConfig(backend=None, flash_attn_version=None, use_prefill_decode_attention=False, flash_attn_max_num_splits_for_cuda_graph=32, use_cudnn_prefill=False, use_trtllm_ragged_deepseek_prefill=False, use_trtllm_attention=None, disable_flashinfer_prefill=True, disable_flashinfer_q_quantization=False, use_prefill_query_quantization=False, use_fp4_indexer_cache=True)}
(APIServer pid=1) INFO 04-24 10:27:00 [config.py:763] Detected quantization_config.scale_fmt=ue8m0; enabling UE8M0 for DeepGEMM.
(APIServer pid=1) INFO 04-24 10:27:00 [config.py:449] Replacing legacy 'type' key with 'rope_type'
(APIServer pid=1) INFO 04-24 10:27:11 [model.py:555] Resolved architecture: DeepseekV4ForCausalLM
(APIServer pid=1) INFO 04-24 10:27:11 [model.py:1689] Using max model len 1048576
(APIServer pid=1) INFO 04-24 10:27:11 [cache.py:267] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor
(APIServer pid=1) INFO 04-24 10:27:11 [scheduler.py:238] Chunked prefill is enabled with max_num_batched_tokens=8192.
(APIServer pid=1) INFO 04-24 10:27:11 [vllm.py:819] Asynchronous scheduling is enabled.
(APIServer pid=1) INFO 04-24 10:27:11 [kernel.py:201] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'])
(APIServer pid=1) WARNING 04-24 10:27:12 [vllm.py:1234] Auto-initialization of reasoning token IDs failed. Please check whether your reasoning parser has implemented the reasoning_start_str and reasoning_end_str.
(APIServer pid=1) INFO 04-24 10:27:12 [compilation.py:294] Enabled custom fusions: norm_quant, act_quant
(EngineCore pid=273) INFO 04-24 10:27:20 [core.py:108] Initializing a V1 LLM engine (v0.1.dev15830+g8d599d76a) with config: model='/models/DeepSeek-V4-Flash', speculative_config=None, tokenizer='/models/DeepSeek-V4-Flash', skip_tokenizer_init=False, tokenizer_mode=deepseek_v4, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=1048576, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=deepseek_v4_fp8, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=fp8, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='deepseek_v4', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=VLLM-MODEL, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+quant_fp8', 'none', '+quant_fp8'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_images_per_batch': 0, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 4, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto')
(EngineCore pid=273) WARNING 04-24 10:27:20 [multiproc_executor.py:1038] Reducing Torch parallelism from 32 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
(EngineCore pid=273) INFO 04-24 10:27:20 [multiproc_executor.py:138] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.17.0.2 (local), world_size=2, local_world_size=2
WARNING 04-24 10:27:29 [interface.py:686] Using 'pin_memory=False' as WSL is detected. This may slow down the performance.
WARNING 04-24 10:27:29 [interface.py:686] Using 'pin_memory=False' as WSL is detected. This may slow down the performance.
(Worker pid=408) INFO 04-24 10:27:29 [parallel_state.py:1400] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:41607 backend=nccl
(Worker pid=409) INFO 04-24 10:27:29 [parallel_state.py:1400] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:41607 backend=nccl
(Worker pid=408) INFO 04-24 10:27:30 [pynccl.py:111] vLLM is using nccl==2.28.9
(Worker_TP1_EP1 pid=409) torch_dtype is deprecated! Use dtype instead!
(Worker_TP0_EP0 pid=408) torch_dtype is deprecated! Use dtype instead!
Loading safetensors checkpoint shards: 0% Completed | 0/46 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 2% Completed | 1/46 [00:06<04:41, 6.26s/it]
(Worker pid=408) WARNING 04-24 10:27:37 [symm_mem.py:66] SymmMemCommunicator: Device capability 12.0 not supported, communicator is not available.
(Worker pid=409) WARNING 04-24 10:27:37 [symm_mem.py:66] SymmMemCommunicator: Device capability 12.0 not supported, communicator is not available.
(Worker pid=408) WARNING 04-24 10:27:37 [custom_all_reduce.py:164] Custom allreduce is disabled because your platform lacks GPU P2P capability or P2P test failed. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(Worker pid=409) WARNING 04-24 10:27:37 [custom_all_reduce.py:164] Custom allreduce is disabled because your platform lacks GPU P2P capability or P2P test failed. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(Worker pid=408) INFO 04-24 10:27:40 [parallel_state.py:1713] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(Worker_TP0_EP0 pid=408) INFO 04-24 10:27:44 [gpu_model_runner.py:4763] Starting to load model /models/DeepSeek-V4-Flash...
(Worker_TP0_EP0 pid=408) INFO 04-24 10:27:45 [init.py:384] Selected CutlassFp8BlockScaledMMKernel for Fp8LinearMethod
(Worker_TP0_EP0 pid=408) INFO 04-24 10:27:49 [deepseek_v4_attention.py:607] Using DeepSeek's fp8_ds_mla KV cache format. To use standard fp8 kv-cache format, please set --attention-backend FLASHINFER_MLA_SPARSE
(Worker_TP0_EP0 pid=408) INFO 04-24 10:27:49 [layer.py:400] [EP Rank 0/2] Expert parallelism is enabled. Expert placement strategy: linear. Local/global number of experts: 128/256. Experts local to global index map: 0->0, 1->1, 2->2, 3->3, 4->4, 5->5, 6->6, 7->7, 8->8, 9->9, 10->10, 11->11, 12->12, 13->13, 14->14, 15->15, 16->16, 17->17, 18->18, 19->19, 20->20, 21->21, 22->22, 23->23, 24->24, 25->25, 26->26, 27->27, 28->28, 29->29, 30->30, 31->31, 32->32, 33->33, 34->34, 35->35, 36->36, 37->37, 38->38, 39->39, 40->40, 41->41, 42->42, 43->43, 44->44, 45->45, 46->46, 47->47, 48->48, 49->49, 50->50, 51->51, 52->52, 53->53, 54->54, 55->55, 56->56, 57->57, 58->58, 59->59, 60->60, 61->61, 62->62, 63->63, 64->64, 65->65, 66->66, 67->67, 68->68, 69->69, 70->70, 71->71, 72->72, 73->73, 74->74, 75->75, 76->76, 77->77, 78->78, 79->79, 80->80, 81->81, 82->82, 83->83, 84->84, 85->85, 86->86, 87->87, 88->88, 89->89, 90->90, 91->91, 92->92, 93->93, 94->94, 95->95, 96->96, 97->97, 98->98, 99->99, 100->100, 101->101, 102->102, 103->103, 104->104, 105->105, 106->106, 107->107, 108->108, 109->109, 110->110, 111->111, 112->112, 113->113, 114->114, 115->115, 116->116, 117->117, 118->118, 119->119, 120->120, 121->121, 122->122, 123->123, 124->124, 125->125, 126->126, 127->127.
(Worker_TP0_EP0 pid=408) INFO 04-24 10:27:50 [mxfp4.py:481] Using 'MARLIN' Mxfp4 MoE backend.
(Worker_TP0_EP0 pid=408) INFO 04-24 10:27:50 [deepseek_v4_attention.py:969] Using MXFP4 indexer cache for Lighening Indexer.
(Worker_TP0_EP0 pid=408) INFO 04-24 10:27:54 [weight_utils.py:904] Filesystem type for checkpoints: 9P. Checkpoint size: 148.66 GiB. Available RAM: 0.65 GiB.
(Worker_TP0_EP0 pid=408) INFO 04-24 10:27:54 [weight_utils.py:934] Auto-prefetch is disabled because the filesystem (9P) is not a recognized network FS (NFS/Lustre) and the checkpoint size (148.66 GiB) exceeds 90% of available RAM (0.65 GiB).
Loading safetensors checkpoint shards: 4% Completed | 2/46 [00:33<13:38, 18.60s/it]
Loading safetensors checkpoint shards: 7% Completed | 3/46 [00:52<13:19, 18.59s/it]
Loading safetensors checkpoint shards: 9% Completed | 4/46 [01:12<13:25, 19.18s/it]
Loading safetensors checkpoint shards: 11% Completed | 5/46 [01:28<12:22, 18.12s/it]
Loading safetensors checkpoint shards: 13% Completed | 6/46 [01:49<12:50, 19.27s/it]
Loading safetensors checkpoint shards: 15% Completed | 7/46 [02:06<11:59, 18.44s/it]
Loading safetensors checkpoint shards: 17% Completed | 8/46 [02:27<12:11, 19.26s/it]
Loading safetensors checkpoint shards: 20% Completed | 9/46 [02:45<11:32, 18.71s/it]
Loading safetensors checkpoint shards: 22% Completed | 10/46 [03:06<11:37, 19.38s/it]
Loading safetensors checkpoint shards: 24% Completed | 11/46 [03:18<10:06, 17.32s/it]
Loading safetensors checkpoint shards: 26% Completed | 12/46 [03:39<10:21, 18.28s/it]
Loading safetensors checkpoint shards: 28% Completed | 13/46 [03:59<10:25, 18.94s/it]
Loading safetensors checkpoint shards: 30% Completed | 14/46 [04:20<10:26, 19.58s/it]
Loading safetensors checkpoint shards: 33% Completed | 15/46 [04:40<10:11, 19.74s/it]
Loading safetensors checkpoint shards: 35% Completed | 16/46 [05:00<09:54, 19.80s/it]
Loading safetensors checkpoint shards: 37% Completed | 17/46 [05:22<09:50, 20.36s/it]
Loading safetensors checkpoint shards: 39% Completed | 18/46 [05:44<09:44, 20.88s/it]
Loading safetensors checkpoint shards: 41% Completed | 19/46 [06:06<09:33, 21.23s/it]
Loading safetensors checkpoint shards: 43% Completed | 20/46 [06:28<09:14, 21.32s/it]
Loading safetensors checkpoint shards: 46% Completed | 21/46 [06:50<09:00, 21.61s/it]
Loading safetensors checkpoint shards: 48% Completed | 22/46 [07:11<08:37, 21.57s/it]
Loading safetensors checkpoint shards: 50% Completed | 23/46 [07:32<08:06, 21.16s/it]
Loading safetensors checkpoint shards: 52% Completed | 24/46 [07:50<07:26, 20.29s/it]
Loading safetensors checkpoint shards: 54% Completed | 25/46 [08:07<06:47, 19.43s/it]
Loading safetensors checkpoint shards: 57% Completed | 26/46 [08:25<06:17, 18.85s/it]
Loading safetensors checkpoint shards: 59% Completed | 27/46 [08:41<05:42, 18.03s/it]
Loading safetensors checkpoint shards: 61% Completed | 28/46 [08:57<05:13, 17.44s/it]
Loading safetensors checkpoint shards: 63% Completed | 29/46 [09:12<04:46, 16.82s/it]
Loading safetensors checkpoint shards: 65% Completed | 30/46 [09:29<04:28, 16.78s/it]
Loading safetensors checkpoint shards: 67% Completed | 31/46 [09:44<04:06, 16.40s/it]
Loading safetensors checkpoint shards: 70% Completed | 32/46 [10:00<03:47, 16.22s/it]
Loading safetensors checkpoint shards: 72% Completed | 33/46 [10:17<03:31, 16.30s/it]
Loading safetensors checkpoint shards: 74% Completed | 34/46 [11:22<06:13, 31.10s/it]
Loading safetensors checkpoint shards: 76% Completed | 35/46 [12:43<08:25, 45.97s/it]
Loading safetensors checkpoint shards: 78% Completed | 36/46 [13:53<08:50, 53.06s/it]
Loading safetensors checkpoint shards: 80% Completed | 37/46 [15:06<08:51, 59.02s/it]
Loading safetensors checkpoint shards: 83% Completed | 38/46 [15:57<07:35, 56.89s/it]
Loading safetensors checkpoint shards: 85% Completed | 39/46 [16:28<05:43, 49.12s/it]
Loading safetensors checkpoint shards: 87% Completed | 40/46 [16:50<04:05, 40.94s/it]
Loading safetensors checkpoint shards: 89% Completed | 41/46 [17:10<02:53, 34.65s/it]
Loading safetensors checkpoint shards: 91% Completed | 42/46 [18:05<02:42, 40.72s/it]
Loading safetensors checkpoint shards: 93% Completed | 43/46 [18:32<01:49, 36.66s/it]
Loading safetensors checkpoint shards: 96% Completed | 44/46 [19:32<01:27, 43.69s/it]
time="2026-04-24T18:49:16+08:00" level=error msg="error waiting for container: unexpected EOF"
Loading safetensors checkpoint shards: 98% Completed | 45/46 [21:28<01:05, 65.38s/it]
Loading safetensors checkpoint shards: 100% Completed | 46/46 [21:30<00:00, 46.37s/it]
Loading safetensors checkpoint shards: 100% Completed | 46/46 [21:30<00:00, 28.07s/it]
(Worker_TP0_EP0 pid=408)
(Worker_TP0_EP0 pid=408) INFO 04-24 10:49:25 [default_loader.py:384] Loading weights took 1292.19 seconds
(Worker_TP0_EP0 pid=408) INFO 04-24 10:49:27 [mxfp4.py:1238] Using MoEPrepareAndFinalizeNoDPEPModular
(Worker_TP0_EP0 pid=408) INFO 04-24 10:50:00 [gpu_model_runner.py:4848] Model loading took 74.05 GiB memory and 1307.649939 seconds
(Worker_TP0_EP0 pid=408) INFO 04-24 10:50:20 [backends.py:1070] Using cache directory: /root/.cache/vllm/torch_compile_cache/6df291e80d/rank_0_0/backbone for vLLM's torch.compile
(Worker_TP0_EP0 pid=408) INFO 04-24 10:50:20 [backends.py:1130] Dynamo bytecode transform time: 19.33 s
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] WorkerProc hit an exception.
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] Traceback (most recent call last):
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 966, in worker_busy_loop
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] output = func(*args, **kwargs)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return func(*args, **kwargs)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 370, in determine_available_memory
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] self.model_runner.profile_run()
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5826, in profile_run
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] hidden_states, last_hidden_states = self._dummy_run(
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return func(*args, **kwargs)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5514, in _dummy_run
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] outputs = self.model(
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/cuda_graph.py", line 254, in call
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return self.runnable(*args, **kwargs)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return self._call_impl(*args, **kwargs)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1790, in _call_impl
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return forward_call(*args, **kwargs)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/deepseek_v4.py", line 833, in forward
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] hidden_states = self.model(
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/decorators.py", line 611, in call
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] self.aot_compiled_fn = self.aot_compile(*args, **kwargs)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/wrapper.py", line 183, in aot_compile
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return self._compiled_callable.aot_compile((args, kwargs))
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/dynamo/eval_frame.py", line 873, in aot_compile
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return aot_compile_fullgraph(
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/dynamo/aot_compile.py", line 368, in aot_compile_fullgraph
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] compiled_fn = backend(
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/init.py", line 2535, in call
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return self.compiler_fn(model, inputs, **self.kwargs)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/lib/python3.12/contextlib.py", line 81, in inner
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return func(*args, **kwds)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/backends.py", line 1196, in call
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] PiecewiseCompileInterpreter(
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return func(*args, **kwargs)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/backends.py", line 722, in run
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return super().run(*args)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/fx/interpreter.py", line 200, in run
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] self.env[node] = self.run_node(node)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/fx/interpreter.py", line 297, in run_node
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return getattr(self, n.op)(n.target, args, kwargs)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/backends.py", line 749, in call_module
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] piecewise_backend = PiecewiseBackend(
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/piecewise_backend.py", line 189, in init
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] self.compile_all_ranges()
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/piecewise_backend.py", line 265, in compile_all_ranges
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] range_entry.runnable = self.vllm_backend.compiler_manager.compile(
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return func(*args, **kwargs)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/backends.py", line 348, in compile
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] compiled_graph, handle = self.compiler.compile(
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/compiler_interface.py", line 351, in compile
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] compiled_graph = standalone_compile(graph, example_inputs, **compile_kwargs)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/init.py", line 444, in standalone_compile
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return standalone_compile(
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/standalone_compile.py", line 444, in standalone_compile
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] compiled_fn = compile_fx(
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 2527, in compile_fx
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return compile_fx(
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 2578, in compile_fx
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return _maybe_wrap_and_compile_fx_main(
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 2655, in _maybe_wrap_and_compile_fx_main
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return _compile_fx_main(
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 2864, in _compile_fx_main
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] raise e.remove_dynamo_frames() from None # see TORCHDYNAMO_VERBOSE=1
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 1053, in _compile_fx_inner
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] raise InductorError(e, currentframe()).with_traceback(
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 1037, in _compile_fx_inner
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] mb_compiled_graph = fx_codegen_and_compile(
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 1798, in fx_codegen_and_compile
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return scheme.codegen_and_compile(gm, example_inputs, inputs_to_check, graph_kwargs)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 1344, in codegen_and_compile
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] _recursive_post_grad_passes(gm, is_inference=is_inference)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 583, in _recursive_post_grad_passes
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] post_grad_passes(gm, is_inference)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/fx_passes/post_grad.py", line 358, in post_grad_passes
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] GraphTransformObserver(gm, "decompose_auto_functionalized").apply_graph_pass(
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/fx/passes/graph_transform_observer.py", line 103, in apply_graph_pass
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return pass_fn(self.gm.graph)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/fx_passes/post_grad.py", line 1392, in decompose_auto_functionalized
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] raise AssertionError("auto_functionalized was not removed")
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] torch._inductor.exc.InductorError: AssertionError: auto_functionalized was not removed
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] Traceback (most recent call last):
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 966, in worker_busy_loop
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] output = func(*args, **kwargs)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return func(*args, **kwargs)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 370, in determine_available_memory
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] self.model_runner.profile_run()
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5826, in profile_run
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] hidden_states, last_hidden_states = self._dummy_run(
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return func(*args, **kwargs)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5514, in _dummy_run
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] outputs = self.model(
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/cuda_graph.py", line 254, in call
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return self.runnable(*args, **kwargs)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return self._call_impl(*args, **kwargs)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1790, in _call_impl
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return forward_call(*args, **kwargs)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/deepseek_v4.py", line 833, in forward
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] hidden_states = self.model(
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/decorators.py", line 611, in call
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] self.aot_compiled_fn = self.aot_compile(*args, **kwargs)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/wrapper.py", line 183, in aot_compile
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return self._compiled_callable.aot_compile((args, kwargs))
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/dynamo/eval_frame.py", line 873, in aot_compile
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return aot_compile_fullgraph(
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/dynamo/aot_compile.py", line 368, in aot_compile_fullgraph
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] compiled_fn = backend(
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/init.py", line 2535, in call
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return self.compiler_fn(model, inputs, **self.kwargs)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/lib/python3.12/contextlib.py", line 81, in inner
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return func(*args, **kwds)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/backends.py", line 1196, in call
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] PiecewiseCompileInterpreter(
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return func(*args, **kwargs)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/backends.py", line 722, in run
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return super().run(*args)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/fx/interpreter.py", line 200, in run
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] self.env[node] = self.run_node(node)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/fx/interpreter.py", line 297, in run_node
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return getattr(self, n.op)(n.target, args, kwargs)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/backends.py", line 749, in call_module
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] piecewise_backend = PiecewiseBackend(
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/piecewise_backend.py", line 189, in init
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] self.compile_all_ranges()
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/piecewise_backend.py", line 265, in compile_all_ranges
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] range_entry.runnable = self.vllm_backend.compiler_manager.compile(
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return func(*args, **kwargs)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/backends.py", line 348, in compile
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] compiled_graph, handle = self.compiler.compile(
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/compiler_interface.py", line 351, in compile
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] compiled_graph = standalone_compile(graph, example_inputs, **compile_kwargs)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/init.py", line 444, in standalone_compile
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return standalone_compile(
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/standalone_compile.py", line 444, in standalone_compile
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] compiled_fn = compile_fx(
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 2527, in compile_fx
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return compile_fx(
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 2578, in compile_fx
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return _maybe_wrap_and_compile_fx_main(
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 2655, in _maybe_wrap_and_compile_fx_main
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return _compile_fx_main(
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 2864, in _compile_fx_main
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] raise e.remove_dynamo_frames() from None # see TORCHDYNAMO_VERBOSE=1
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 1053, in _compile_fx_inner
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] raise InductorError(e, currentframe()).with_traceback(
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 1037, in _compile_fx_inner
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] mb_compiled_graph = fx_codegen_and_compile(
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 1798, in fx_codegen_and_compile
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return scheme.codegen_and_compile(gm, example_inputs, inputs_to_check, graph_kwargs)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 1344, in codegen_and_compile
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] _recursive_post_grad_passes(gm, is_inference=is_inference)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 583, in _recursive_post_grad_passes
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] post_grad_passes(gm, is_inference)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/fx_passes/post_grad.py", line 358, in post_grad_passes
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] GraphTransformObserver(gm, "decompose_auto_functionalized").apply_graph_pass(
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/fx/passes/graph_transform_observer.py", line 103, in apply_graph_pass
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return pass_fn(self.gm.graph)
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/fx_passes/post_grad.py", line 1392, in decompose_auto_functionalized
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] raise AssertionError("auto_functionalized was not removed")
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971] torch._inductor.exc.InductorError: AssertionError: auto_functionalized was not removed
(Worker_TP0_EP0 pid=408) ERROR 04-24 10:50:25 [multiproc_executor.py:971]
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] WorkerProc hit an exception.
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] Traceback (most recent call last):
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 966, in worker_busy_loop
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] output = func(*args, **kwargs)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return func(*args, **kwargs)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 370, in determine_available_memory
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] self.model_runner.profile_run()
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5826, in profile_run
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] hidden_states, last_hidden_states = self._dummy_run(
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return func(*args, **kwargs)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5514, in _dummy_run
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] outputs = self.model(
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/cuda_graph.py", line 254, in call
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return self.runnable(*args, **kwargs)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return self._call_impl(*args, **kwargs)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1790, in _call_impl
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return forward_call(*args, **kwargs)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/deepseek_v4.py", line 833, in forward
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] hidden_states = self.model(
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/decorators.py", line 611, in call
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] self.aot_compiled_fn = self.aot_compile(*args, **kwargs)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/wrapper.py", line 183, in aot_compile
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return self._compiled_callable.aot_compile((args, kwargs))
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/dynamo/eval_frame.py", line 873, in aot_compile
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return aot_compile_fullgraph(
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/dynamo/aot_compile.py", line 368, in aot_compile_fullgraph
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] compiled_fn = backend(
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/init.py", line 2535, in call
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return self.compiler_fn(model, inputs, **self.kwargs)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/lib/python3.12/contextlib.py", line 81, in inner
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return func(*args, **kwds)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/backends.py", line 1196, in call
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] PiecewiseCompileInterpreter(
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return func(*args, **kwargs)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/backends.py", line 722, in run
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return super().run(*args)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/fx/interpreter.py", line 200, in run
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] self.env[node] = self.run_node(node)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/fx/interpreter.py", line 297, in run_node
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return getattr(self, n.op)(n.target, args, kwargs)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/backends.py", line 749, in call_module
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] piecewise_backend = PiecewiseBackend(
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/piecewise_backend.py", line 189, in init
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] self.compile_all_ranges()
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/piecewise_backend.py", line 265, in compile_all_ranges
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] range_entry.runnable = self.vllm_backend.compiler_manager.compile(
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return func(*args, **kwargs)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/backends.py", line 348, in compile
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] compiled_graph, handle = self.compiler.compile(
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/compiler_interface.py", line 351, in compile
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] compiled_graph = standalone_compile(graph, example_inputs, **compile_kwargs)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/init.py", line 444, in standalone_compile
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return standalone_compile(
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/standalone_compile.py", line 444, in standalone_compile
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] compiled_fn = compile_fx(
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 2527, in compile_fx
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return compile_fx(
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 2578, in compile_fx
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return _maybe_wrap_and_compile_fx_main(
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 2655, in _maybe_wrap_and_compile_fx_main
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return _compile_fx_main(
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 2864, in _compile_fx_main
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] raise e.remove_dynamo_frames() from None # see TORCHDYNAMO_VERBOSE=1
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 1053, in _compile_fx_inner
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] raise InductorError(e, currentframe()).with_traceback(
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 1037, in _compile_fx_inner
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] mb_compiled_graph = fx_codegen_and_compile(
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 1798, in fx_codegen_and_compile
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return scheme.codegen_and_compile(gm, example_inputs, inputs_to_check, graph_kwargs)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 1344, in codegen_and_compile
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] _recursive_post_grad_passes(gm, is_inference=is_inference)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 583, in _recursive_post_grad_passes
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] post_grad_passes(gm, is_inference)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/fx_passes/post_grad.py", line 358, in post_grad_passes
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] GraphTransformObserver(gm, "decompose_auto_functionalized").apply_graph_pass(
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/fx/passes/graph_transform_observer.py", line 103, in apply_graph_pass
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return pass_fn(self.gm.graph)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/fx_passes/post_grad.py", line 1392, in decompose_auto_functionalized
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] raise AssertionError("auto_functionalized was not removed")
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] torch._inductor.exc.InductorError: AssertionError: auto_functionalized was not removed
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] Traceback (most recent call last):
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 966, in worker_busy_loop
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] output = func(*args, **kwargs)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return func(*args, **kwargs)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 370, in determine_available_memory
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] self.model_runner.profile_run()
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5826, in profile_run
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] hidden_states, last_hidden_states = self._dummy_run(
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return func(*args, **kwargs)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5514, in _dummy_run
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] outputs = self.model(
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/cuda_graph.py", line 254, in call
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return self.runnable(*args, **kwargs)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return self._call_impl(*args, **kwargs)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1790, in _call_impl
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return forward_call(*args, **kwargs)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/deepseek_v4.py", line 833, in forward
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] hidden_states = self.model(
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/decorators.py", line 611, in call
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] self.aot_compiled_fn = self.aot_compile(*args, **kwargs)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/wrapper.py", line 183, in aot_compile
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return self._compiled_callable.aot_compile((args, kwargs))
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/dynamo/eval_frame.py", line 873, in aot_compile
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return aot_compile_fullgraph(
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/dynamo/aot_compile.py", line 368, in aot_compile_fullgraph
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] compiled_fn = backend(
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/init.py", line 2535, in call
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return self.compiler_fn(model, inputs, **self.kwargs)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/lib/python3.12/contextlib.py", line 81, in inner
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return func(*args, **kwds)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/backends.py", line 1196, in call
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] PiecewiseCompileInterpreter(
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return func(*args, **kwargs)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/backends.py", line 722, in run
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return super().run(*args)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/fx/interpreter.py", line 200, in run
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] self.env[node] = self.run_node(node)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/fx/interpreter.py", line 297, in run_node
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return getattr(self, n.op)(n.target, args, kwargs)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/backends.py", line 749, in call_module
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] piecewise_backend = PiecewiseBackend(
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/piecewise_backend.py", line 189, in init
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] self.compile_all_ranges()
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/piecewise_backend.py", line 265, in compile_all_ranges
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] range_entry.runnable = self.vllm_backend.compiler_manager.compile(
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return func(*args, **kwargs)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/backends.py", line 348, in compile
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] compiled_graph, handle = self.compiler.compile(
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/compiler_interface.py", line 351, in compile
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] compiled_graph = standalone_compile(graph, example_inputs, **compile_kwargs)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/init.py", line 444, in standalone_compile
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return standalone_compile(
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/standalone_compile.py", line 444, in standalone_compile
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] compiled_fn = compile_fx(
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 2527, in compile_fx
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return compile_fx(
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 2578, in compile_fx
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return _maybe_wrap_and_compile_fx_main(
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 2655, in _maybe_wrap_and_compile_fx_main
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return _compile_fx_main(
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 2864, in _compile_fx_main
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] raise e.remove_dynamo_frames() from None # see TORCHDYNAMO_VERBOSE=1
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 1053, in _compile_fx_inner
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] raise InductorError(e, currentframe()).with_traceback(
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 1037, in _compile_fx_inner
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] mb_compiled_graph = fx_codegen_and_compile(
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 1798, in fx_codegen_and_compile
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return scheme.codegen_and_compile(gm, example_inputs, inputs_to_check, graph_kwargs)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 1344, in codegen_and_compile
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] _recursive_post_grad_passes(gm, is_inference=is_inference)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 583, in _recursive_post_grad_passes
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] post_grad_passes(gm, is_inference)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/fx_passes/post_grad.py", line 358, in post_grad_passes
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] GraphTransformObserver(gm, "decompose_auto_functionalized").apply_graph_pass(
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/fx/passes/graph_transform_observer.py", line 103, in apply_graph_pass
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] return pass_fn(self.gm.graph)
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] ^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/fx_passes/post_grad.py", line 1392, in decompose_auto_functionalized
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] raise AssertionError("auto_functionalized was not removed")
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971] torch._inductor.exc.InductorError: AssertionError: auto_functionalized was not removed
(Worker_TP1_EP1 pid=409) ERROR 04-24 10:50:25 [multiproc_executor.py:971]
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] EngineCore failed to start.
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] Traceback (most recent call last):
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1084, in run_engine_core
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] return func(*args, **kwargs)
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 850, in init
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] super().init(
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 127, in init
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] return func(*args, **kwargs)
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 249, in _initialize_kv_caches
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 142, in determine_available_memory
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] return self.collective_rpc("determine_available_memory")
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 412, in collective_rpc
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] return future if non_block else future.result()
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] ^^^^^^^^^^^^^^^
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 89, in result
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] return super().result()
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] ^^^^^^^^^^^^^^^^
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] return self.__get_result()
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] raise self._exception
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 93, in _wait_for_response
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] response = self.aggregate(self.get_response())
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 399, in get_response
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] raise RuntimeError(
(EngineCore pid=273) ERROR 04-24 10:50:25 [core.py:1110] RuntimeError: Worker failed with error 'AssertionError: auto_functionalized was not removed', please check the stack trace above for the root cause
(Worker_TP0_EP0 pid=408) WARNING 04-24 10:50:25 [multiproc_executor.py:893] WorkerProc was terminated
(Worker_TP1_EP1 pid=409) WARNING 04-24 10:50:25 [multiproc_executor.py:893] WorkerProc was terminated
(EngineCore pid=273) ERROR 04-24 10:50:38 [multiproc_executor.py:292] Worker proc VllmWorker-0 died unexpectedly, shutting down executor.
(EngineCore pid=273) Process EngineCore:
(EngineCore pid=273) Traceback (most recent call last):
(EngineCore pid=273) File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore pid=273) self.run()
(EngineCore pid=273) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore pid=273) self._target(*self._args, **self._kwargs)
(EngineCore pid=273) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1114, in run_engine_core
(EngineCore pid=273) raise e
(EngineCore pid=273) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1084, in run_engine_core
(EngineCore pid=273) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=273) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=273) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=273) return func(*args, **kwargs)
(EngineCore pid=273) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=273) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 850, in init
(EngineCore pid=273) super().init(
(EngineCore pid=273) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 127, in init
(EngineCore pid=273) kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=273) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=273) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=273) return func(*args, **kwargs)
(EngineCore pid=273) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=273) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 249, in _initialize_kv_caches
(EngineCore pid=273) available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=273) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=273) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 142, in determine_available_memory
(EngineCore pid=273) return self.collective_rpc("determine_available_memory")
(EngineCore pid=273) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=273) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 412, in collective_rpc
(EngineCore pid=273) return future if non_block else future.result()
(EngineCore pid=273) ^^^^^^^^^^^^^^^
(EngineCore pid=273) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 89, in result
(EngineCore pid=273) return super().result()
(EngineCore pid=273) ^^^^^^^^^^^^^^^^
(EngineCore pid=273) File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
(EngineCore pid=273) return self.__get_result()
(EngineCore pid=273) ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=273) File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(EngineCore pid=273) raise self._exception
(EngineCore pid=273) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 93, in _wait_for_response
(EngineCore pid=273) response = self.aggregate(self.get_response())
(EngineCore pid=273) ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=273) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 399, in get_response
(EngineCore pid=273) raise RuntimeError(
(EngineCore pid=273) RuntimeError: Worker failed with error 'AssertionError: auto_functionalized was not removed', please check the stack trace above for the root cause
(APIServer pid=1) Traceback (most recent call last):
(APIServer pid=1) File "/usr/local/bin/vllm", line 10, in
(APIServer pid=1) sys.exit(main())
(APIServer pid=1) ^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 75, in main
(APIServer pid=1) args.dispatch_function(args)
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/serve.py", line 122, in cmd
(APIServer pid=1) uvloop.run(run_server(args))
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/uvloop/init.py", line 96, in run
(APIServer pid=1) return __asyncio.run(
(APIServer pid=1) ^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run
(APIServer pid=1) return runner.run(main)
(APIServer pid=1) ^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=1) return self._loop.run_until_complete(task)
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/uvloop/init.py", line 48, in wrapper
(APIServer pid=1) return await main
(APIServer pid=1) ^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 680, in run_server
(APIServer pid=1) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 694, in run_server_worker
(APIServer pid=1) async with build_async_engine_client(
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
(APIServer pid=1) return await anext(self.gen)
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 100, in build_async_engine_client
(APIServer pid=1) async with build_async_engine_client_from_engine_args(
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
(APIServer pid=1) return await anext(self.gen)
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 136, in build_async_engine_client_from_engine_args
(APIServer pid=1) async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 219, in from_vllm_config
(APIServer pid=1) return cls(
(APIServer pid=1) ^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 148, in init
(APIServer pid=1) self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=1) return func(*args, **kwargs)
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 130, in make_async_mp_client
(APIServer pid=1) return AsyncMPClient(*client_args)
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=1) return func(*args, **kwargs)
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 900, in init
(APIServer pid=1) super().init(
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 535, in init
(APIServer pid=1) with launch_core_engines(
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/lib/python3.12/contextlib.py", line 144, in exit
(APIServer pid=1) next(self.gen)
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 1094, in launch_core_engines
(APIServer pid=1) wait_for_engine_startup(
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 1153, in wait_for_engine_startup
(APIServer pid=1) raise RuntimeError(
(APIServer pid=1) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore': 1}
/usr/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 2 leaked shared_memory objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '

What's next:
Debug this container error with Gordon → docker ai "help me fix this container error"
PS C:\Users\wuwen>

[dongfangzan]

"I successfully deployed the model on 8x NVIDIA H100 GPUs, but the Time to First Token (TTFT) is extremely long, making the inference very slow. My deployment script is as follows:"

services:
  vllm:
    image: vllm/vllm-openai:deepseekv4-cu130
    container_name: vllm-deepseek-v4
    privileged: true
    ipc: host
    ports:
      - "8000:8000"
    volumes:
      - /apps/mymodels/local/cache:/root/.cache/huggingface
    environment:
      VLLM_ENGINE_READY_TIMEOUT_S: "3600"
    command: >
      /root/.cache/huggingface/hub/deepseek-ai/DeepSeek-V4-Flash
      --served-model-name deepseek-v4
      --trust-remote-code
      --kv-cache-dtype fp8
      --block-size 256
      --enable-expert-parallel
      --data-parallel-size 8
      --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE", "custom_ops":["all"]}'
      --tokenizer-mode deepseek_v4
      --tool-call-parser deepseek_v4
      --enable-auto-tool-choice
      --reasoning-parser deepseek_v4
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Additionally, when using it within Claude Code, there are numerous errors related to tool calls, as shown in the image below:

[Mewral]

@dongfangzan 老哥，我也是8卡H100，跟你命令一样，但是启动就报错，有遇到这个问题吗
报错信息：

(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971] WorkerProc hit an exception.
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971] Traceback (most recent call last):
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 966, in worker_busy_loop
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]     output = func(*args, **kwargs)
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]              ^^^^^^^^^^^^^^^^^^^^^
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]     return func(*args, **kwargs)
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 381, in determine_available_memory
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]     cudagraph_memory_estimate = self.model_runner.profile_cudagraph_memory()
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]     return func(*args, **kwargs)
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5915, in profile_cudagraph_memory
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]     self._init_minimal_kv_cache_for_profiling()
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5859, in _init_minimal_kv_cache_for_profiling
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]     self.initialize_kv_cache(minimal_config, is_profiling=True)
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 6805, in initialize_kv_cache
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]     self.may_reinitialize_input_batch(kv_cache_config, kernel_block_sizes)
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 6462, in may_reinitialize_input_batch
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]     self.input_batch = InputBatch(
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]                        ^^^^^^^^^^^
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_input_batch.py", line 153, in __init__
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]     self.block_table = MultiGroupBlockTable(
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]                        ^^^^^^^^^^^^^^^^^^^^^
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/block_table.py", line 261, in __init__
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]     BlockTable(
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/block_table.py", line 70, in __init__
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]     self.block_table = self._make_buffer(
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]                        ^^^^^^^^^^^^^^^^^^
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/block_table.py", line 218, in _make_buffer
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]     return CpuGpuBuffer(
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]            ^^^^^^^^^^^^^
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/utils.py", line 119, in __init__
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]     self.cpu = torch.zeros(*size, dtype=dtype, device="cpu", pin_memory=pin_memory)
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971] torch.AcceleratorError: CUDA error: invalid argument
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971] Search for `cudaErrorInvalidValue' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971] Traceback (most recent call last):
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 966, in worker_busy_loop
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]     output = func(*args, **kwargs)
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]              ^^^^^^^^^^^^^^^^^^^^^
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]     return func(*args, **kwargs)
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 381, in determine_available_memory
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]     cudagraph_memory_estimate = self.model_runner.profile_cudagraph_memory()
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]     return func(*args, **kwargs)
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5915, in profile_cudagraph_memory
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]     self._init_minimal_kv_cache_for_profiling()
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5859, in _init_minimal_kv_cache_for_profiling
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]     self.initialize_kv_cache(minimal_config, is_profiling=True)
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 6805, in initialize_kv_cache
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]     self.may_reinitialize_input_batch(kv_cache_config, kernel_block_sizes)
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 6462, in may_reinitialize_input_batch
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]     self.input_batch = InputBatch(
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]                        ^^^^^^^^^^^
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_input_batch.py", line 153, in __init__
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]     self.block_table = MultiGroupBlockTable(
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]                        ^^^^^^^^^^^^^^^^^^^^^
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/block_table.py", line 261, in __init__
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]     BlockTable(
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/block_table.py", line 70, in __init__
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]     self.block_table = self._make_buffer(
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]                        ^^^^^^^^^^^^^^^^^^
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/block_table.py", line 218, in _make_buffer
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]     return CpuGpuBuffer(
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]            ^^^^^^^^^^^^^
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/utils.py", line 119, in __init__
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]     self.cpu = torch.zeros(*size, dtype=dtype, device="cpu", pin_memory=pin_memory)
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971] torch.AcceleratorError: CUDA error: invalid argument
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971] Search for `cudaErrorInvalidValue' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]
(Worker_DP2_EP2 pid=3809) ERROR 04-25 03:28:44 [multiproc_executor.py:971]
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110] EngineCore failed to start.
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110] Traceback (most recent call last):
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1076, in run_engine_core
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]     engine_core = DPEngineCoreProc(*args, **kwargs)
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1618, in __init__
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]     super().__init__(
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]     return func(*args, **kwargs)
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 850, in __init__
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]     super().__init__(
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 127, in __init__
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]     kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]     return func(*args, **kwargs)
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 249, in _initialize_kv_caches
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 142, in determine_available_memory
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]     return self.collective_rpc("determine_available_memory")
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 412, in collective_rpc
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]     return future if non_block else future.result()
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]                                     ^^^^^^^^^^^^^^^
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 89, in result
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]     return super().result()
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]            ^^^^^^^^^^^^^^^^
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]   File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]     return self.__get_result()
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]            ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]   File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]     raise self._exception
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 93, in _wait_for_response
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]     response = self.aggregate(self.get_response())
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]                               ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 399, in get_response
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110]     raise RuntimeError(
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110] RuntimeError: Worker failed with error 'CUDA error: invalid argument
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110] Search for `cudaErrorInvalidValue' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(EngineCore_DP2 pid=595) ERROR 04-25 03:28:44 [core.py:1110] ', please check the stack trace above for the root cause
(Worker_DP2_EP2 pid=3809) WARNING 04-25 03:28:44 [multiproc_executor.py:893] WorkerProc was terminated

[F0undLinks]

mark