[Usage]: Does DeepSeek-R1 1.58-bit Dynamic Quant work on VLLM?

[shimmyshimmer]

Your current environment

Hey guys! Recently in our blogpost we wrote that vLLM supports GGUFs however we've been getting many people saying that the R1 GGUFs don't actually work in VLLM at the moment and they get errors.

I'm guessing it's not supported at the moment? Thank you! :)

Blog: https://unsloth.ai/blog/deepseekr1-dynamic
Model: https://huggingface.co/unsloth/DeepSeek-R1-GGUF

How would you like to use vllm

Run DeepSeek-R1 1.58

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[robertgshaw2-redhat]

Im not sure of the state. Can you try it?

[Isotr0py]

I'm afraid not. Because the GGUF support in vllm is dependent on the GGUF interoperability in transformers (we depend on it to extract hf_config from GGUF), and Deepseek and its GGUF interoperability hasn't supported in transformers yet: https://huggingface.co/docs/transformers/v4.48.2/en/gguf#supported-model-architectures

[shimmyshimmer]

Im not sure of the state. Can you try it?

Oh yes we tested it a few hours unfortunately it doesnt work

I'm afraid not. Because the GGUF support in vllm is dependent on the GGUF interoperability in transformers (we depend on it to extract hf_config from GGUF), and Deepseek and its GGUF interoperability hasn't supported in transformers yet: https://huggingface.co/docs/transformers/v4.48.2/en/gguf#supported-model-architectures

Alright thanks so much for letting me know. Once it's supported we'll let others know as well :)

[dannydabbles]

Steps to reproduce the vLLM issues I'm seeing here via the standard vllm/vllm-openai:latest Docker image.

ValueError: GGUF model with architecture deepseek2 is not supported yet.

[tugot17]

@shimmyshimmer have you seen if this is started within transformer?

[0x9334]

Is this feature not supported yet? I tested it using the vLLM image v0.7.2 and encountered several bugs:

2025-02-07T08:33:07.776941692Z ERROR 02-07 00:33:07 engine.py:389]   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 380, in run_mp_engine
2025-02-07T08:33:07.776972829Z ERROR 02-07 00:33:07 engine.py:389]     engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
2025-02-07T08:33:07.776974912Z ERROR 02-07 00:33:07 engine.py:389]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-07T08:33:07.776976415Z ERROR 02-07 00:33:07 engine.py:389]   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 118, in from_engine_args
2025-02-07T08:33:07.776978141Z ERROR 02-07 00:33:07 engine.py:389]     engine_config = engine_args.create_engine_config(usage_context)
2025-02-07T08:33:07.776979226Z ERROR 02-07 00:33:07 engine.py:389]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-07T08:33:07.776980637Z ERROR 02-07 00:33:07 engine.py:389]   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 1075, in create_engine_config
2025-02-07T08:33:07.776981730Z ERROR 02-07 00:33:07 engine.py:389]     model_config = self.create_model_config()
2025-02-07T08:33:07.776983621Z ERROR 02-07 00:33:07 engine.py:389]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-07T08:33:07.776984750Z ERROR 02-07 00:33:07 engine.py:389]   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 998, in create_model_config
2025-02-07T08:33:07.776986439Z ERROR 02-07 00:33:07 engine.py:389]     return ModelConfig(
2025-02-07T08:33:07.776988123Z ERROR 02-07 00:33:07 engine.py:389]            ^^^^^^^^^^^^
2025-02-07T08:33:07.776989268Z ERROR 02-07 00:33:07 engine.py:389]   File "/usr/local/lib/python3.12/dist-packages/vllm/config.py", line 302, in __init__
2025-02-07T08:33:07.776990641Z ERROR 02-07 00:33:07 engine.py:389]     hf_config = get_config(self.model, trust_remote_code, revision,
2025-02-07T08:33:07.776991921Z ERROR 02-07 00:33:07 engine.py:389]                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-07T08:33:07.776992892Z ERROR 02-07 00:33:07 engine.py:389]   File "/usr/local/lib/python3.12/dist-packages/vllm/transformers_utils/config.py", line 204, in get_config
2025-02-07T08:33:07.776994229Z ERROR 02-07 00:33:07 engine.py:389]     config_dict, _ = PretrainedConfig.get_config_dict(
2025-02-07T08:33:07.776996662Z ERROR 02-07 00:33:07 engine.py:389]                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-07T08:33:07.776998004Z ERROR 02-07 00:33:07 engine.py:389]   File "/usr/local/lib/python3.12/dist-packages/transformers/configuration_utils.py", line 591, in get_config_dict
2025-02-07T08:33:07.776999097Z ERROR 02-07 00:33:07 engine.py:389]     config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
2025-02-07T08:33:07.777000273Z ERROR 02-07 00:33:07 engine.py:389]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-07T08:33:07.777001317Z ERROR 02-07 00:33:07 engine.py:389]   File "/usr/local/lib/python3.12/dist-packages/transformers/configuration_utils.py", line 682, in _get_config_dict
2025-02-07T08:33:07.777002531Z ERROR 02-07 00:33:07 engine.py:389]     config_dict = load_gguf_checkpoint(resolved_config_file, return_tensors=False)["config"]
2025-02-07T08:33:07.777003754Z ERROR 02-07 00:33:07 engine.py:389]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-07T08:33:07.777004829Z ERROR 02-07 00:33:07 engine.py:389]   File "/usr/local/lib/python3.12/dist-packages/transformers/modeling_gguf_pytorch_utils.py", line 387, in load_gguf_checkpoint
2025-02-07T08:33:07.777006234Z ERROR 02-07 00:33:07 engine.py:389]     raise ValueError(f"GGUF model with architecture {architecture} is not supported yet.")
2025-02-07T08:33:07.777007264Z ERROR 02-07 00:33:07 engine.py:389] ValueError: GGUF model with architecture deepseek2 is not supported yet.

[shimmyshimmer]

@shimmyshimmer have you seen if this is started within transformer?

We haven't tested it yet unfortunately. :(

[chuangzhidan]

no luck here too

[lys791227]

ValueError: GGUF model with architecture deepseek2 is not supported yet.

I also same.

[Jackarry188]

same too, ValueError: GGUF model with architecture deepseek2 is not supported yet.

[SzymonOzog]

Added support in #13167 still unoptimized though

[shimmyshimmer]

Added support in #13167 still unoptimized though

Fantastic thanks guys I'll let people know

[kissint8]

Unfortunately, I encountered the error GGUF model with architecture deepseek2 is not supported yet., when I tried to deploy the DeepSeek R1 Q4_K_M model.