[Bug]: --speculative-config '{"method": "ngram", "num_speculative_tokens": 10, "ngram_prompt_lookup_max": 10}' does not work

[bradleyx12]

Your current environment

vllm serve [model] ----speculative-config '{"method": "ngram", "num_speculative_tokens": 10, "ngram_prompt_lookup_max": 10}' does not work. Pass in any model you would like.

🐛 Describe the bug

Bug: --speculative-config argument parsing fails with AttributeError

Problem

When running vllm serve (model) --speculative-config, the command fails because the argument gets parsed differently than expected, causing it to be converted into a SpeculativeConfig object instead of remaining as a dictionary.

Root Cause

The issue occurs in vllm/engine/arg_utils.py where --speculative-config is added through the VllmConfig argument group:

vllm_kwargs = get_kwargs(VllmConfig)
vllm_group = parser.add_argument_group(
    title="VllmConfig",
    description=VllmConfig.__doc__,
)
vllm_group.add_argument("--speculative-config",
                        **vllm_kwargs["speculative_config"])

This causes the argument to be automatically parsed into a SpeculativeConfig object. However, downstream code expects it to be a dictionary and calls .get() on it, resulting in:

AttributeError: 'SpeculativeConfig' object has no attribute 'get'

This error occurs in vllm/engine/arg_utils.py, line 1458, in is_v1_supported_oracle:

self.speculative_config.get("method") == "draft_model"

Solution

Remove --speculative-config from the VllmConfig argument group and add it as a separate argument with proper JSON parsing:

vllm_kwargs = get_kwargs(VllmConfig)
vllm_group = parser.add_argument_group(
    title="VllmConfig",
    description=VllmConfig.__doc__,
)
# vllm_group.add_argument("--speculative-config",
#                         **vllm_kwargs["speculative_config"])
vllm_group.add_argument("--kv-transfer-config",
                        **vllm_kwargs["kv_transfer_config"])
vllm_group.add_argument('--kv-events-config',
                        **vllm_kwargs["kv_events_config"])
vllm_group.add_argument("--compilation-config", "-O",
                        **vllm_kwargs["compilation_config"])
vllm_group.add_argument("--additional-config",
                        **vllm_kwargs["additional_config"])
# Other arguments
parser.add_argument("--speculative-config", type=json.loads)

This fix ensures --speculative-config remains as a dictionary, which is what the downstream code expects.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[yyzxw]

I am happy to fix it

[hmellor]

Just FYI for anyone that reads this thread. get_kwargs is quite smart, if the field is type hinted as a dataclass, then the value for type in the dictionary returned by get_kwargs is a pydantic.TypeAdapter(YourClass).validate_json which does do proper json parsing (and type validation via pydantic).

In this case we already delayed the type validation by overriding this returned value to json.loads:

vllm/vllm/engine/arg_utils.py

Lines 914 to 919 in 273690a

	# We construct SpeculativeConfig using fields from other configs in
	# create_engine_config. So we set the type to a JSON string here to
	# delay the Pydantic validation that comes with SpeculativeConfig.
	vllm_kwargs["speculative_config"]["type"] = optional_type(json.loads)
	vllm_group.add_argument("--speculative-config",
	**vllm_kwargs["speculative_config"])