[Doc]: Gemma 4 assistant speculative decoding docs do not match actual behavior on vLLM 0.20.1

[mohyeon]

📚 The doc issue

Environment
vLLM version: 0.20.1
Launch method: vllm serve
GPU: single GPU via CUDA_VISIBLE_DEVICES=0
Model under test: gemma-4-E2B-it
Speculative config: assistant model via google/gemma-4-E2B-it-assistant

Relevant docs currently show Gemma 4 assistant-model speculative decoding examples, and the Gemma 4 recipe presents this as a supported usage path.

Reproduction
I tried to run Gemma 4 assistant speculative decoding on vllm==0.20.1 with the following command:

CUDA_VISIBLE_DEVICES=0 vllm serve google/gemma-4-E2B-it
--host 0.0.0.0
--port 8002
--tensor-parallel-size 1
--max-model-len 8192
--speculative-config '{"model": "google/gemma-4-E2B-it-assistant", "num_speculative_tokens": 4}'

The attached runtime log shows the full startup sequence and failure during engine initialization.

Expected
Based on the current Gemma 4 documentation, I expected the documented assistant-model speculative decoding path to start successfully, or at minimum for the documentation to clearly state that this path is not supported on the latest stable PyPI release.

Actual
Engine initialization fails with the following error:

NotImplementedError: Speculative Decoding with draft models or parallel drafting does not support multimodal models yet

The log shows that vLLM resolves the Gemma 4 target model, initializes speculative decoding, and then aborts because the draft-model speculative path rejects multimodal models.

More specifically, the failure occurs after the engine is created with:

speculative_config=SpeculativeConfig(method='draft_model', model='google/gemma-4-E2B-it-assistant', num_spec_tokens=4)

and then terminates in llm_base_proposer.py with the multimodal restriction.

Why this is confusing
This is especially confusing because I am already using speculative decoding successfully in production with the following Qwen3.5-4B configuration:

CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen3.5-4B
--served-model-name Qwen3.5-4B
--host 0.0.0.0
--port 8002
--tensor-parallel-size 1
--dtype bfloat16
--max-model-len 131072
--block-size 16
--gpu-memory-utilization 0.9
--max-num-seqs 2
--max-num-batched-tokens 4096
--enable-prefix-caching
--prefix-caching-hash-algo sha256
--speculative-config '{"model":"Qwen/Qwen3.5-4B","num_speculative_tokens":2}'

From a user perspective, this makes the current behavior look inconsistent. The issue is not that “speculative decoding is generally unavailable,” but that the documented Gemma 4 assistant path appears available while failing on the latest stable version with a multimodal restriction. The speculative decoding docs also distinguish between draft-model speculation and other speculation paths, which makes this difference important to document clearly.

Request
Please clarify the following:

1. Is Gemma 4 assistant speculative decoding officially supported on stable vllm==0.20.1?
2. If not, can the Gemma 4 recipe page explicitly warn that the documented command does not work on the latest stable PyPI release?
3. If support currently requires nightly or a specific Docker image, can you provide a minimal reproducible example for that exact supported environment?
4. Can you document clearly how the Gemma 4 assistant speculative path differs from currently working speculative configurations that users may already be running?

At the moment, the main issue is not only support status, but also that the current documentation and actual stable-runtime behavior do not line up clearly enough for users.

Suggest a potential alternative/fix

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[nikhilesh-csa]

Draft models are not supported with Gemma 4 as of right now. However, you can use eagle3 for spec method and it should work.

--speculative-config "{\"method\": \"eagle3\", \"model\": \"RedHatAI/gemma-4-31B-it-speculator.eagle3\", \"num_speculative_tokens\": 3 }" \

something like that

[ChandanShastri]

I can confirm that we are seeing the same behaviour with Gemma-4 26B model with Gemma assistant Draft Model.

vLLM version: 0.20.1
Model: google/gemma-4-26B-A4B-it
Draft Model: google/gemma-4-26B-A4B-it-assistant

speculative-config '{"model": "google/gemma-4-26B-A4B-it-assistant", "num_speculative_tokens": 4}'

Error:
NotImplementedError: Speculative Decoding with draft models or parallel drafting does not support multimodal models yet

[ChandanShastri]

@DarkLight1337

After using the updated config from that Pull request:

--speculative-config '{"method":"mtp","model":"google/gemma-4-26B-A4B-it-assistant","num_speculative_tokens":1}'

Error:
NotImplementedError: Unsupported speculative method: 'mtp'

vLLM v0.20.1

[DarkLight1337]

I think you need to install vLLM nightly since v0.20.1 (even v0.20.2) doesn't include #41745

[nikhilesh-csa]

Just tried this on the nightly build, it does not still work. @DarkLight1337
tried it with MTP for the gemma 31B dense model with the https://huggingface.co/google/gemma-4-31B-it-assistant assistant as MTP model.

tied with both MTP and draft_model. MTP says not implemented, draft does not work as expected.

any help would be appreciated @DarkLight1337

the docs as per @AbhiOnGithub edits, should work with MTP but it does not. Don't know the documentation is future proofing or just wrong? Would appreciate any help/comments

[nikhilesh-csa]

@ChandanShastri did you get it to work?

[DarkLight1337]

Could you show your command and error message?

[shencan]

vllm --version

0.20.2+cu132

vllm cli

vllm serve google/gemma-4-31B-it
--host 0.0.0.0
--port 8000
--max-model-len 42768
--max-num-batched-tokens 8192
--gpu-memory-utilization 0.95
--limit-mm-per-prompt '{"image": 0}'
--enable-prefix-caching
--async-scheduling
--enable-prompt-tokens-details
--enable-auto-tool-choice
--tool-call-parser gemma4
--reasoning-parser gemma4
--default-chat-template-kwargs '{"enable_thinking": false}'
--speculative-config '{"model": "google/gemma-4-31B-it-assistant", "num_speculative_tokens": 4}'

the same error NotImplementedError: Speculative Decoding with draft models or parallel drafting does not support multimodal models yet

[nikhilesh-csa]

CMD sh -c 'vllm serve google/gemma-4-31B-it \
     --port 8082 \
     --tensor-parallel-size ${TENSOR_PARALLEL_SIZE:-4} \
     --max-model-len ${MAX_MODEL_LEN:-15388} \
     --quantization fp8 \
     #--enforce-eager \
     --enable-chunked-prefill \
     --enable-prefix-caching \
     --language-model-only \
     #--moe-backend marlin \
     --enable-auto-tool-choice \
     --tool-call-parser gemma4 \
     --reasoning-parser gemma4 \
     --gpu-memory-utilization ${GPU_MEM_UTIL:-0.75} \
     --max-num-seqs 50 \
     --download-dir /models \
     --kv-cache-dtype fp8 \
     #--speculative-config "{\"method\": \"mtp\", \"model\":\"google/gemma-4-31B-it-assistant\", \"num_speculative_tokens\": 3}" \
     #--default-chat-template-kwargs "{\"enable_thinking\": true}" \
     --trust-remote-code \
     --api-key ${LLM_INTERNAL_TOKEN:-changeme}' 

running it inisde docker with these. changed method from mtp to draft_model and both dont work. mtp gives a notimplemented error, draft model gives ValueError: Unrecognized configuration class <class 'transformers.models.gemma4_assistant.configuration_gemma4_assistant.Gemma4AssistantConfig'> for this kind of AutoModel: AutoModel.
@DarkLight1337

[DarkLight1337]

Maybe you need to further update your transformers version. Have you tried nightly docker?