[Bug]: vllm/vllm-openai:latest fails to start Gemma4 with Transformers 5.15.0

[yasu-oh]

Your current environment

Docker image: vllm/vllm-openai:latest
vLLM: 0.27.0
Transformers: 5.15.0
Model: yasu-oh/gemma-4-31B-it-qat-NVFP4
Tensor parallel size: 2
KV cache dtype: fp8

The Transformers version in the image can be confirmed with:

docker run --rm \
  --entrypoint python3 \
  vllm/vllm-openai:latest \
  -c 'import vllm, transformers; print("vLLM:", vllm.__version__); print("Transformers:", transformers.__version__)'

Output:

vLLM: 0.27.0
Transformers: 5.15.0

🐛 Describe the bug

The current official release image vllm/vllm-openai:latest contains:

vLLM: 0.27.0
Transformers: 5.15.0

When serving:

yasu-oh/gemma-4-31B-it-qat-NVFP4

vLLM fails during startup with:

transformers.integrations.heterogeneity.configuration_utils.AmbiguousGlobalPerLayerAttributeError:
'head_dim' is a per-layer attribute and may vary across layers.
Access it via config.per_layer_config[i].head_dim.

The exception occurs in:

vllm/transformers_utils/model_arch_config_convertor.py

while accessing the Gemma4 head_dim.

Workaround

Downgrading only Transformers to 5.14.1 while keeping the same vllm/vllm-openai:latest base image and serving configuration resolves the issue.

FROM vllm/vllm-openai:latest

RUN /usr/bin/python3 -m pip install --no-cache-dir 'transformers==5.14.1'

With Transformers 5.14.1:

• the same model loads successfully,
• the vLLM engine initializes successfully,
• the API server starts successfully,
• /v1/chat/completions returns HTTP 200,
• 32 concurrent requests run successfully.

Related PRs

• Fix Gemma 4 for upcoming Transformers version #49797 ? Fix Gemma 4 for upcoming Transformers version
• [ROCm][CI] Handle transformers 5.15 gemma4 schema #49959 ? Handle Transformers 5.15 Gemma4 schema

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[quivent]

@yasu-oh I wrote the fix and connected it to my bug report two before yours. I should probably make that more visible.

[yasu-oh]

@quivent
Thanks for pointing me to #51737.
It looks related since both issues involve Gemma 4's heterogeneous per-layer head_dim, but I think the failure occurs at a different stage in my case.
#51737 fails during MTP QKV parameter loading, while #51744 fails earlier during ModelConfig initialization in Gemma4ModelArchConfigConvertor.get_head_size(), before any weights are loaded.
My traceback appears to match the Transformers 5.15 heterogeneous-config compatibility issue addressed by #49797 more directly.

[quivent]

You would laugh if you looked at my screen: "Missing model_arch_config_convertor.py fix. None of the 8 options are directly applicable." - I'm short hands today, we just found it and are updating the fix

@yasu-oh

[quivent]

Ok updated to include the model arch fix.

[yasu-oh]

@lucianommartins Looping you in from the Gemma team for visibility.

This appears to be triggered by the heterogeneous config changes for Gemma 4 in Transformers 5.15.0. With the current vLLM release image, initialization fails when vLLM accesses the global head_dim, while downgrading Transformers to 5.14.1 works with the exact same model and configuration.

#49797 looks like the corresponding vLLM-side adaptation, so this may not be a Gemma issue itself, but I’d appreciate any insight from the Gemma side on whether the current per-layer head_dim behavior in Transformers 5.15 is as intended, or if there are any additional compatibility considerations vLLM should account for.

[quivent]

@yasu-oh I have to say I had a really nice laugh when I saw this ticket - kudos for finding it. FYI to all of you, I'm a huge Gemma 4 advocate, if you trace back the trained MTP heads and the optimistic prediction, Jeff & co integrated them from optimization work I posted actually for Qwen's MTP, I had no idea until recently (I was shocked( but I exclusively work with Gemma 4 and have some optimizations we've done so if anyone is interested just drop a comment.

[alex-chaiko]

This issue also repeats for CPU:

docker vllm/vllm-openai-cpu:v0.27.0 run google/gemma-4-26B-A4B-it
. . .
transformers.integrations.heterogeneity.configuration_utils.AmbiguousGlobalPerLayerAttributeError: 'head_dim' is a per-layer attribute and may vary across layers. Access it via the individual layer configs instead (e.g. config.per_layer_config[i].head_dim). To read the global config value from config.head_dim anyway, set allow_global_per_layer_attribute_access to True on the config. Warning: only do this if the caller can safely handle heterogeneous configs; code that assumes a homogeneous model may use the global value incorrectly.

@louie-tsai @bigPYJ1151

[quivent]

backing @alex-chaiko up on this since I wrote that knowing it would not be spot on with the architecture, @alex-chaiko, I have a PR in, if you guys can find someone to merge. I don't do these fixes that often and am a bit laissez faire when it comes to pushing for integration.

#51757

[aaronsmulktis]

@yasu-oh I have to say I had a really nice laugh when I saw this ticket - kudos for finding it. FYI to all of you, I'm a huge Gemma 4 advocate, if you trace back the trained MTP heads and the optimistic prediction, Jeff & co integrated them from optimization work I posted actually for Qwen's MTP, I had no idea until recently (I was shocked( but I exclusively work with Gemma 4 and have some optimizations we've done so if anyone is interested just drop a comment.

I'd be interested in seeing those )

[bigPYJ1151]

cc @hmellor @Isotr0py

[quivent]

@aaronsmulktis thats so funny because I just decided to open this up (I stopped checking after the merge) to ask a very real question - does anyone want to learn how to give Gemma persistent memory? It's not a hypothetical. I want to see who takes me up on this and how far it goes, because Anthropic may release it soon and we can have the headstart.

Email me directly if you want a faster reply: jbenjaminkr@gmail.com

This is out to anyone who takes the time to read it.

[hnt2601]

same error

[hmellor]

This is already fixed on main by #49797, which is not in 0.27.0

[harlenalvarez]

Same error here running google/gemma-4-12B-it-qat-w4a16-ct. I had to manually downgrade transformers to version 5.14.1 and that works.