[Bug]: Gemma 4 E4B extremely slow on v0.19.0 forced TRITON_ATTN fallback yields ~9 tok/s on RTX 4090 (vs ~100+ tok/s for comparable Llama 3B)

[CunXin1]

Your current environment

PyTorch version : 2.10.0+cu129
Python version : 3.12.13 (64-bit runtime)
CUDA runtime version : 12.9.86
vLLM Version : 0.19.0

[pip3] torch==2.10.0+cu129
[pip3] transformers==5.5.0
[pip3] triton==3.6.0
[pip3] flashinfer-python==0.6.6

🐛 Describe the bug

Gemma 4 E4B (google/gemma-4-e4b-it, 4.5B effective parameters) generates at only ~9 tokens/s on an RTX 4090 with vLLM v0.19.0. For comparison, a similarly-sized Llama 3.2 3B model on the same hardware with the same vLLM version generates at 100+ tokens/s.

The root cause is that Gemma 4's heterogeneous attention head dimensions force vLLM to disable FlashAttention and fall back to a much slower Triton attention kernel. Additionally, custom_ops is set to ['none'], meaning no vLLM-native CUDA kernels are used.

From vLLM server logs during inference:

INFO [config.py:104] Gemma4 model has heterogeneous head dimensions (head_dim=256, global_head_dim=512).
  Forcing TRITON_ATTN backend to prevent mixed-backend numerical divergence.

INFO [cuda.py:274] Using AttentionBackendEnum.TRITON_ATTN backend.

INFO [loggers.py:259] Engine 000: Avg prompt throughput: 1.6 tokens/s,
  Avg generation throughput: 9.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs,
  GPU KV cache usage: 1.9%, Prefix cache hit rate: 94.9%

Expected behavior

A 4.5B parameter model on an RTX 4090 (24GB VRAM, BF16) should generate in the range of 50-100+ tokens/s, comparable to other models of similar size (e.g., Llama 3.2 3B at ~100-200 tok/s on the same hardware).

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[jump-drive]

Hardware: NVIDIA DRIVE AGX Thor (Blackwell SM_110, 128GB unified LPDDR5X)
vLLM: 0.17.0 (custom build against CUDA 13.0)
Model: nvidia/Gemma-4-31B-IT-NVFP4
Result: ~7.6 tok/s generation throughput (single request)

Seeing the same issue. The Gemma4 model has heterogeneous head dimensions (head_dim=256, global_head_dim=512). Forcing TRITON_ATTN backend message is the bottleneck. FlashInfer explicitly rejects the config with head_size not supported.

[859569967]

同样遇到 cuda13.1+移动端5090 最多9/tokens

[pcgeek86]

I'm running Windows 11 with Docker Desktop (WSL2) using the following command, and I'm getting about 35 tokens / sec. My workstation has an RTX 5080 and RTX 5060 Ti 16 GB. The actual vLLM version for the gemma4 container image tag is 0.18.2rc1.dev73+gdb7a17ecc.

Seems to be about what I would expect for my system. You might want to use this for now.

docker run --rm -it `
   --gpus all `
   --ipc=host `
   -p 8000:8000 `
   -v "$HOME\.cache\huggingface:/root/.cache/huggingface" `
   -e "HF_TOKEN=xxxxxxxxxxxxx" `
   -e "CUDA_DEVICE_ORDER=PCI_BUS_ID" `
   -e "CUDA_VISIBLE_DEVICES=0,1" `
   vllm/vllm-openai:gemma4 `
   --gpu-memory-utilization 0.7 `
   --tensor-parallel-size 2 `
   google/gemma-4-E4B-it `
   --dtype auto `
  --enable-auto-tool-choice `
   --tool-call-parser gemma4

[hxypqr]

@pcgeek86 This Docker image works pretty well. I ran a single-threaded gemma4 31b on a Pro6000 and it achieved 24 tokens / sec. With 30 concurrent connections, it reached 400 tokens / sec.

[ecevit2804]

@pcgeek86 This Docker image works pretty well. I ran a single-threaded gemma4 31b on a Pro6000 and it achieved 24 tokens / sec. With 30 concurrent connections, it reached 400 tokens / sec.

which docker image version did you use ?

[fikrikarim]

Any update on this? Retested on an RTX 5090 with a recent vllm/vllm-openai:gemma4-cu130 build. The triton fallback is still being selected.

For what it's worth, single-stream decode on 5090 measures ~130 tok/s for google/gemma-4-E4B-it, noticeably better than the ~9 tok/s originally reported on 4090, but still presumably well below if FlashAttention is enabled.

[Suleman-Elahi]

I kind of have same issue. Ollama version works perfectly and very accurate. For me apart from being slow it's not as accurate as well.

[dchichkov]

I may have the same issue. Getting ~810 tok/s for 8 x H100 (102 tok/s per GPU), when the context size is around 22,000-25,000 tokens, with gemma4-26b4a, with fp8 prefix cache, expert parallel, TRITON backend fallback.

The prefix cache hit rate is at 90%, only 35% used, 128 parallel streams per GPU. This is not reasonable for a 4B MoE, the expectation is >3k output tok/s per H100, around 30 times faster.

[zishraq]

Same issue here on consumer Ada hardware.

Environment

• GPU: RTX 4070 Ti Super (16 GB, Ada / SM 8.9)
• vLLM: 0.22.0
• Model: google/gemma-4-E4B-it, self-quantized to FP8 (compressed-tensors, FP8_DYNAMIC)
• Running under WSL2 on Windows
• Serving a single-user RAG app (one request at a time)

What we're seeing

Single-request generation sits at ~3.5–4 tok/s. The startup log shows the same Forcing TRITON_ATTN backend line everyone else is hitting, and during generation the GPU is pinned at 100% util the whole time — so it's clearly working hard but producing very little.

The thing that finally made it click for us: we ran vllm bench serve with 4 concurrent requests and got ~53 tok/s aggregate (~14 tok/s per stream), which looked fine at first — but a single request is only ~3.8 tok/s. Looking at the per-step timing, each decode step takes roughly the same wall-clock whether it's serving 1 request or 4, so batching just amortizes a big fixed per-step cost. For anything single-stream (a normal chatbot/RAG query) that fixed cost dominates and the speed is rough. Matches what others reported on 4090/5090 here.

What we tried (and what kept happening)

• Turning off thinking, fixing KV cache memory, dropping --enforce-eager so CUDA graphs are on — all helped correctness/startup but didn't move single-stream tok/s.
• VLLM_ATTENTION_BACKEND=FLASHINFER was rejected as an unknown env var on 0.22.0, and the log still forced TRITON_ATTN anyway.
• Confirmed it's not our client/framework: the raw openai client and a LangChain path give identical tok/s, so the bottleneck is entirely server-side.

So nothing on the user side seems to change it. It really does look like the forced Triton fallback for the heterogeneous head dims (256 sliding / 512 global), as the issue describes.

This makes the smaller Gemma 4 models hard to use on consumer GPUs right now. I'm open to run more tests or share logs if that helps.

[dchichkov]

From what I see, with the applied patch from #38891 the accuracy is not affected. The performance is either same or better by a small margin. From digging into the code, Blackwell GPUs also can be affected.

There is another patch apparently, that improves prefill - #38879