[BUG]: Gemma 4 is not supported on Dynamo vLLM backend ≤1.1.0-dev.1

[ritazh]

Describe the Bug

Gemma 4 (google/gemma-4-31b-it) cannot be served on Dynamo ≤1.1.0-dev.1 because the bundled vLLM 0.17.1 lacks a native gemma4 model implementation. It falls back to the generic TransformersMultiModalForCausalLM which fails during weight loading with ValueError: There is no module or parameter named 'model.language_model.layers.0.layer_scalar' in TransformersMultiModalForCausalLM.

Upgrading to transformers 5.x (which adds gemma4 architecture recognition) alone is insufficient, and upgrading vLLM in-container to ≥0.19 breaks the Dynamo integration layer ModuleNotFoundError: No module named 'vllm.inputs.data'.

A Dynamo release built against vLLM ≥0.19 (which adds native Gemma 4 support) is needed to unblock this.

Steps to Reproduce

Deployed this

apiVersion: nvidia.com/v1alpha1
kind: DynamoGraphDeployment
metadata:
  name: gemma4-31b-agg
spec:
  backendFramework: vllm
  pvcs:
    - name: pvc-lustre
      create: false
  services:
    Frontend:
      envFromSecret: hf-token-secret
      componentType: frontend
      replicas: 1
      volumeMounts:
        - name: pvc-lustre
          mountPoint: /model-cache
      extraPodSpec:
        mainContainer:
          startupProbe:
            httpGet:
              path: /health
              port: 8000
            periodSeconds: 10
            timeoutSeconds: 1800
            failureThreshold: 60
          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.1.0-dev.1
          command:
            - /bin/sh
            - -c
          args:
            - python3 -m dynamo.frontend --router-mode kv --no-kv-events --http-port 8000
      envs:
        - name: HF_HOME
          value: /model-cache
    VllmDecodeWorker:
      envFromSecret: hf-token-secret
      componentType: worker
      replicas: 1
      resources:
        limits:
          gpu: "1"
        requests:
          gpu: "1"
      volumeMounts:
        - mountPoint: /model-cache
          name: pvc-lustre
          useAsCompilationCache: false
      extraPodSpec:
        mainContainer:
          startupProbe:
            httpGet:
              path: /health
              port: 9090
            periodSeconds: 10
            timeoutSeconds: 10
            failureThreshold: 120
          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.1.0-dev.1
          env:
            - name: HF_HOME
              value: /model-cache
            # Workaround for vllm/vllm#35772: FlashInfer allreduce can hang during
            # CUDA graph capture with TP>1. Fixed in vllm/vllm#35793 (default changed
            # to trtllm). The 1.0 container ships vLLM 0.16.0, so set explicitly.
            - name: VLLM_FLASHINFER_ALLREDUCE_BACKEND
              value: "trtllm"
          workingDir: /workspace/examples/backends/vllm
          command:
            - /bin/sh
            - -c
          args:
            - pip install "vllm>=0.19" && pip install --no-deps "transformers>=5" "huggingface_hub>=0.37" && python3 -m dynamo.vllm --model google/gemma-4-31b-it --tensor-parallel-size 1 --max-model-len 8192 --enable-multimodal --enforce-eager

Expected Behavior

Model deployed and can serve requests

Actual Behavior

vllmdecodeworker pod CrashLoopBackOff

Environment

• k8s version: v1.33.6
• 2× H100 80GB

Additional Context

No response

Screenshots

No response

[ritazh]

Confirming vllm/vllm-openai:v0.19.0 + transformers==5.6.0.dev0 works for google/gemma-4-31b-it

# Ray Head + vLLM server (Node 1)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-gemma-4-31b-head
  labels:
    app: vllm-gemma-4-31b
    role: head
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-gemma-4-31b
      role: head
  template:
    metadata:
      labels:
        app: vllm-gemma-4-31b
        role: head
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.19.0
          command: ["/bin/bash", "-c"]
          args:
            - |
              set -e
              NODE_IP=$(hostname -i)
              echo "Head node IP: $NODE_IP"
              pip install ray[default] https://github.com/huggingface/transformers/archive/refs/heads/main.zip
              RAY_BIND_ADDRESS=0.0.0.0 ray start --head \
                --port=6379 \
                --node-ip-address=$NODE_IP \
                --dashboard-host=0.0.0.0
              echo "Verifying Ray GCS is listening..."
              python3 -c "
              import socket, sys
              s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
              try:
                  s.connect(('0.0.0.0', 6379))
                  print('GCS port 6379 is open')
              except Exception as e:
                  print(f'GCS port 6379 NOT open: {e}')
                  sys.exit(1)
              finally:
                  s.close()
              "
              python3 -c "
              import ray, time
              ray.init(address='auto')
              while True:
                  resources = ray.cluster_resources()
                  gpus = resources.get('GPU', 0)
                  print(f'Waiting for 2 GPUs in Ray cluster... currently {gpus}')
                  if gpus >= 2:
                      break
                  time.sleep(5)
              print('All GPUs available, starting vLLM...')
              "
              vllm serve google/gemma-4-31b-it \
                --served-model-name google/gemma-4-31b-it \
                --tensor-parallel-size 2 \
                --distributed-executor-backend ray \
                --max-model-len 8192 \
                --gpu-memory-utilization 0.90 \
                --port 8000
          ports:
            - containerPort: 8000
              name: http
            - containerPort: 6379
              name: ray-gcs
          env:
            - name: HF_HOME
              value: /model-cache
            - name: HF_HUB_OFFLINE
              value: "1"
          volumeMounts:
            - name: model-cache
              mountPath: /model-cache
            - name: dshm
              mountPath: /dev/shm
          resources:
            limits:
              nvidia.com/gpu: "1"
            requests:
              nvidia.com/gpu: "1"
          startupProbe:
            httpGet:
              path: /health
              port: 8000
            periodSeconds: 10
            timeoutSeconds: 10
            failureThreshold: 180
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            periodSeconds: 10
            timeoutSeconds: 5
      volumes:
        - name: model-cache
          persistentVolumeClaim:
            claimName: pvc-lustre
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: 16Gi
---
# ClusterIP service for Ray worker -> head discovery
apiVersion: v1
kind: Service
metadata:
  name: vllm-gemma-4-31b-head
spec:
  selector:
    app: vllm-gemma-4-31b
    role: head
  ports:
    - port: 6379
      targetPort: 6379
      name: ray-gcs
  type: ClusterIP
  publishNotReadyAddresses: true
---
# Ray Worker (Node 2)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-gemma-4-31b-worker
  labels:
    app: vllm-gemma-4-31b
    role: worker
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-gemma-4-31b
      role: worker
  template:
    metadata:
      labels:
        app: vllm-gemma-4-31b
        role: worker
    spec:
      containers:
        - name: ray-worker
          image: vllm/vllm-openai:v0.19.0
          command: ["/bin/bash", "-c"]
          args:
            - |
              pip install ray[default] https://github.com/huggingface/transformers/archive/refs/heads/main.zip && \
              ray start --address=vllm-gemma-4-31b-head:6379 --block
          env:
            - name: HF_HOME
              value: /model-cache
            - name: HF_HUB_OFFLINE
              value: "1"
          volumeMounts:
            - name: model-cache
              mountPath: /model-cache
            - name: dshm
              mountPath: /dev/shm
          resources:
            limits:
              nvidia.com/gpu: "1"
            requests:
              nvidia.com/gpu: "1"
      volumes:
        - name: model-cache
          persistentVolumeClaim:
            claimName: pvc-lustre
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: 16Gi
---
# External-facing service for API access
apiVersion: v1
kind: Service
metadata:
  name: vllm-gemma-4-31b
spec:
  selector:
    app: vllm-gemma-4-31b
    role: head
  ports:
    - port: 8000
      targetPort: 8000
      protocol: TCP
  type: ClusterIP

[nealvaidya]

Thanks for reporting @ritazh! It looks like there are also some issues with gemma 4 in vllm 0.19, and so the vllm team is working on a vllm 0.19.1 release specifically for gemma 4 support: https://vllm-dev.slack.com/archives/C08U97ZRC0J/p1775588919991689

Once 0.19.1 releases, we'll incorporate it into dynamo and test it out.

[filippo82]

0.19.1 is out: https://github.com/vllm-project/vllm/releases

[itay]

0.19.1 is out: https://github.com/vllm-project/vllm/releases

This will be part of the upcoming 1.1.0 release

[GuanLuo]

@ritazh have you tried 1.1.0 release to see if the model is supported out of the box?

[azzaea]

I can reproduce a related failure with nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.1.1.

Environment:

• Kubernetes deployment using DynamoGraphDeployment
• image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.1.1
• model: google/gemma-4-31B-it
• production node supports AVX2/AVX512

Observations:

• vLLM package in the image contains Gemma 4 references:
  • vllm/reasoning/__init__.py includes gemma4
  • vllm/tool_parsers/__init__.py includes gemma4
  • vllm/model_executor/models/registry.py includes Gemma4ForCausalLM / Gemma4ForConditionalGeneration
• Dynamo wrapper does not accept:
  • --dyn-reasoning-parser gemma4
  • --dyn-tool-call-parser gemma4
• It also does not accept vLLM flags:
  • --tool-call-parser gemma4
  • --enable-auto-tool-choice
• It also does not have the examples/chat_templates/gemma4_tool.jinja file
• With only --reasoning-parser gemma4, startup reaches model loading but fails with:

pydantic_core._pydantic_core.ValidationError: 1 validation error for ModelConfig
Value error, The checkpoint you are trying to load has model type `gemma4` but Transformers does not recognize this architecture.

This suggests the 1.1.1 runtime image still lacks the required Transformers/Gemma 4 support, despite the current docs mentioning Gemma 4 parsing support.

[GuanLuo]

@azzaea thanks for verifying, will take a closer look as vLLM's doc does highlight its upgrade on Transformers v5. @keivenchang can you chime in on the tool calling support? I am not familiar with the Dynamo side support and whether we have framework fallback in 1.1.1

[azzaea]

@GuanLuo - thank you.

One clarification: I think there are two separate issues here.

1. the Gemma 4 parser flags shown in the Dynamo docs are not accepted by the nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.1.1 image. For example:

   • --dyn-reasoning-parser gemma4: fails because gemma4 is not one of the accepted Dynamo parser choices. Also, the vLLM-style flags:
   • --tool-call-parser gemma4 --enable-auto-tool-choice: are not recognized by the Dynamo wrapper.

2. even if I remove the tool-calling flags and keep only basic model loading, google/gemma-4-31B-it still fails with: Value error, The checkpoint you are trying to load has model type gemma4 but Transformers does not recognize this architecture.

So from my side, the immediate problem looks broader than tool calling: the released 1.1.1 runtime image seems inconsistent with the current Gemma 4 documentation/support expectations.

[GuanLuo]

@azzaea I just confirmed that nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.1.1 doesn't have transformers 5 (has 4.57.6) and thus failed to serve gemma 4. We are in the process of getting 1.2.0 ready and release, which I have confirmed that:

• Gemma 4 can be served (using our example script MAX_MODEL_LEN=32768 ./agg.sh --model google/gemma-4-E2B-it)
• reasoning parser and tool call parser options are recognized (--dyn-reasoning-parser gemma4 --dyn-tool-call-parser gemma4) not that the tool call argument is different from yours
• Reasoning can't verify as it appears that the model doesn't spit out thinking content (didn't observe any thinking token with parser off)
• Verified that tool call are being parsed properly:

{"id":"chatcmpl-9bd2759a-6bae-428f-bcde-db40f5e1ff9a","choices":[{"index":0,"message":{"content":null,"tool_calls":[{"id":"call-7f357f31-1932-498e-95cb-d27c84f96e69","type":"function","function":{"name":"get_weather","arguments":"{\"city\":\"Tokyo\"}"}}],"role":"assistant","reasoning_content":null},"finish_reason":"tool_calls","logprobs":null}],"created":1779303006,"model":"google/gemma-4-E2B-it","service_tier":null,"system_fingerprint":null,"object":"chat.completion","usage":{"prompt_tokens":85,"completion_tokens":15,"total_tokens":100}}

That being said, I would suggest to try Dynamo 1.2.0 when it is released and follow up with new issue if any.

[azzaea]

Perfect! Thank you for looking @GuanLuo - will try out the next release as it comes out. Any ETA though?

[GuanLuo]

I don't have the exact timeline, likely in the next week or two. If you can build from source, you can either try main or the release branch (1.2.0)