Gemma 4 Audio Understanding on vLLM (Docker)

Deploys Google's Gemma 4 E4B-IT model on vLLM with audio understanding support, served as an OpenAI-compatible API.

Why E4B?

Audio understanding in Gemma 4 is only supported on the E2B and E4B variants (not the 31B). E4B is the larger of the two and fits comfortably on a single GPU with 24GB+ VRAM.

Hardware

Tested on:

• 1x NVIDIA L40S (46GB VRAM)
• Any GPU with 24GB+ VRAM should work

Project Structure

.
├── Dockerfile           # Extends vllm/vllm-openai:gemma4 with audio deps
├── docker-compose.yml   # Deployable vLLM server config
├── test_audio.py        # Test client — sends audio files to the API
├── setup_test_env.sh    # Creates Python venv for the test client
├── sample_audio.mp3     # Sample audio file for testing
└── sample_audio.wav     # Sample audio file for testing

Setup

1. Build and start the server

docker compose up -d --build

First run will download the model weights (~8GB) to ~/.cache/huggingface. Subsequent starts use the cache and take ~3 minutes (CUDA graph compilation).

2. Wait for the server to be ready

docker logs -f gemma4-vllm

Look for Application startup complete. — the API is then live on http://localhost:8000.

3. Set up the test client

bash setup_test_env.sh

This creates a Python venv with the openai package installed.

4. Test audio understanding

source .venv/bin/activate
python test_audio.py sample_audio.mp3

Pass any WAV or MP3 file as an argument.

What the Dockerfile does

The base vllm/vllm-openai:gemma4 image ships without audio dependencies. The Dockerfile adds them:

FROM vllm/vllm-openai:gemma4

RUN pip install --no-deps av resampy scipy soundfile "mistral_common[audio]"

--no-deps is important — a bare pip install "vllm[audio]" would upgrade transformers and break the pinned version (5.5.0) that Gemma 4 requires.

Server configuration

Key flags in docker-compose.yml:

Flag	Value	Purpose
--max-model-len	16384	Max context length (up to 131072, but uses more VRAM)
--gpu-memory-utilization	0.90	Fraction of GPU memory for KV cache
--limit-mm-per-prompt	{"image": 4, "audio": 2}	Max multimodal inputs per request

API usage

The server exposes an OpenAI-compatible API. Audio is sent as base64-encoded data in the input_audio content block:

from openai import OpenAI
import base64

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

with open("audio.mp3", "rb") as f:
    audio_b64 = base64.b64encode(f.read()).decode("utf-8")

response = client.chat.completions.create(
    model="google/gemma-4-E4B-it",
    messages=[{
        "role": "user",
        "content": [
            {
                "type": "input_audio",
                "input_audio": {"data": audio_b64, "format": "mp3"},
            },
            {
                "type": "text",
                "text": "Transcribe this audio.",
            },
        ],
    }],
    max_tokens=512,
)
print(response.choices[0].message.content)

curl example

AUDIO_B64=$(base64 -w 0 audio.mp3)

curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-4-E4B-it",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "input_audio", "input_audio": {"data": "'"$AUDIO_B64"'", "format": "mp3"}},
        {"type": "text", "text": "What do you hear in this audio?"}
      ]
    }],
    "max_tokens": 512
  }'

Gotchas encountered during setup

1. --limit-mm-per-prompt expects JSON — older docs show image=4,audio=2 but current vLLM requires {"image": 4, "audio": 2}.
2. --model is now positional — vllm serve google/gemma-4-E4B-it, not vllm serve --model google/gemma-4-E4B-it.
3. Audio deps missing from base image — vllm/vllm-openai:gemma4 does not include vllm[audio] extras. Must be added manually.
4. Don't upgrade transformers — the image pins transformers==5.5.0 for Gemma 4 support. A naive pip install "vllm[audio]" pulls a newer version that breaks the gemma4 architecture registration.

Managing the server

# Start
docker compose up -d

# Stop
docker compose down

# Rebuild after Dockerfile changes
docker compose up -d --build

# View logs
docker logs -f gemma4-vllm

# Check model is loaded
curl -s http://localhost:8000/v1/models | python3 -m json.tool