vLLM Metal Model Support

[WindChimeRan]

Tracking issue for community testing of models on vllm-metal. If you have an Apple Silicon Mac, we'd love your help expanding verified model coverage.

How to help

Pick a model, run it with vllm-metal, and report back. Good starting points are models not tested in the supported models table, but any text-only LLM on Hugging Face is welcome.

Workflow

It works? Open a PR updating https://github.com/vllm-project/vllm-metal/blob/main/docs/supported_models.md

• Add or update the model's row (status, attention kernel, notes)
• Link back to this issue in the PR description

It doesn't work? File a new issue with:

• Model (HF repo ID)
• Hardware (chip + unified memory, e.g. M3 Max / 64 GB)
• vllm-metal commit or version
• The exact command you ran
• Error message or unexpected output
• Relevant logs

Pass bar for a 🟡 → ✅ / 🔵 status bump

At minimum:

• Loads without errors
• Generates coherent output on a standard prompt
• Hardware and environment documented in the PR

Numerical parity checks or throughput measurements are welcome but not required for an initial status update.

Note: vllm-metal currently focuses on text-only LLMs. Multi-modal (vision / audio) support is not yet in scope.

Resources

• Contributor guide: https://docs.vllm.ai/projects/vllm-metal/en/latest/CONTRIBUTING/
• Supported models: https://github.com/vllm-project/vllm-metal/blob/main/docs/supported_models.md

Questions? Drop a comment below. PRs welcome!

[komoike-oss28-ui]

Hi, I'd like to help test models on vllm-metal!

My environment:

• Apple M1 MacBook Air / 8 GB unified memory
• Planning to test: Llama3 and/or Gemma3 (starting with 1B 4-bit quantized variants from mlx-community)

I'll report back with results.

[komoike-oss28-ui]

Hi, I tried testing Llama-3.2-1B-Instruct-4bit on my M1 MacBook Air (8GB),
but encountered a build error with the Metal kernel on macOS 26.

Error:
RuntimeError: Failed to load Metal kernel: Failed to build paged_ops extension:
no member named 'get_command_encoder' in 'mlx::core::metal::Device'
no member named 'add_temporary' in 'mlx::core::metal::Device'

Environment:

• Hardware: Apple M1 / 8GB unified memory
• macOS: 26.3.1 (beta)
• vllm-metal: 0.19.1
• Command: vllm serve mlx-community/Llama-3.2-1B-Instruct-4bit --max-model-len 8192

This appears to be an MLX API compatibility issue with macOS 26.

[ricky-chaoju]

Hi, I tried testing Llama-3.2-1B-Instruct-4bit on my M1 MacBook Air (8GB), but encountered a build error with the Metal kernel on macOS 26.

Error: RuntimeError: Failed to load Metal kernel: Failed to build paged_ops extension: no member named 'get_command_encoder' in 'mlx::core::metal::Device' no member named 'add_temporary' in 'mlx::core::metal::Device'

Environment:

• Hardware: Apple M1 / 8GB unified memory
• macOS: 26.3.1 (beta)
• vllm-metal: 0.19.1
• Command: vllm serve mlx-community/Llama-3.2-1B-Instruct-4bit --max-model-len 8192

This appears to be an MLX API compatibility issue with macOS 26.

Hi @komoike-oss28-ui, this is an mlx version skew — paged_ops.cpp used private APIs that moved off Device in mlx 0.31.2. #291 adds a shim and pins mlx<0.32.0. Please reinstall from latest main.

[komoike-oss28-ui]

Tested mlx-community/Llama-3.2-1B-Instruct-4bit on M1 MacBook Air (8GB).

Result: ✅ Working

Environment:

• Hardware: Apple M1 / 8GB unified memory
• macOS: 26.3.1
• vllm-metal: 0.19.1 (latest main)
• Command: vllm serve mlx-community/Llama-3.2-1B-Instruct-4bit --max-model-len 8192

Note: max-model-len must be set to 8192 or less due to memory constraints on 8GB devices.

[komoike-oss28-ui]

Tested on M1 MacBook Air (8GB unified memory), macOS 26.3.1, vllm-metal 0.19.1.

mlx-community/gemma-3-1b-it-4bit ⚠️

• Command: vllm serve mlx-community/gemma-3-1b-it-4bit --max-model-len 8192
• Result: Server starts and responds, but output quality is poor (repetitive/incoherent).
• Note: Heavy memory pressure observed on 8GB devices.

[WindChimeRan]

mistakenly closed this issue. Now re-open

[zhaiGOGO]

Hi, I tested the Gemma 3 MLX checkpoints locally and opened a docs PR with the verification details:

• mlx-community/gemma-3-1b-it-qat-4bit
• mlx-community/gemma-3-4b-it-4bit for text-only generation

PR: #302

Note: the 4B checkpoint resolves as Gemma3ForConditionalGeneration; vllm-metal loads it in text-only mode and bypasses
image inputs.