mlx-serve: native Zig LLM server for Apple Silicon - OpenAI + Anthropic + Ollama APIs, no Python

[ddalcu]

Hi all - I've been building mlx-serve, a native Zig server that runs MLX-format models on Apple Silicon and exposes OpenAI-compatible, Anthropic-compatible, and Ollama-compatible HTTP APIs from a single binary.

Why Zig?

No Python runtime, no conda, no virtualenv. One binary you can drop anywhere:

brew tap ddalcu/mlx-serve https://github.com/ddalcu/mlx-serve
brew install --cask mlx-core   # GUI menu bar app
brew install mlx-serve          # CLI server only

What it does

• Full OpenAI API: /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/images/generations, /v1/audio/speech, /v1/video/generations
• • Anthropic Messages API: drop-in for Claude Code - streaming, tool calls, thinking blocks all work locally
• • Ollama drop-in: /api/chat, /api/generate, /api/embed - works with Raycast, Open WebUI, Obsidian, ollama-python
• • Media generation: FLUX.2 (image), LTX-Video (video), Qwen3-TTS (audio) in the same process
• • Agent Sandbox: isolated Linux VM via Virtualization.framework with live port forwarding to localhost
• • Spec decode: PLD + Gemma 4 drafter + Qwen3.6 native MTP head
• • KV cache quant: 4-bit, 8-bit, TurboQuant
• • macOS menu-bar app: chat, agent mode, Control+Space quick launcher, voice cloning

Performance

35%+ faster decode than LM Studio on Gemma 4 E4B 4-bit. Prefill matches mlx-lm (both are MLX-kernel-bound at the memory wall).

Links

• Website: https://mlxserve.com
• • Run Claude Code locally: https://mlxserve.com/claude-code-local/
• • vs LM Studio: https://mlxserve.com/lm-studio-alternative/
• • vs Ollama: https://mlxserve.com/ollama-alternative/
    Happy to answer questions - especially interested in feedback from anyone using mlx-lm directly on whether the API surface covers your use cases.