Apple Metal and MLX

Apple: Metal and MLX

Measured 2026-09-23 on a rented Mac: Apple M4 (10-core), 16 GB, macOS 26.6, Xcode with the Metal Toolchain 17F109. There are two separate paths.

1. The MLX engine: lemon-mlx-engine (fork bong-water-water-bong/lemon-mlx-engine)

This is a C++ MLX inference engine, the one scripts/install-lemon-mlx.sh in 1bit-MONSTER builds for ROCm. Its CMake builds ROCm off Apple and MLX's default Metal backend on Apple. It pulls MLX from NripeshN/mlx@rocm-support. On the M4 it builds as-is (338/338) once these are present:

• the Metal Toolchain: xcodebuild -downloadComponent MetalToolchain (688 MB; without it, metal is missing at configure);
• Rust, because third_party/fastokens_cpp is built with cargo: rustup, minimal profile;
• cmake and ninja: pip install cmake ninja in a venv, since there is no Homebrew on the rented Mac.

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DMLX_LM_BUILD_TESTS=OFF
cmake --build build -j 10          # -> build/chat, build/server, build/convert, build/diagnose
build/server --port 8765           # OpenAI chat/completions; the model is a Hugging Face id per request

Qwen3-0.6B-4bit (mlx-community), greedy, "What is the capital of France?": both the engine and Apple's mlx-lm answer "The capital of France is Paris."

256-token decode, 2 rounds	tok/s
lemon-mlx-engine (Metal)	150.9 / 150.5
Apple mlx-lm 0.29.1 (mlx 0.29.3)	241.1 / 241.5

The fork decodes at about 62% of upstream MLX on Metal. A likely cause, not yet investigated: its MLX pin is a ROCm-focused branch.

The fork also has an MLX_LM_BUILD_NPU option ("AMD XDNA NPU backend via IRON"), which is not yet looked at.

2. The Metal backend in 1bit-MONSTER (src/backend_metal.mm)

A 625-line backend: MPS GEMV for the projections, with RMS norm, attention and the embedding lookup on the host, and fp16 activations.

It is a prototype.

• There is no RoPE, and the RMS epsilon is fixed at 1e-5.
• forward() and lm_head() are not implemented.
• So it is only correct at position 0.

Test	Result on the M4
Testing/metal_smoke.mm (synthesised tiny model)	PASS, the first time this path has run
Testing/metal_real_gate.mm, fixed (below), SmolLM2-135M	30/30 tokens exact vs torch fp32
same, Llama-3.2-1B	30/30 exact, 9.2 s, 2.9 GB RSS

The gate as committed had three bugs:

• Parser: jlist() stepped past ], so every seed's chain ran into the next ones.
• Printout: printf("%zu", int) printed garbage counts.
• Oracle mismatch: it compared the exporter's single-token, position-0 oracle against a stateful generate(). The fix calls reset() before each step.

Models are exported on a Linux box with torch: Testing/export_metal_model.py <hf_id> <dir> (Llama-style tensors only; no Qwen3 q/k-norm).

Which to use

On Apple, serve through MLX. The MLX engine does full inference with any context length today, and Apple's mlx-lm is faster still. The 1bit-MONSTER Metal backend needs RoPE, a Metal-side attention and norm, and forward()/lm_head() before it is more than a position-0 check. None of the three is wired into Lemonade yet: our vendored Lemonade v11.9 has no MLX backend.