Skip to content

docs(readme): refresh performance tables for v0.3.0 #300

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
inureyes merged 1 commit into from
Jun 16, 2026
Merged

docs(readme): refresh performance tables for v0.3.0 #300

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
108 changes: 66 additions & 42 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,74 +138,98 @@ Linux/CUDA builds use the `cuda` feature and require the CUDA toolkit plus the s
## Performance

mlxcel targets near-`mlx-lm` / `mlx-vlm` decode throughput for MLX-format
checkpoints while keeping a native Rust runtime. In the mlxcel 0.1.0 M5 Max
128GB benchmark set, the headline result has two parts: faster short-prompt
text prefill and near-reference decode throughput.
checkpoints while keeping a native Rust runtime. In the M5 Max 128GB benchmark
campaign, the headline result has two parts: faster short-prompt text prefill
and near-reference decode throughput.

### Prefill: prompt ingestion before the first generated token

Short-prompt text prefill is the standout result. mlxcel measured **2.70x**
the `mlx-lm` median on M5 Max across 66 comparable text pairs, and **1.76x**
on M1 Ultra across 73 comparable text pairs. VLM prefill is listed separately
Short-prompt text prefill is the standout result. mlxcel measured **2.78x**
the `mlx-lm` median on M5 Max across 67 comparable text pairs, and **1.79x**
on M1 Ultra across 74 comparable text pairs. VLM prefill is listed separately
because image preprocessing, vision encoder, and projector work can be included
in the prefill path.

| Mode | Baseline | M5 Max pairs | M5 Max median vs baseline | M1 Ultra pairs | M1 Ultra median vs baseline |
|------|----------|-------------:|--------------------------:|---------------:|----------------------------:|
| Text | `mlx-lm` | 66 | **2.70x** | 73 | **1.76x** |
| VLM | `mlx-vlm` | 20 | 0.94x | 17 | **1.33x** |
| Text | `mlx-lm` | 67 | **2.78x** | 74 | **1.79x** |
| VLM | `mlx-vlm` | 25 | **1.01x** | 20 | **1.05x** |

### Decode: steady-state token generation

Decode stays close to the Python MLX references on the same host. For M5 Max,
text decode averaged **99%** of `mlx-lm` with a **99%** median, while VLM decode
averaged **102%** of `mlx-vlm` with a **101%** median.
text decode averaged **99%** of `mlx-lm` with a **100%** median, while VLM decode
averaged **98%** of `mlx-vlm` with a **98%** median.

| Mode | Baseline | Comparable pairs | Average vs baseline | Median vs baseline | >=90% parity | >= baseline | Range |
|------|----------|-----------------:|--------------------:|-------------------:|-------------:|------------:|------:|
| Text | `mlx-lm` | 66 | 99% | **99%** | 62 / 66 (94%) | 27 / 66 (41%) | 72%-127% |
| VLM | `mlx-vlm` | 22 | 102% | **101%** | 18 / 22 (82%) | 11 / 22 (50%) | 74%-123% |

Representative decode throughput is shown below in tokens per second. M5 Max
reference columns are same-host `mlx-lm` or `mlx-vlm` runs; M1 Ultra values are
included as mlxcel-only capacity references. Absolute results depend on model
family, quantization, prompt shape, decode length, and hardware. See
| Text | `mlx-lm` | 67 | 99% | **100%** | 62 / 67 (93%) | 31 / 67 (46%) | 45%-129% |
| VLM | `mlx-vlm` | 24 | 98% | **98%** | 18 / 24 (75%) | 10 / 24 (42%) | 59%-121% |

Representative decode throughput is shown below in tokens per second. The
mlxcel columns are the 2026-06-15 sweep on each host (v0.3.0, including the fix
to a quantized-decode regression on bf16-scale checkpoints that mostly affected
M1 Ultra). The M5 Max `mlx-lm` / `mlx-vlm` reference columns are retained from
the earlier same-host campaign, so each ratio is mlxcel (2026-06-15) over that
retained reference; a fresh same-host mlx-lm / mlx-vlm run validated that the
reference is stable. M1 Ultra values are mlxcel-only capacity references.
Absolute results depend on model family, quantization, prompt shape, decode
length, and hardware. See
[Benchmark results](docs/benchmark_results/benchmark-report.md) and
[Benchmarks](docs/benchmarks.md) for methodology and caveats.

| Text model | M1 Ultra mlxcel | M5 Max mlxcel | M5 Max mlx-lm | mlxcel / mlx-lm |
|------------|----------------:|--------------:|--------------:|----------------:|
| SmolLM-135M 4bit | 384 tok/s | 905 tok/s | 712 tok/s | 127% |
| Llama 3.1 8B 4bit | 109 tok/s | 117 tok/s | 117 tok/s | 99% |
| SmolLM-135M 4bit | 375 tok/s | 917 tok/s | 712 tok/s | 129% |
| Llama 3.1 8B 4bit | 108 tok/s | 117 tok/s | 117 tok/s | 100% |
| Qwen2.5 7B 4bit | 113 tok/s | 126 tok/s | 124 tok/s | 102% |
| Gemma 2B 4bit | 195 tok/s | 217 tok/s | 223 tok/s | 97% |
| Gemma 3 4B 4bit | 118 tok/s | 182 tok/s | 182 tok/s | 100% |
| Gemma 2 2B 4bit | 170 tok/s | 242 tok/s | 242 tok/s | 100% |
| Phi-3.5-mini 4bit | 167 tok/s | 205 tok/s | 208 tok/s | 98% |
| Jamba v0.1 4bit (hybrid SSM) | 124 tok/s | 216 tok/s | 219 tok/s | 98% |
| Gemma 4 26B-A4B 4bit | 72 tok/s | 137 tok/s | 141 tok/s | 97% |
| Qwen3 MoE 30B 4bit | 71 tok/s | 156 tok/s | 147 tok/s | 106% |
| GLM-4 Flash 4bit | 48 tok/s | 104 tok/s | 104 tok/s | 100% |
| Nemotron-H 30B 4bit | 92 tok/s | 177 tok/s | 179 tok/s | 99% |
| Mixtral 8x7B 4bit | 55 tok/s | 65 tok/s | 66 tok/s | 99% |
| StarCoder2 3B 4bit | 173 tok/s | 216 tok/s | 215 tok/s | 101% |
| Qwen3.5 0.8B 4bit | 244 tok/s | 517 tok/s | 545 tok/s | 95% |
| Qwen3-VL 30B-A3B 4bit, text path | 71 tok/s | 151 tok/s | 147 tok/s | 103% |
| Qwen3-VL 32B 4bit, text path | 21 tok/s | 28 tok/s | 29 tok/s | 96% |
| GPT-OSS 120B 4bit | 61 tok/s | 114 tok/s | 110 tok/s | 103% |
| Solar Open 100B 4bit | 36 tok/s | 65 tok/s | 66 tok/s | 99% |
| Gemma 2B 4bit | 196 tok/s | 215 tok/s | 223 tok/s | 96% |
| Gemma 3 4B 4bit | 117 tok/s | 183 tok/s | 182 tok/s | 101% |
| Gemma 2 2B 4bit | 166 tok/s | 241 tok/s | 242 tok/s | 100% |
| Phi-3.5-mini 4bit | 164 tok/s | 203 tok/s | 208 tok/s | 98% |
| Jamba v0.1 4bit (hybrid SSM) | 122 tok/s | 216 tok/s | 219 tok/s | 99% |
| Gemma 4 26B-A4B 4bit | 80 tok/s | 151 tok/s | 141 tok/s | 107% |
| Qwen3 MoE 30B 4bit | 84 tok/s | 176 tok/s | 147 tok/s | 120% |
| GLM-4 Flash 4bit | 46 tok/s | 104 tok/s | 104 tok/s | 100% |
| Nemotron-H 30B 4bit | 92 tok/s | 176 tok/s | 179 tok/s | 98% |
| Mixtral 8x7B 4bit | 54 tok/s | 65 tok/s | 66 tok/s | 98% |
| StarCoder2 3B 4bit | 166 tok/s | 216 tok/s | 215 tok/s | 100% |
| Qwen3.5 0.8B 4bit | 230 tok/s | 504 tok/s | 545 tok/s | 92% |
| Qwen3-VL 30B-A3B 4bit, text path | 69 tok/s | 151 tok/s | 147 tok/s | 103% |
| Qwen3-VL 32B 4bit, text path | 21 tok/s | 27 tok/s | 29 tok/s | 93% |
| GPT-OSS 120B 4bit | 58 tok/s | 114 tok/s | 110 tok/s | 104% |
| Solar Open 100B 4bit | 33 tok/s | 65 tok/s | 66 tok/s | 98% |

| VLM model | M1 Ultra mlxcel | M5 Max mlxcel | M5 Max mlx-vlm | mlxcel / mlx-vlm |
|-----------|----------------:|--------------:|---------------:|-----------------:|
| LLaVA Interleave Qwen 0.5B bf16 | 266 tok/s | 344 tok/s | 345 tok/s | 100% |
| Qwen3.5 0.8B 4bit | 234 tok/s | 506 tok/s | 411 tok/s | 123% |
| Qwen3.5 35B-A3B 4bit | 70 tok/s | 151 tok/s | 129 tok/s | 117% |
| Gemma 4 E2B 4bit | 107 tok/s | 217 tok/s | 202 tok/s | 108% |
| LLaVA Interleave Qwen 0.5B bf16 | 265 tok/s | 341 tok/s | 345 tok/s | 99% |
| Qwen3.5 0.8B 4bit | 232 tok/s | 454 tok/s | 411 tok/s | 110% |
| Qwen3.5 35B-A3B 4bit | 75 tok/s | 149 tok/s | 129 tok/s | 116% |
| Gemma 4 E2B 4bit | 106 tok/s | 220 tok/s | 202 tok/s | 109% |
| Gemma 3n E2B 4bit | 73 tok/s | 151 tok/s | 125 tok/s | 121% |
| InternVL3 1B | 229 tok/s | 601 tok/s | 529 tok/s | 114% |
| Gemma 4 26B-A4B 4bit | 66 tok/s | 134 tok/s | 137 tok/s | 98% |
| InternVL3 1B | 238 tok/s | 575 tok/s | 529 tok/s | 109% |
| Gemma 4 26B-A4B 4bit | 70 tok/s | 144 tok/s | 137 tok/s | 105% |
| Molmo2 4B | 60 tok/s | 64 tok/s | 67 tok/s | 96% |
| Phi 3.5 Vision 4bit | 123 tok/s | 169 tok/s | 160 tok/s | 106% |
| Phi 3.5 Vision 4bit | 122 tok/s | 168 tok/s | 160 tok/s | 105% |

### DiffusionGemma (block diffusion)

DiffusionGemma generates a canvas block at a time through iterative denoising
rather than left-to-right autoregression. The decode harness above measures
inter-token timing, which does not apply to diffusion's burst output, so the
automated sweep records this checkpoint as a benchmark failure. The numbers
below are a manual same-host comparison (192-token generation, chat template,
seed 42, `max_denoising_steps=48`, median of 3 runs):

| Diffusion model | M1 Ultra mlxcel | M1 Ultra mlx-vlm | mlxcel / mlx-vlm |
|-----------------|----------------:|-----------------:|-----------------:|
| DiffusionGemma 26B-A4B 4bit | 32 tok/s | 29 tok/s | 110% |

Released `mlx-vlm` (0.4.4) does not include `diffusion_gemma`, so the reference
column is `mlx-vlm` upstream `main`. The reported tok/s amortizes the per-block
denoising passes and is not directly comparable to the autoregressive decode
rows above. No M5 Max figure is listed because that comparison was not run on
the same-host campaign.

The M5 Max sweep covers 98 text model directories and a matching 98-entry VLM
mode pass. Ratio summaries include only rows where both mlxcel and the Python
Expand Down