diff --git a/README.md b/README.md index e5885b76c..4f69053aa 100644 --- a/README.md +++ b/README.md @@ -138,74 +138,98 @@ Linux/CUDA builds use the `cuda` feature and require the CUDA toolkit plus the s ## Performance mlxcel targets near-`mlx-lm` / `mlx-vlm` decode throughput for MLX-format -checkpoints while keeping a native Rust runtime. In the mlxcel 0.1.0 M5 Max -128GB benchmark set, the headline result has two parts: faster short-prompt -text prefill and near-reference decode throughput. +checkpoints while keeping a native Rust runtime. In the M5 Max 128GB benchmark +campaign, the headline result has two parts: faster short-prompt text prefill +and near-reference decode throughput. ### Prefill: prompt ingestion before the first generated token -Short-prompt text prefill is the standout result. mlxcel measured **2.70x** -the `mlx-lm` median on M5 Max across 66 comparable text pairs, and **1.76x** -on M1 Ultra across 73 comparable text pairs. VLM prefill is listed separately +Short-prompt text prefill is the standout result. mlxcel measured **2.78x** +the `mlx-lm` median on M5 Max across 67 comparable text pairs, and **1.79x** +on M1 Ultra across 74 comparable text pairs. VLM prefill is listed separately because image preprocessing, vision encoder, and projector work can be included in the prefill path. | Mode | Baseline | M5 Max pairs | M5 Max median vs baseline | M1 Ultra pairs | M1 Ultra median vs baseline | |------|----------|-------------:|--------------------------:|---------------:|----------------------------:| -| Text | `mlx-lm` | 66 | **2.70x** | 73 | **1.76x** | -| VLM | `mlx-vlm` | 20 | 0.94x | 17 | **1.33x** | +| Text | `mlx-lm` | 67 | **2.78x** | 74 | **1.79x** | +| VLM | `mlx-vlm` | 25 | **1.01x** | 20 | **1.05x** | ### Decode: steady-state token generation Decode stays close to the Python MLX references on the same host. For M5 Max, -text decode averaged **99%** of `mlx-lm` with a **99%** median, while VLM decode -averaged **102%** of `mlx-vlm` with a **101%** median. +text decode averaged **99%** of `mlx-lm` with a **100%** median, while VLM decode +averaged **98%** of `mlx-vlm` with a **98%** median. | Mode | Baseline | Comparable pairs | Average vs baseline | Median vs baseline | >=90% parity | >= baseline | Range | |------|----------|-----------------:|--------------------:|-------------------:|-------------:|------------:|------:| -| Text | `mlx-lm` | 66 | 99% | **99%** | 62 / 66 (94%) | 27 / 66 (41%) | 72%-127% | -| VLM | `mlx-vlm` | 22 | 102% | **101%** | 18 / 22 (82%) | 11 / 22 (50%) | 74%-123% | - -Representative decode throughput is shown below in tokens per second. M5 Max -reference columns are same-host `mlx-lm` or `mlx-vlm` runs; M1 Ultra values are -included as mlxcel-only capacity references. Absolute results depend on model -family, quantization, prompt shape, decode length, and hardware. See +| Text | `mlx-lm` | 67 | 99% | **100%** | 62 / 67 (93%) | 31 / 67 (46%) | 45%-129% | +| VLM | `mlx-vlm` | 24 | 98% | **98%** | 18 / 24 (75%) | 10 / 24 (42%) | 59%-121% | + +Representative decode throughput is shown below in tokens per second. The +mlxcel columns are the 2026-06-15 sweep on each host (v0.3.0, including the fix +to a quantized-decode regression on bf16-scale checkpoints that mostly affected +M1 Ultra). The M5 Max `mlx-lm` / `mlx-vlm` reference columns are retained from +the earlier same-host campaign, so each ratio is mlxcel (2026-06-15) over that +retained reference; a fresh same-host mlx-lm / mlx-vlm run validated that the +reference is stable. M1 Ultra values are mlxcel-only capacity references. +Absolute results depend on model family, quantization, prompt shape, decode +length, and hardware. See [Benchmark results](docs/benchmark_results/benchmark-report.md) and [Benchmarks](docs/benchmarks.md) for methodology and caveats. | Text model | M1 Ultra mlxcel | M5 Max mlxcel | M5 Max mlx-lm | mlxcel / mlx-lm | |------------|----------------:|--------------:|--------------:|----------------:| -| SmolLM-135M 4bit | 384 tok/s | 905 tok/s | 712 tok/s | 127% | -| Llama 3.1 8B 4bit | 109 tok/s | 117 tok/s | 117 tok/s | 99% | +| SmolLM-135M 4bit | 375 tok/s | 917 tok/s | 712 tok/s | 129% | +| Llama 3.1 8B 4bit | 108 tok/s | 117 tok/s | 117 tok/s | 100% | | Qwen2.5 7B 4bit | 113 tok/s | 126 tok/s | 124 tok/s | 102% | -| Gemma 2B 4bit | 195 tok/s | 217 tok/s | 223 tok/s | 97% | -| Gemma 3 4B 4bit | 118 tok/s | 182 tok/s | 182 tok/s | 100% | -| Gemma 2 2B 4bit | 170 tok/s | 242 tok/s | 242 tok/s | 100% | -| Phi-3.5-mini 4bit | 167 tok/s | 205 tok/s | 208 tok/s | 98% | -| Jamba v0.1 4bit (hybrid SSM) | 124 tok/s | 216 tok/s | 219 tok/s | 98% | -| Gemma 4 26B-A4B 4bit | 72 tok/s | 137 tok/s | 141 tok/s | 97% | -| Qwen3 MoE 30B 4bit | 71 tok/s | 156 tok/s | 147 tok/s | 106% | -| GLM-4 Flash 4bit | 48 tok/s | 104 tok/s | 104 tok/s | 100% | -| Nemotron-H 30B 4bit | 92 tok/s | 177 tok/s | 179 tok/s | 99% | -| Mixtral 8x7B 4bit | 55 tok/s | 65 tok/s | 66 tok/s | 99% | -| StarCoder2 3B 4bit | 173 tok/s | 216 tok/s | 215 tok/s | 101% | -| Qwen3.5 0.8B 4bit | 244 tok/s | 517 tok/s | 545 tok/s | 95% | -| Qwen3-VL 30B-A3B 4bit, text path | 71 tok/s | 151 tok/s | 147 tok/s | 103% | -| Qwen3-VL 32B 4bit, text path | 21 tok/s | 28 tok/s | 29 tok/s | 96% | -| GPT-OSS 120B 4bit | 61 tok/s | 114 tok/s | 110 tok/s | 103% | -| Solar Open 100B 4bit | 36 tok/s | 65 tok/s | 66 tok/s | 99% | +| Gemma 2B 4bit | 196 tok/s | 215 tok/s | 223 tok/s | 96% | +| Gemma 3 4B 4bit | 117 tok/s | 183 tok/s | 182 tok/s | 101% | +| Gemma 2 2B 4bit | 166 tok/s | 241 tok/s | 242 tok/s | 100% | +| Phi-3.5-mini 4bit | 164 tok/s | 203 tok/s | 208 tok/s | 98% | +| Jamba v0.1 4bit (hybrid SSM) | 122 tok/s | 216 tok/s | 219 tok/s | 99% | +| Gemma 4 26B-A4B 4bit | 80 tok/s | 151 tok/s | 141 tok/s | 107% | +| Qwen3 MoE 30B 4bit | 84 tok/s | 176 tok/s | 147 tok/s | 120% | +| GLM-4 Flash 4bit | 46 tok/s | 104 tok/s | 104 tok/s | 100% | +| Nemotron-H 30B 4bit | 92 tok/s | 176 tok/s | 179 tok/s | 98% | +| Mixtral 8x7B 4bit | 54 tok/s | 65 tok/s | 66 tok/s | 98% | +| StarCoder2 3B 4bit | 166 tok/s | 216 tok/s | 215 tok/s | 100% | +| Qwen3.5 0.8B 4bit | 230 tok/s | 504 tok/s | 545 tok/s | 92% | +| Qwen3-VL 30B-A3B 4bit, text path | 69 tok/s | 151 tok/s | 147 tok/s | 103% | +| Qwen3-VL 32B 4bit, text path | 21 tok/s | 27 tok/s | 29 tok/s | 93% | +| GPT-OSS 120B 4bit | 58 tok/s | 114 tok/s | 110 tok/s | 104% | +| Solar Open 100B 4bit | 33 tok/s | 65 tok/s | 66 tok/s | 98% | | VLM model | M1 Ultra mlxcel | M5 Max mlxcel | M5 Max mlx-vlm | mlxcel / mlx-vlm | |-----------|----------------:|--------------:|---------------:|-----------------:| -| LLaVA Interleave Qwen 0.5B bf16 | 266 tok/s | 344 tok/s | 345 tok/s | 100% | -| Qwen3.5 0.8B 4bit | 234 tok/s | 506 tok/s | 411 tok/s | 123% | -| Qwen3.5 35B-A3B 4bit | 70 tok/s | 151 tok/s | 129 tok/s | 117% | -| Gemma 4 E2B 4bit | 107 tok/s | 217 tok/s | 202 tok/s | 108% | +| LLaVA Interleave Qwen 0.5B bf16 | 265 tok/s | 341 tok/s | 345 tok/s | 99% | +| Qwen3.5 0.8B 4bit | 232 tok/s | 454 tok/s | 411 tok/s | 110% | +| Qwen3.5 35B-A3B 4bit | 75 tok/s | 149 tok/s | 129 tok/s | 116% | +| Gemma 4 E2B 4bit | 106 tok/s | 220 tok/s | 202 tok/s | 109% | | Gemma 3n E2B 4bit | 73 tok/s | 151 tok/s | 125 tok/s | 121% | -| InternVL3 1B | 229 tok/s | 601 tok/s | 529 tok/s | 114% | -| Gemma 4 26B-A4B 4bit | 66 tok/s | 134 tok/s | 137 tok/s | 98% | +| InternVL3 1B | 238 tok/s | 575 tok/s | 529 tok/s | 109% | +| Gemma 4 26B-A4B 4bit | 70 tok/s | 144 tok/s | 137 tok/s | 105% | | Molmo2 4B | 60 tok/s | 64 tok/s | 67 tok/s | 96% | -| Phi 3.5 Vision 4bit | 123 tok/s | 169 tok/s | 160 tok/s | 106% | +| Phi 3.5 Vision 4bit | 122 tok/s | 168 tok/s | 160 tok/s | 105% | + +### DiffusionGemma (block diffusion) + +DiffusionGemma generates a canvas block at a time through iterative denoising +rather than left-to-right autoregression. The decode harness above measures +inter-token timing, which does not apply to diffusion's burst output, so the +automated sweep records this checkpoint as a benchmark failure. The numbers +below are a manual same-host comparison (192-token generation, chat template, +seed 42, `max_denoising_steps=48`, median of 3 runs): + +| Diffusion model | M1 Ultra mlxcel | M1 Ultra mlx-vlm | mlxcel / mlx-vlm | +|-----------------|----------------:|-----------------:|-----------------:| +| DiffusionGemma 26B-A4B 4bit | 32 tok/s | 29 tok/s | 110% | + +Released `mlx-vlm` (0.4.4) does not include `diffusion_gemma`, so the reference +column is `mlx-vlm` upstream `main`. The reported tok/s amortizes the per-block +denoising passes and is not directly comparable to the autoregressive decode +rows above. No M5 Max figure is listed because that comparison was not run on +the same-host campaign. The M5 Max sweep covers 98 text model directories and a matching 98-entry VLM mode pass. Ratio summaries include only rows where both mlxcel and the Python