Skip to content

v0.27.3 Muse prefill preview 1

Pre-release
Pre-release

Choose a tag to compare

@justinchuby justinchuby released this 12 Aug 22:22
Immutable release. Only release title and notes can be modified.

Preview wheel with a pinned, bundled MLX mixed-BF16/FP16 QMM runtime. It does not require a patched system MLX installation.

Enable the optimized path with:

export MLX_BF16_QMM_FP16=1
export ONNXRUNTIME_EP_MLX_LAYER_PARTITIONS=8

M1 Max / Muse-Glimmer-30B results from the same pinned runtime: 138.67 tok/s median for 512-token prefill and 14.79 tok/s for a 200-token decode.

Source PRs: onnxruntime-mlx#27 and ml-explore/mlx#4216.