Skip to content

v0.1.0

Choose a tag to compare

View all tags
@igurss igurss released this 29 May 15:06
· 109 commits to main since this release
v0.1.0
97ee058

First public stable release of mlx-Chronos.

mlx-Chronos is a benchmark suite and community leaderboard for local LLM inference on Apple Silicon, focused on comparable results across Ollama, oMLX, Rapid-MLX, and mlx-lm.

Highlights

  • Added the mlx-chronos validate command to check hardware detection, engine availability, server health, model listing, and optional completion access before running a benchmark.

  • Published PyPI-ready packaging, so the CLI can now be installed with:

    pip install mlx-chronos

  • Improved benchmark reliability:

    • preserves full model identifiers such as org/model-name
    • rejects blank model names before execution
    • reports clearer engine/model request errors with URL, engine, model, status code, and response body context
    • hardens /models and completion-response parsing
    • adds safer timeout handling for Rapid-MLX version detection

  • Clarified TTFT semantics: whitespace-only streamed text is counted as the first generated token when measuring time to first token.

  • Refreshed Apple M2 sample results and updated the generated leaderboard index.

  • Added macOS CI smoke checks and release automation for test, build, and PyPI publishing.

Install

pip install mlx-chronos
mlx-chronos engines
mlx-chronos validate --engine omlx --model "Qwen3.5-4B-OptiQ-4bit"
mlx-chronos run --engine omlx --model "Qwen3.5-4B-OptiQ-4bit"

Known Limitations

  • mlx-chronos submit is not implemented yet.
  • Tool-calling benchmark support is planned for a future release.
  • Published sample results currently cover Apple M2 8GB only.

Full Diff

v0.1.0-beta.1...v0.1.0

Assets 2
Loading