Skip to content

v0.2.0

Choose a tag to compare

View all tags
@igurss igurss released this 12 Jun 10:25
· 46 commits to main since this release
v0.2.0
1c21d6f

Minor release focused on stricter public-leaderboard comparability, clearer benchmark metadata, and broader MLX engine support. Local benchmarking remains flexible, while publishable results now follow a more explicit standard policy.

Highlights

  • Adds vllm-mlx engine support, including model listing, version detection, port override support, and server identity checks.
  • Uses a persistent HTTP client by default across benchmark phases to reduce per-request connection setup noise and better match repeated local agent loops.
  • Records benchmark request semantics in result metadata, including streaming mode, usage metadata, deterministic generation parameters, prompt text, token bounds, and HTTP connection mode.
  • Hardens public submission validation with exact standard-protocol checks, integrity seals, fixed publishable trial counts, Low Power Mode rejection, usage-based token counts, and an 80% minimum throughput output threshold.
  • Prevents oMLX/vllm-mlx confusion on the default port by checking the oMLX listener process.
  • Refreshes the leaderboard UI with a clearer compare/raw-data flow, persistent light/dark theme toggle, cleaner optional columns, and fixed column-menu clipping.
  • Adds optional model provenance fields to the schema for future stronger model identity metadata.

Compatibility

  • Result JSON produced by 0.1.x remains useful for local analysis, but is not publishable under the 0.2 public leaderboard policy.
  • Protocol numbers are treated as internal compatibility labels, not public protocol versions; the documentation describes the methodological changes instead of presenting them as user-facing version jumps.

Maintenance and Docs

  • Removes debug __main__ blocks from library modules; the supported entrypoint remains the mlx-chronos CLI.
  • Updates documentation for persistent HTTP behavior, stricter public submissions, vllm-mlx setup, oMLX listener inspection, and the local-vs-public benchmark split.
  • The Python package is published by the release workflow attached to this tag.

See CHANGELOG.md for the full change list.

Assets 2
Loading