v0.4.1

Patch release with two bug fixes reported by the community.

Fixes

• Low Power Mode detection (#36): pmset -g no longer exposes the legacy lowpowermode key on Macs that support High Power Mode (for example, MacBook Pro M4 Pro/M5 Pro). mlx-Chronos now falls back to powermode when needed.
• TTFT for reasoning models (#37): Gemma 4 on Ollama can consume the only available token (max_tokens=1) in hidden reasoning. Ollama’s OpenAI-compatible endpoint does not expose that text (known upstream issue: ollama/ollama#15288). measure_ttft now also treats a token-limit stop signal as a valid measurement.

Thanks to @umigliore for both reports, a working workaround already included in #36 and a precise diagnosis in #37.

v0.4.0

mlx-Chronos 0.4.0 makes public leaderboard-ready benchmark runs easier to diagnose, configure, and verify before submission.

Features

• Adds mlx-chronos doctor, a single diagnostic command for Apple Silicon readiness, supported engine status, running servers, model access, model reference URLs, and public leaderboard blockers.
• Adds mlx-chronos run --publishable, which fails early unless a run uses the required public leaderboard settings: a model URL, JSON output, persistent connections, preflight validation, and Low Power Mode disabled.
• Adds an actionable post-run leaderboard-readiness summary that reports whether a result is ready to submit or remains local-only, including the first blocker and a concrete fix.
• Makes the interactive wizard more guided for public submissions by surfacing the doctor flow, prioritizing running engines, requiring model URLs for publishable runs, and generating equivalent --publishable commands.

Documentation

• Expands the README quickstart with a more plug-and-play path from installation and diagnostics to a publishable benchmark run.

Maintenance

• Adds CLI and wizard test coverage for the new diagnostics, publishability checks, readiness reporting, and guided public-run flow.

v0.3.1

mlx-Chronos 0.3.1 is a patch release that simplifies public model identity metadata after 0.3.0 and tightens release validation.

Bug Fixes

• Removes optional model identity fields beyond model.reference_url from the result schema, submitted archive files, generated leaderboard index, and public leaderboard UI.
• Stops recording Ollama family and parameter_size details in benchmark result JSON while keeping MLX format and quantization validation.
• Regenerates the public leaderboard index without the removed model metadata columns.

Maintenance

• Adds tests that prevent removed model metadata fields from returning to the schema or generated public index.
• Runs Ruff, mypy, frontend tests, coverage, generated-index drift checks, and twine check in the release workflow before publishing.

v0.3.0

mlx-Chronos 0.3.0 is a minor release focused on guided CLI workflows, auditable measurements, and stricter public leaderboard integrity.

Features

• Adds an interactive mlx-chronos wizard with server model discovery and reusable command previews.
• Adds background PyPI update checks, mlx-chronos upgrade, and optional model preflight validation.
• Adds required model reference metadata, reconstructible raw decode timing, and a concise post-run result summary.

Reliability and Integrity

• Requires publishable results to use known engine versions, Apple Silicon metadata, clean warmup, plausible timestamps, and error-free RAM and thermal monitoring.
• Prevents retries from contaminating timed TTFT and throughput phases and stops cached-TTFT measurement when priming fails.
• Uses Ollama server metadata as the authority for version, MLX format, and quantization.
• Rejects duplicate results, secures leaderboard rendering, preserves model URL variants, and ranks comparisons by end-to-end request throughput.
• Adds bounded transient retries for untimed operations, atomic report writes, and stricter timing and thermal consistency checks.

Maintenance

• Moves leaderboard generation into a tested Python module and adds Ruff, mypy, coverage, and frontend security gates to CI.
• Makes pyproject.toml the only manually maintained package-version source, with no stale numeric fallback.

v0.2.1

mlx-Chronos 0.2.1 is a patch release for engine adapter correctness after validating 0.2.0 against real local servers. It does not change the result schema, public leaderboard policy, or internal benchmark compatibility label.

Bug Fixes

• Recognizes omlx-server as a valid oMLX listener process, avoiding false "server is not running" errors with current oMLX releases.
• Resolves local mlx-lm model aliases so servers started from local model paths can still be benchmarked with user-facing short model names.
• Accepts the current vllm-mlx /health response shape with status=healthy, ready=true, and a loaded model_name.
• Avoids printing the cached-TTFT warning twice for the same benchmark run.

v0.2.0

Minor release focused on stricter public-leaderboard comparability, clearer benchmark metadata, and broader MLX engine support. Local benchmarking remains flexible, while publishable results now follow a more explicit standard policy.

Highlights

• Adds vllm-mlx engine support, including model listing, version detection, port override support, and server identity checks.
• Uses a persistent HTTP client by default across benchmark phases to reduce per-request connection setup noise and better match repeated local agent loops.
• Records benchmark request semantics in result metadata, including streaming mode, usage metadata, deterministic generation parameters, prompt text, token bounds, and HTTP connection mode.
• Hardens public submission validation with exact standard-protocol checks, integrity seals, fixed publishable trial counts, Low Power Mode rejection, usage-based token counts, and an 80% minimum throughput output threshold.
• Prevents oMLX/vllm-mlx confusion on the default port by checking the oMLX listener process.
• Refreshes the leaderboard UI with a clearer compare/raw-data flow, persistent light/dark theme toggle, cleaner optional columns, and fixed column-menu clipping.
• Adds optional model provenance fields to the schema for future stronger model identity metadata.

Compatibility

• Result JSON produced by 0.1.x remains useful for local analysis, but is not publishable under the 0.2 public leaderboard policy.
• Protocol numbers are treated as internal compatibility labels, not public protocol versions; the documentation describes the methodological changes instead of presenting them as user-facing version jumps.

Maintenance and Docs

• Removes debug __main__ blocks from library modules; the supported entrypoint remains the mlx-chronos CLI.
• Updates documentation for persistent HTTP behavior, stricter public submissions, vllm-mlx setup, oMLX listener inspection, and the local-vs-public benchmark split.
• The Python package is published by the release workflow attached to this tag.

See CHANGELOG.md for the full change list.

v0.1.3

Patch release focused on benchmark measurement reliability, leaderboard clarity, and release hardening.

Highlights

• Tightens throughput timing, sustained-run warnings, warmup failure handling, and cached-TTFT warnings.
• Clarifies System RAM Peak as the comparable memory metric and keeps Engine RSS as a legacy diagnostic field in raw result details.
• Improves the public leaderboard compare view, generated index metadata, decode throughput visibility, and power-condition reporting.
• Hardens submission and release workflows, including PR labeling, mixed-content submission rejection, tag/version checks, and Python 3.10-3.14 release tests.
• Adds mlx-chronos --version, mlx-chronos models, optional thermal install support, and updated methodology docs.

See CHANGELOG.md for the full change list.

v0.1.2

Patch release focused on benchmark comparability, sustained-run visibility, and clearer local result warnings.

Highlights

• Adds benchmark protocol metadata, throughput token bounds, protocol v2 streaming throughput, and client-observed decode throughput when reliable usage data is available.
• Adds --profile sustained with progress samples, --cooldown-seconds, elapsed-since-prior-result metadata, phase timings, and lightweight thermal monitoring.
• Adds throughput max-token metadata/filtering to the leaderboard and clearer warning metadata for word fallback counts and unknown engine versions.

Bug Fixes

• Fixes sustained progress sample rounding so saved JSON validates against the same elapsed value it reports.
• Removes the unproduced engine_response decode timing source from runtime/schema metadata.
• Requires throughput completion-token and elapsed-time raw fields to be present together.
• Avoids duplicate total-runtime rows in Markdown reports.
• Improves oMLX version detection, including /v1/models metadata fallback.

Compatibility

• Existing protocol v1 leaderboard rows remain valid, but protocol v2 throughput uses streaming requests, so v1 and v2 throughput workloads are not identical.

v0.1.1

Patch release focused on safer community submissions, clearer leaderboard semantics, and stronger validation.

Highlights:

• Adds mlx-chronos submit with dry-run validation and optional contact email.
• Requires usage.completion_tokens for public leaderboard submissions.
• Clarifies TTFT and memory methodology, and uses System RAM Peak as the public memory metric.
• Adds completion-token visibility, benchmark-condition warnings, and mock OpenAI-compatible integration tests.

Full details are in CHANGELOG.md.

v0.1.0

First public stable release of mlx-Chronos.

mlx-Chronos is a benchmark suite and community leaderboard for local LLM inference on Apple Silicon, focused on comparable results across Ollama, oMLX, Rapid-MLX, and mlx-lm.

Highlights

• Added the mlx-chronos validate command to check hardware detection, engine availability, server health, model listing, and optional completion access before running a benchmark.

• Published PyPI-ready packaging, so the CLI can now be installed with:

  pip install mlx-chronos

• Improved benchmark reliability:

  • preserves full model identifiers such as org/model-name
  • rejects blank model names before execution
  • reports clearer engine/model request errors with URL, engine, model, status code, and response body context
  • hardens /models and completion-response parsing
  • adds safer timeout handling for Rapid-MLX version detection

• Clarified TTFT semantics: whitespace-only streamed text is counted as the first generated token when measuring time to first token.

• Refreshed Apple M2 sample results and updated the generated leaderboard index.

• Added macOS CI smoke checks and release automation for test, build, and PyPI publishing.

Install

pip install mlx-chronos
mlx-chronos engines
mlx-chronos validate --engine omlx --model "Qwen3.5-4B-OptiQ-4bit"
mlx-chronos run --engine omlx --model "Qwen3.5-4B-OptiQ-4bit"

Known Limitations

• mlx-chronos submit is not implemented yet.
• Tool-calling benchmark support is planned for a future release.
• Published sample results currently cover Apple M2 8GB only.

Full Diff

v0.1.0-beta.1...v0.1.0