Skip to content

v0.1.2

Choose a tag to compare

@igurss igurss released this 05 Jun 15:02
· 77 commits to main since this release

Patch release focused on benchmark comparability, sustained-run visibility, and clearer local result warnings.

Highlights

  • Adds benchmark protocol metadata, throughput token bounds, protocol v2 streaming throughput, and client-observed decode throughput when reliable usage data is available.
  • Adds --profile sustained with progress samples, --cooldown-seconds, elapsed-since-prior-result metadata, phase timings, and lightweight thermal monitoring.
  • Adds throughput max-token metadata/filtering to the leaderboard and clearer warning metadata for word fallback counts and unknown engine versions.

Bug Fixes

  • Fixes sustained progress sample rounding so saved JSON validates against the same elapsed value it reports.
  • Removes the unproduced engine_response decode timing source from runtime/schema metadata.
  • Requires throughput completion-token and elapsed-time raw fields to be present together.
  • Avoids duplicate total-runtime rows in Markdown reports.
  • Improves oMLX version detection, including /v1/models metadata fallback.

Compatibility

  • Existing protocol v1 leaderboard rows remain valid, but protocol v2 throughput uses streaming requests, so v1 and v2 throughput workloads are not identical.