Skip to content

v0.3.0

Choose a tag to compare

@github-actions github-actions released this 15 Sep 16:57

[0.3.0] - 2026-09-15

✨ Added

  • Side-by-side diff view: compare any two model outputs line-by-line, with similarity percentage, added/removed line counts, pair picker (for 3–4 model runs), and a toggle to hide unchanged lines — no new dependencies (custom LCS-based diff)
  • Winner votes: crown the best answer per run with a per-column vote button; the winner gets a highlighted column, a badge, and is remembered in history
  • Model leaderboard: aggregated per-model stats (run count, average TPS, win count, win rate) in a sortable sidebar panel, persisted in SQLite
  • CSV export: download the full benchmark history as CSV (one row per model result: run id, date, prompt, model, TPS, estimated flag, TTFT, winner, output)
  • Keyboard shortcuts: Ctrl+Enter (or Cmd+Enter) to run, Esc to stop generation or close panels

🔄 Changed

  • Unified TPS definition: TPS now measures the generation phase only (excluding time-to-first-token) on every provider, matching the meaning of Ollama's eval_count/eval_duration. When a server doesn't report token usage (some LM Studio / llama.cpp setups), TPS is estimated from output length and clearly marked with ~ in columns, charts, history, and Markdown/PDF exports
  • Settings now persist: theme, language, provider, base URL, and advanced controls (system prompt, temperature, top-p, context length) survive app restarts via persisted store
  • Database schema extended (non-destructive, auto-migrated on launch): new winner_model column on benchmark_runs, plus tps_estimated and token_source on benchmark_results. Databases from v0.2.0 open without data loss
  • Version is now centralized in src/version.ts (kept in sync with package.json, tauri.conf.json, and Cargo.toml); app window title and default size updated (1280×800, min 1024×640)

🐛 Fixed

  • Run button could get stuck in the "running" state forever when any single model column errored — completion is now tracked per-slot (success, error, or stop), partial runs are always saved to history, and failed/stopped columns get a dedicated Retry button that re-runs only that slot
  • Switching providers no longer wipes a manually edited server URL unless the provider actually changed
  • JSON.parse on Ollama NDJSON streams is now guarded — a single malformed line no longer kills the whole stream
  • Removed unused i18n key (app.version) that showed a stale hardcoded version in the header