Repository navigation
Benchmark for local models served by Ollama: time to first token, tokens/s and VRAM usage while generating, manual classification of answers, history and CSV/HTML export.
Requirements
- Windows 10/11 (x64)
- Ollama installed and running separately - it is not bundled. Everything here was measured against Ollama 0.34.3.
- Optional: an NVIDIA card and driver for the VRAM metrics (NVML). Without one the app still works and only those metrics are skipped.
Install
- Download
AI-Benchmark_1.0.0_x64-setup.exebelow and run it. It installs per user (no administrator prompt) and is removed from "Apps & features" like any other program. - The installer is not code-signed, so SmartScreen shows "Windows protected your PC" the first time you run it: choose More info -> Run anyway.
- Models are not bundled either. They are downloaded on demand from the Models tab, and the app shows the size of every one before the download.
SHA-256 49e95110d1487a9019696f7f1921f4d0c59ebe2c5773edbf2a5e03e1009e4bc7
Uninstalling leaves your data alone: %APPDATA%\com.aibenchmark.dev\ (model list, settings, history) and the exports in Downloads\AI Benchmark\.
What is inside
- Live metrics: TTFT, tokens/s (Ollama's own numbers plus a wall-clock measurement), VRAM peak and the GPU/CPU split.
- A memory forecast with four states before you download anything: on the card / on the card without headroom / partly on the CPU / will not run.
- Categories with your own prompts: coding, chat, images (VLM), reasoning, JSON, long context.
- Manual classification of answers - completed / refused / completed with limits or changes - three buttons, no second model guessing.
- History of runs, A/B comparison over time, CSV and HTML export.
- HuggingFace catalogue with per-variant sizes and memory verdicts.
- Seven interface languages: English, Polish, German, Spanish, French, Portuguese, Italian.
Screenshots and full documentation: README.