ModelBench

An on-device LLM benchmark harness for Apple Silicon. ModelBench runs standardized eval suites against local inference backends (Ollama and mlx_lm.server), captures live performance telemetry — token throughput, time-to-first-token, memory footprint, and thermal state — and persists every run for historical comparison.

Built with SwiftUI, Swift 6 strict concurrency, Swift Charts, and SwiftData, targeting macOS 26+.

Status: early. The app builds, the service layer is tested against a live Ollama instance, and the UI runs. A writeup is coming.

Screenshots

A live run — the tokens/sec chart captures a thermal-throttle dip mid-run, with readouts for current rate, memory, time-to-first-token, and thermal state, plus per-case results as they complete:

History with a two-run comparison:

Why

Local model throughput depends on far more than parameter count: quantization, KV-cache pressure, thermal throttling, and prompt shape all move the numbers. ModelBench makes those numbers visible and comparable, and pairs them with correctness via three eval suites so you can weigh speed against quality on your own hardware.

Features

• Two backends behind one protocol — OllamaBackend (native /api/chat, with authoritative eval_count/eval_duration) and MLXBackend (OpenAI-compatible /v1, the wire protocol mlx_lm.server speaks).
• Live telemetry — instantaneous tokens/sec, TTFT, model/server memory (the inference process's resident size, summed by the helper via ps, falling back to the app's own task_info/phys_footprint), and host thermal state (ProcessInfo.thermalState), sampled every 500 ms and merged with the inference stream.
• Three eval suites
  • IFEval (subset) — 6 instruction-following cases scored deterministically (bullet/sentence/word counts, required/forbidden keywords, JSON validity).
  • SwiftUI Generation — 8 code-generation prompts graded by an LLM judge against modern-SwiftUI rubrics.
  • HumanEval (subset) — 10 Python problems with executable tests. Scoring runs generated code, which requires the app sandbox to be disabled (see Code execution); in the sandboxed build these report as execution-disabled.
• History & comparison — runs are stored in SwiftData; select two to compare throughput, pass rate, and peak memory side by side.

Architecture

Domain    pure Swift, Sendable value types — no framework imports
          EvalCase · EvalSuite · ScoringStrategy · BenchmarkRun · TelemetrySample · ThermalState

Services  actors + structured concurrency
          ModelBackend → MLXBackend · OllamaBackend
          TelemetryMonitor  (AsyncStream of system samples)
          BenchmarkRunner   (TaskGroup: inference ⊕ telemetry → RunEvent stream)
          Scorer · JudgeClient

Storage   SwiftData
          RunRecord · TelemetryRecord · EvalResultRecord · @ModelActor BenchmarkStore

UI        SwiftUI + @Observable (@MainActor)
          BenchmarkViewModel · RunConfigView · LiveRunView · HistoryView

TelemetryMonitor knows nothing about inference — it emits raw system samples. BenchmarkRunner owns the token counter and, on each sample, computes instantaneous tok/s before yielding a merged TelemetrySample. All run state is actor-isolated; the inference loop and telemetry consumer run concurrently under one withThrowingTaskGroup with no data races and no Task.detached.

Requirements

• macOS 26+, Xcode 26+
• XcodeGen (brew install xcodegen) — the .xcodeproj is generated, not checked in
• At least one backend running:
  • Ollama on :11434 (ollama serve), or
  • mlx_lm.server on :8080

Getting started

git clone https://github.com/wesmatlock/ModelBench.git
cd ModelBench
xcodegen generate
open ModelBench.xcodeproj

Then select the ModelBench scheme and run. Pick a backend and model, choose an eval suite, and hit Run Benchmark.

To build/test from the command line:

xcodebuild -scheme ModelBench -destination 'platform=macOS' build
xcodebuild -scheme ModelBench -destination 'platform=macOS' test

The backend tests exercise a live Ollama instance when one is reachable and no-op (pass) when it isn't, so the suite stays green offline.

Code execution

HumanEval scoring compiles and runs model-generated Python. The macOS app sandbox blocks spawning python3 directly, so rather than dropping the app's sandbox, ModelBench runs the code in a small unsandboxed XPC helper (CodeRunner.xpc, bundle id net.insoc.ModelBench.CodeRunner). The main app stays sandboxed; it sends the model's output and the test over XPC, the helper writes them to a temp dir, runs /usr/bin/python3 with a timeout, and replies with pass/fail. If the helper is unreachable, those cases report "execution unavailable" rather than faking a score.

Note this still executes untrusted model output on your machine — just inside a separate process with a clear boundary instead of in-app.

Project layout

Path	Contents
Domain/	Pure value types
Services/	Backends, telemetry, runner, scorer, judge, XPC client
Helper/	CodeRunner XPC service (runs HumanEval Python)
Shared/	XPC contract shared by app and helper
Evals/	The three eval suites
Storage/	SwiftData models and the @ModelActor store
ViewModels/	BenchmarkViewModel
Views/	SwiftUI views
Tests/	Swift Testing suites
project.yml	XcodeGen manifest

License

MIT © 2026 Wesley Matlock