LLM-Benchmarker

A transparent HTTP proxy that intercepts LLM API calls (Ollama, llama.cpp, OpenAI-compatible), captures full request/response data, and displays real-time telemetry through a web dashboard.

• No code changes — sits between your client and LLM engine
• Captures everything — full request JSON and response body for every call
• Live metrics — TPS, TTFT, token count per request, per-model
• Searchable history — filter by endpoint, click any request for details
• Request Replay — re-send any captured request and see the result inline
• Benchmark Suite — context-scaling, parallel-scaling, and combined load tests
• Scheduling — run benchmarks on a cron preset (5m, 15m, hourly, daily)
• Alert Thresholds — define rules (TPS < 10, TTFT > 5000ms) that flag matching requests
• Session Grouping — requests grouped by client IP
• Prometheus Metrics — /metrics endpoint exposes standard LLM telemetry
• Single binary — Go + embedded React UI, no runtime dependencies

Install

Pre-built binaries

Download from GitHub Releases:

Platform	File
Linux x86_64	llm-benchmarker-linux-amd64
Linux ARM64	llm-benchmarker-linux-arm64
macOS Intel	llm-benchmarker-darwin-amd64
macOS Apple Silicon	llm-benchmarker-darwin-arm64
macOS Universal	llm-benchmarker-darwin-universal

Build from source

git clone https://github.com/codewithwest/llm-benchmarker.git
cd llm-benchmarker

# Build everything in one step:
./start.sh

# Or manually:
cd ui && npm install && npm run build && cd ..
go build -buildvcs=false -o llm-benchmarker .

Requires Go 1.25+ and Node.js 22+.

Quick Start

Point it at a running Ollama instance:

# Ollama runs on the default port 11434
./llm-benchmarker

Open http://localhost:8080 in your browser. The dashboard loads — send a prompt through the chat panel. Every response is recorded as a card in the Requests tab.

Transparent intercept mode

Replace Ollama's port so existing clients are intercepted automatically:

# 1. Move Ollama to port 11435
OLLAMA_HOST=0.0.0.0:11435 ollama serve

# 2. Run benchmarker on port 11434, forwarding to 11435
./llm-benchmarker -port 8090 -intercept-port 11434 -target http://127.0.0.1:11435

# 3. Any client hitting port 11434 gets intercepted and proxied to 11435

CLI Flags

Flag	Default	Description
-port	8080	Port for the web UI and API
-target	http://127.0.0.1:11434	Upstream LLM engine URL
-intercept-port	0	If set, also listen on this port as a transparent intercept proxy
-db	benchmarks.db	Path to SQLite database file

Dashboard

Dashboard tab

• KPIs — active providers, average TPS, average TTFT, total requests, triggered alerts
• TPS Trend — real-time chart of tokens-per-second over time
• Recent Activity — last 5 requests as clickable cards
• Active Sessions — unique client IPs with request count and model list

Requests tab

• Full history of all intercepted requests
• Filter by endpoint
• Click any card to open the detail modal showing:
  • Telemetry (TPS, TTFT, tokens, model, client IP, duration)
  • Formatted Request JSON
  • Formatted Response Body
  • Replay button — re-sends the original request and shows the live result inline

Providers tab

• Register and monitor remote LLM engine nodes
• Each node is health-checked every 5 seconds
• Select an active provider for the chat panel

Benchmarks tab

• Run Tests — select a model from an active provider, then run:
  • Context Scaling — tests prompt lengths of 256, 512, 1024, 2048, 4096, 8192 tokens
  • Parallel Scaling — tests 1, 2, 4, 8 concurrent requests
  • Combined Matrix — 5 context-sizes × 3 concurrency levels (full suite)
• Run History — table with status, model, score, and test types
• Detail View — per-test tables (context, tokens, TPS, TTFT, duration), TPS/TTFT bar charts, LLM-generated summary report, CSV/JSON export
• Comparison — check two runs and view side-by-side LLM reports
• Scheduling — add/remove schedules with presets (5m, 15m, 30m, hourly, daily, weekly); background scheduler auto-launches them

Settings tab

• Active Alerts — shows currently triggered thresholds (metric, value, actual, model, timestamp)
• Alert Thresholds — configure rules with metric (TPS, TTFT, duration), operator (<, >), value, and optional model filter

Chat panel

• FAB button (bottom-right) opens a streaming chat overlay
• Select a model and provider, type a prompt, hit Enter
• Response streams in real-time

Architecture

┌──────────────┐      ┌──────────────────┐      ┌──────────────┐
│   Browser    │ HTTP  │  llm-benchmarker │ HTTP  │    Ollama    │
│  (React UI)  │──────▶│   (Go Proxy)    │──────▶│  (Engine)    │
│              │◀──────│                  │◀──────│              │
└──────────────┘      │   ┌──────────┐   │      └──────────────┘
                      │   │ SQLite   │   │
                      │   │  store   │   │
                      │   └──────────┘   │
                      └──────────────────┘

1. Browser sends a prompt to the Go server
2. Server forwards the request to the upstream LLM engine
3. Response streams back to the browser in real-time
4. On completion, full request/response is saved to SQLite

The proxy handles both streaming and non-streaming responses, counting tokens by tracking newline-delimited JSON objects (streaming) and whitespace-separated words (non-streaming fallback).

Use Cases

• Benchmarking — measure TPS, TTFT across different models and hardware
• Auditing — inspect every prompt and response sent to your LLM
• Debugging — see exactly what your client sends and what the engine returns
• Monitoring — track usage patterns, response quality, latency trends

Development

# Frontend dev server (hot reload)
cd ui && npm run dev

# Build everything
./start.sh

# Run tests
go test ./internal/...

CI/CD

Push to main triggers:

• ci.yml — builds frontend, runs Go tests, compiles binary

Push a tag (v*) triggers:

• release.yml — cross-compiles for linux/darwin (amd64 + arm64), creates a macOS universal binary with lipo, publishes to GitHub Releases

License

MIT