v0.4.0

v0.4.0 — Reliable Telemetry & Request Operations

New Features

• Request outcome telemetry — captured requests now include model, HTTP status, error message, and token-count source
• Server-side request filtering — filter request history by provider, endpoint, model, date range, text search, and success/error status
• Filtered CSV export — export the currently visible request set from the Requests tab
• Health check — GET /healthz returns a lightweight service status response
• Replay endpoint targeting — replay uses the original captured endpoint, including /api/generate and /v1/chat/completions

Improvements

• Failed and zero-token requests are now retained instead of disappearing from telemetry
• Streaming response flushing is preserved through the tracking response writer
• Captured request and response bodies are limited to 10 MiB to protect memory and storage
• Replay is restricted to registered provider URLs
• Added SQLite indexes for timestamp, endpoint, and model queries
• Request cards and detail views show model, outcome status, and token-source metadata
• Background request refreshes preserve the latest active filters
• Existing benchmark persistence APIs remain backward-compatible while new metadata is stored explicitly

Bug Fixes

• Fixed request replay returning 405 method not allowed by avoiding POST requests to the provider root
• Fixed stale polling responses replacing newer filtered Requests results
• Fixed response capture accounting so the stored body is capped without truncating bytes forwarded upstream
• Fixed backend release builds failing when proxy status and error metadata fields were missing from the response tracker
• Fixed database test and build mismatches after adding model, outcome, and request-filter metadata

Tests

• Added replay URL construction tests for Ollama and OpenAI-compatible endpoints
• Added persistence and filtering tests for model and outcome metadata
• Added proxy coverage for failed upstream responses and zero-token requests
• Full Go test suite and frontend production build pass

Compatibility Notes

• Existing SQLite databases are migrated automatically with the new benchmark metadata columns and indexes
• Rebuild the application binary after upgrading so the embedded UI and API changes are included

What's Changed

• feat: enrich request handling and telemetry by @codewithwest in #7

Full Changelog: v0.3.1...v0.4.0

What's Changed

• feat: enrich request handling and telemetry by @codewithwest in #7

Full Changelog: v0.3.1...v0.4.0

v0.3.1

v0.3.1 — Requests Filter Refresh Fix

Bug Fixes

• Preserved Requests filters during polling — selecting a path no longer resets the Requests tab to all requests when new traffic arrives
• Prevented stale refresh results — older polling responses can no longer overwrite a newer filtered result
• Model name on request cards — cards now show the captured model before opening request details

Improvements

• Request filter state remains active across automatic refreshes
• Request cards keep endpoint and model context visible at a glance

What's Changed

• 5 filter flash on all request by @codewithwest in #6

Full Changelog: v0.3.0...v0.3.1

What's Changed

• 5 filter flash on all request by @codewithwest in #6

Full Changelog: v0.3.0...v0.3.1

v0.3.0

v0.3.0 — Dynamic Proxy Target, Request Filters, Provider Editing & Data Retention

New Features

• Dynamic Proxy Target — change the proxy forwarding target at runtime via POST /api/proxy/target or by clicking a provider card in the UI; backed by atomic target URL storage
• Request Filtering — filter intercepted requests by provider, endpoint/path, and date range (from/to) on the Requests tab
• Provider Editing — edit provider name and URL inline via the new edit button on provider cards; backed by PATCH /api/dashboard/providers/{id}
• Filter Options API — GET /api/dashboard/filters returns distinct providers and endpoints to populate filter dropdowns
• Data Retention — new -retention flag auto-purges benchmark data older than N hours (0 = disable)

Improvements

• Provider URL validation on add/update: rejects non-http:///https:// schemes with a 400 error
• More accurate token counting: the proxy interceptor now tracks SSE data: lines as tokens, using the max of all three counting methods
• X-Target-Provider header scheme validation: unsupported schemes are logged and fall through to the default target
• Scheduled benchmarks update their last-run status to "completed" on success
• UI layout: proper scroll containment, separate filtered/unfiltered stats, auto-refresh of filtered stats on polling

Bug Fixes

• Panic recovery (defer/recover) added to all background goroutines — benchmark runs, health poller, scheduler, parallel scaling workers — preventing a single panic from crashing the server
• Provider update returns a proper error if the provider ID does not exist
• Cleaner request body handling on read errors

Chores

• New API routes: POST /api/proxy/target, PATCH /api/dashboard/providers/{id}, GET /api/dashboard/filters
• New CLI flag: -retention
• React ProviderCard component extracted for reuse
• Added sync/atomic and net/url imports

What's Changed

• Refactor panic recovery, add dynamic proxy config, and update provider logic by @codewithwest in #3

Full Changelog: v0.2.0...v0.3.0

What's Changed

• Refactor panic recovery, add dynamic proxy config, and update provider logic by @codewithwest in #3

Full Changelog: v0.2.0...v0.3.0

v0.2.0

v0.2.0 — Prometheus, Replay, Benchmarks, Scheduling & Alerts

New Features

• Prometheus /metrics — exposes llm_requests_total, llm_tokens_total, llm_tps_last, llm_ttft_ms_last, llm_duration_ms_last with per-model labels
• Request Replay — replay any captured request from the detail modal; result displayed inline
• Session Grouping — dashboard shows active sessions grouped by client IP with request/model counts
• Benchmark Suite — context-scaling, parallel-scaling, and combined matrix load tests with tables, Recharts, and LLM-generated summary reports
• Benchmark Comparison — select two runs and view side-by-side LLM reports
• Benchmark Scheduling — cron presets (5m, 15m, 30m, hourly, daily, weekly) with background scheduler goroutine
• Alert Thresholds — configurable metric thresholds (TPS, TTFT, duration) with operator, value, and optional model filter; live checking on each poll
• Settings Tab — UI for managing alert thresholds and viewing triggered alerts
• Client IP & Duration — captured on every request, shown in cards and detail modal
• Tab Routing — hash-based routes so refreshing or sharing a URL preserves the active tab
• Stylized Dropdowns — custom Select component shared across model picker, schedule form, and alert threshold form
• Grid/List Toggle — benchmark runs can be viewed as a grid or list

Improvements

• Orphaned benchmark runs are marked failed on server restart
• Benchmark form pulls models from the active provider service
• Scroll fix for benchmark detail view (overflow-y wrapper)
• SVG logo and favicon
• Shared Select component extracted for consistent UI

Bug Fixes

• CheckAlerts no longer returns JSON null (was crashing the frontend with Cannot read properties of null (reading 'length'))

Chores

• MIT license
• Full CI/CD workflows (ci.yml, release.yml)
• Updated README with all features, flags, and architecture
• .gitignore updated
• SEO meta tags in index.html

What's Changed

• feat: add benchmark suite, Prometheus metrics, alert thresholds, and … by @codewithwest in #1

New Contributors

• @codewithwest made their first contribution in #1

Full Changelog: v0.1.0...v0.2.0

What's Changed

• feat: add benchmark suite, Prometheus metrics, alert thresholds, and … by @codewithwest in #1

New Contributors

• @codewithwest made their first contribution in #1

Full Changelog: v0.1.0...v0.2.0

LLM-Benchmarker

LLM-Benchmarker

A transparent HTTP proxy that intercepts LLM API calls (Ollama, llama.cpp, OpenAI-compatible), captures full request/response data, and displays real-time telemetry through a web dashboard.

• No code changes — sits between your client and LLM engine
• Captures everything — full request JSON and response body for every call
• Live metrics — TPS, TTFT, token count per request
• Searchable history — filter by endpoint, click any request for details
• Single binary — Go + embedded React UI, no runtime dependencies

Install

Pre-built binaries

Download from GitHub Releases:

Platform	File
Linux x86_64	llm-benchmarker-linux-amd64
Linux ARM64	llm-benchmarker-linux-arm64
macOS Intel	llm-benchmarker-darwin-amd64
macOS Apple Silicon	llm-benchmarker-darwin-arm64
macOS Universal	llm-benchmarker-darwin-universal

Build from source

git clone https://github.com/codewithwest/llm-benchmarker.git
cd llm-benchmarker

# Build everything in one step:
./start.sh

# Or manually:
cd ui && npm install && npm run build && cd ..
go build -buildvcs=false -o llm-benchmarker .

Requires Go 1.25+ and Node.js 22+.

Quick Start

Point it at a running Ollama instance:

# Ollama runs on the default port 11434
./llm-benchmarker

Open http://localhost:8080 in your browser. The dashboard loads — send a prompt through the chat panel. Every response is recorded as a card in the Requests tab.

Transparent intercept mode

Replace Ollama's port so existing clients are intercepted automatically:

# 1. Move Ollama to port 11435
OLLAMA_HOST=0.0.0.0:11435 ollama serve

# 2. Run benchmarker on port 11434, forwarding to 11435
./llm-benchmarker -port 8090 -intercept-port 11434 -target http://127.0.0.1:11435

# 3. Any client hitting port 11434 gets intercepted and proxied to 11435

CLI Flags

Flag	Default	Description
-port	8080	Port for the web UI and API
-target	http://127.0.0.1:11434	Upstream LLM engine URL
-intercept-port	0	If set, also listen on this port as a transparent intercept proxy
-db	benchmarks.db	Path to SQLite database file

Dashboard

Dashboard tab

• KPIs — active providers, average TPS, average TTFT, total requests
• TPS Trend — real-time chart of tokens-per-second over time
• Recent Activity — last 5 requests as clickable cards

Requests tab

• Full history of all intercepted requests
• Filter by endpoint
• Click any card to open the detail modal showing:
  • Telemetry (TPS, TTFT, tokens, model)
  • Formatted Request JSON
  • Formatted Response Body

Providers tab

• Register and monitor remote LLM engine nodes
• Each node is health-checked every 5 seconds
• Select an active provider for the chat panel

Chat panel

• FAB button (bottom-right) opens a streaming chat overlay
• Select a model and provider, type a prompt, hit Enter
• Response streams in real-time

Architecture

┌──────────────┐      ┌──────────────────┐      ┌──────────────┐
│   Browser    │ HTTP  │  llm-benchmarker │ HTTP  │    Ollama    │
│  (React UI)  │──────▶│   (Go Proxy)    │──────▶│  (Engine)    │
│              │◀──────│                  │◀──────│              │
└──────────────┘      │   ┌──────────┐   │      └──────────────┘
                      │   │ SQLite   │   │
                      │   │  store   │   │
                      │   └──────────┘   │
                      └──────────────────┘

1. Browser sends a prompt to the Go server
2. Server forwards the request to the upstream LLM engine
3. Response streams back to the browser in real-time
4. On completion, full request/response is saved to SQLite

The proxy handles both streaming and non-streaming responses, counting tokens by tracking newline-delimited JSON objects (streaming) and whitespace-separated words (non-streaming fallback).

Use Cases

• Benchmarking — measure TPS, TTFT across different models and hardware
• Auditing — inspect every prompt and response sent to your LLM
• Debugging — see exactly what your client sends and what the engine returns
• Monitoring — track usage patterns, response quality, latency trends

Development

# Frontend dev server (hot reload)
cd ui && npm run dev

# Build everything
./start.sh

# Run tests
go test ./internal/...

CI/CD

Push to main triggers:

• ci.yml — builds frontend, runs Go tests, compiles binary

Push a tag (v*) triggers:

• release.yml — cross-compiles for linux/darwin (amd64 + arm64), creates a macOS universal binary with lipo, publishes to GitHub Releases

License

MIT

Full Changelog: https://github.com/codewithwest/llm-benched/commits/v0.1.0

Full Changelog: https://github.com/codewithwest/llm-benched/commits/v0.1.0