A terminal benchmark dashboard for LLM inference servers, speaking the
OpenAI-compatible /v1/chat/completions API. Works against Ollama, vLLM,
llama.cpp's server, LM Studio, TGI, or any hosted OpenAI-compatible endpoint.
Streams responses over SSE and reports live throughput, latency percentiles, and time-to-first-token while requests are in flight.
Mid-sweep at 12k context and concurrency 16: completed phases accumulate above, in-flight requests and their streaming output below.
The same run at concurrency 1, where a single request's prefill and decode rates are easiest to read.
curl -fsSL https://raw.githubusercontent.com/chaiops/infrk/main/install.sh | shThis downloads the latest release for your platform, verifies its SHA-256
checksum against the published SHA256SUMS, and installs to ~/.local/bin
(or /usr/local/bin when that is writable).
If you would rather read the script before running it — a reasonable habit for anything piped to a shell — fetch it first:
curl -fsSL https://raw.githubusercontent.com/chaiops/infrk/main/install.sh -o install.sh
less install.sh
sh install.shKnobs, as environment variables:
INFRK_VERSION=v0.1.0 # pin a version instead of taking the latest
INFRK_BIN_DIR=~/bin # choose the install directory
INFRK_NO_VERIFY=1 # skip checksum verification (not recommended)Note that VAR=x curl ... | sh sets the variable for curl, not for the
script. Export it, or pass it to sh:
export INFRK_BIN_DIR=~/bin
curl -fsSL https://raw.githubusercontent.com/chaiops/infrk/main/install.sh | sh
# or
curl -fsSL https://raw.githubusercontent.com/chaiops/infrk/main/install.sh \
| INFRK_BIN_DIR=~/bin shEvery release attaches archives for:
| Platform | Target |
|---|---|
| Linux x86_64 | x86_64-unknown-linux-gnu |
| Linux x86_64 (static) | x86_64-unknown-linux-musl |
| Linux ARM64 | aarch64-unknown-linux-gnu |
| macOS Intel | x86_64-apple-darwin |
| macOS Apple Silicon | aarch64-apple-darwin |
Grab one from Releases if you prefer not to run the installer. The musl build is statically linked, so it works on Alpine and on distros whose glibc predates the build machine's.
cargo install --git https://github.com/chaiops/infrkOr clone and build:
git clone https://github.com/chaiops/infrk
cd infrk
cargo build --release
./target/release/infrk --helpinfrk -m llama3.2:3b -n 200 -c 12
Against a non-default endpoint:
infrk --base-url http://localhost:8000/v1 --api-key "$KEY" \
-m mistral-7b -n 500 -c 32 --max-tokens 512
Compare several models in one run (each gets -n requests):
infrk -m llama3.2:3b,gemma3:4b,qwen2.5:7b -n 100 -c 8
Pass several -c values to measure the throughput-vs-concurrency curve. Each
level runs to completion before the next begins — they are never overlapped,
since concurrent levels would contend and corrupt each other's numbers.
infrk -m llama3.2:3b -n 50 -c 1,2,4,8,16,32
The TUI shows the current level in the header and keeps finished levels on screen; the report at the end marks the level with peak aggregate throughput and prints a per-model scaling table with speedup relative to the lowest level tested, alongside prefill throughput at each level. That's where you find the saturation point: the concurrency past which throughput flattens while latency keeps climbing.
Note -n is per model per phase, so -n 50 -c 1,4,8 with two models is
50 x 2 x 3 = 300 requests; adding -p 512,2048 doubles that again.
Pass -p (llama-bench's flag) to sweep prompt length and see how prefill cost
grows with context:
infrk -m llama3.2:3b -p 512,1024,2048,4096 -n 20
Prompts are synthesized to hit each target size. -p combines with -c into a
full matrix, context varying slowest so each context block reads as its own
concurrency curve:
infrk -m llama3.2:3b -p 512,4096 -c 1,8 -n 20 # 4 phases, 80 requests
The report then adds a context-scaling table — the reason to run this sweep:
gemma3:4b — context scaling at concurrency 2
Ctx tok TTFT p50 Prefill t/s TTFT/ctx x Decode t/s
512 278 391 1.00x / 1x 50.8
2048 1022 1598 3.67x / 4x 39.2
8192 4017 2223 14.44x / 16x 15.7
Read the TTFT/ctx x column: TTFT grew 14.4x for 16x the context, so prefill
is scaling slightly sub-linearly here as batching amortizes fixed overhead.
A ratio climbing faster than the context ratio is the signal to look for —
that means quadratic attention cost or a batch-size limit is biting.
Each prompt is filler text sized to the target token count, and every prompt begins with a unique random salt. That salt is not cosmetic. Servers with prefix caching (vLLM, SGLang, recent llama.cpp) reuse KV blocks for any shared leading prefix, and a cache hit skips the very prefill being measured — vLLM #20015 reports ~60% hit rates on nominally random benchmark data because a fixed preamble sat in front of the random part. Putting the salt first means no two requests in a run share even one cache block.
--seed makes a run reproducible; vary it between runs against the same server
if you suspect cross-run cache reuse (vLLM's own docs recommend the same, or
restarting the server between runs).
Token counts are close but not exact — filler averages ~1 token per word, so a
-p 2048 request lands around 2049 real tokens. The report's Prefill t column
shows what the server actually counted, not the target.
-p cannot be combined with --prompt-text or --prompt-file; a context
sweep synthesizes its own prompts, and silently ignoring either flag would be
worse than an error.
Cycle real prompts from a file, one per line:
infrk -m llama3.2:3b --prompt-file prompts.txt -n 200
--image benchmarks a vision-language model such as
dots.ocr / dots.mocr by sending a
synthesized document page with every request:
infrk -m model --image -n 50 -c 4
infrk -m model --image 1024x1400,1600x2200 -c 1,4 # sweep page size
Against a vLLM-served dots.mocr:
vllm serve rednote-hilab/dots.mocr --served-model-name model --trust-remote-code
infrk --base-url http://localhost:8000/v1 -m model --image 1024x1400 -c 1,2,4,8Image size replaces context size as the outer sweep axis, since for a vision model the image dominates prefill. The report then splits prefill into its image and text parts:
dots.mocr — image scaling at concurrency 4
Image Px Img tok Txt tok TTFT p50 Prefill t/s Decode t/s
672x896 0.60M 192 15 162 553 185.8
1064x1400 1.49M 475 15 320 728 117.7
Img tok is estimated from the resized dimensions; Txt tok is the
server-reported prefill minus that estimate. Here the image is ~97% of prefill,
which is what explains the TTFT — the instruction is nearly free.
--image-prompt takes a dots.ocr mode name or any literal text. The modes are
the project's own instruction strings, and they are not interchangeable for a
decode benchmark — ocr returns plain text while layout-all returns a JSON
object with a bbox per element, decoding far more tokens:
| Mode | Output |
|---|---|
ocr (default) |
Plain extracted text |
layout-only |
JSON: bbox + category per element, no text |
layout-all |
JSON: bbox, category, and text; tables as HTML, formulas as LaTeX |
infrk -m model --image 1024x1400 --image-prompt layout-all
infrk -m model --image 1024x1400 --image-prompt "Extract the invoice total."Each request gets a freshly rendered page: random document-like text (headings, indents, monetary figures) drawn with a bundled font, then scanner-style grain and specks. This is real OCR work, not a blank canvas, and it needs no input files.
Dimensions are snapped to a multiple of 28 px with dots.ocr's own smart_resize
rules (including its 3,136 and 11,289,600 pixel bounds), so the estimated token
count matches what the server will actually charge.
Every page is unique. vLLM can cache processed images, and a cache hit would
skip the vision encode being measured — the same hazard as prefix caching for
text. --seed keeps a run reproducible; vary it between runs against a live
server.
--image cannot be combined with -p/--n-prompt or --prompt-file; use
--image-prompt for the instruction.
| Flag | Default | Meaning |
|---|---|---|
--base-url |
http://localhost:11434/v1 |
API base (also INFRK_BASE_URL) |
--api-key |
empty | Bearer token (also INFRK_API_KEY) |
-m, --model |
required | Model(s), comma-separated |
-n, --requests |
200 |
Requests per model |
-c, --concurrency |
12 |
Max in-flight per model; comma-separated to sweep |
-p, --n-prompt |
— | Context sizes to sweep, in prompt tokens |
--prompt-text |
a short poem prompt | Literal prompt, instead of a context sweep |
--prompt-file |
— | Prompts from file, cycled |
--image |
— | Benchmark a vision/OCR model; sizes as WxH to sweep |
--image-prompt |
ocr |
Instruction sent with the image |
--seed |
0 |
Seed for synthesized prompts and images |
--max-tokens |
256 |
Cap on generated tokens |
--temperature |
0.7 |
Sampling temperature |
--timeout |
300 |
Per-request timeout (s) |
--no-stream |
off | Disable SSE streaming |
--no-report |
off | Skip the summary printed after the run |
--theme |
pantera |
Built-in name or path to a theme file (also INFRK_THEME) |
--list-themes |
— | Print built-ins and the Charmtone palette, then exit |
--json <path> |
— | Also write the report as JSON |
p preview · g graphs · i info · r restart · h help · q quit
A summary prints to stdout when the run ends (suppress with --no-report).
Colors are omitted automatically when stdout is not a terminal, so piping to
a file yields clean text. --json <path> writes the same measurements
machine-readably, including p99 latency and per-request tok/s that the text
table omits for width.
Ctx Conc Model Reqs Err Lat avg Lat p50 Lat p95 TTFT avg RPS Prefill t Prefil t/s Decode t/s
8192 1 main 10 0 15500 13825 24219 6555 0.06 82,127 530 10.4
8192 2 main 10 0 22710 22191 32947 8421 0.09 82,134 712 15.0
8192 4 main 10 0 38214 39745 57489 16357 0.10 82,139 830 16.5
8192 8 main 10 0 43766 49890 55853 23884 0.16 82,132 1336 27.2 ◄
8192 16 main 10 0 83576 82977 87455 52998 0.11 82,134 939 17.7
◄ peak decode throughput 27.2 tok/s at concurrency 8 (main)
main — decode throughput scaling at ctx 8192
1 ████████████ 10.4 dec t/s 1.00x prefill 530 t/s lat p50 13825ms
2 ██████████████████ 15.0 dec t/s 1.44x prefill 712 t/s lat p50 22191ms
4 ███████████████████ 16.5 dec t/s 1.58x prefill 830 t/s lat p50 39745ms
8 ████████████████████████████████ 27.2 dec t/s 2.60x prefill 1336 t/s lat p50 49890ms
16 ████████████████████ 17.7 dec t/s 1.70x prefill 939 t/s lat p50 82977ms
That run shows exactly what a sweep is for: decode throughput climbs to 27.2 tok/s at concurrency 8, then falls to 17.7 at 16 while p50 latency keeps rising to 83s. Past 8, the extra concurrency buys nothing and costs latency — concurrency 8 is this server's operating point at 8k context.
Theming follows the approach Charm uses in Crush:
a theme assigns semantic roles, not widget colors. You set primary,
bg_base, success, warning and so on; every panel, table, gauge, and
sparkline style is then derived from those roles. Re-skinning the dashboard
means changing a handful of role assignments, and no widget hardcodes a color.
Colors are named from the Charmtone
palette — Charple, Malibu, Julep, Pepper — so a theme reads as design
decisions rather than hex noise. --list-themes prints all 70 names.
| Theme | Look |
|---|---|
pantera |
Charmtone dark (default) — violet primary, mint accents |
obsidian |
Dark with cooler blues, for terminals where violet fights the background |
dawn |
Light; foreground/background roles swap and the subtle ramp inverts |
mono |
Monochrome, for fixed-palette terminals |
infrk -m llama3.2:3b --theme dawn
INFRK_THEME=mono infrk -m llama3.2:3b
Pass a path to a JSON file. It inherits from a built-in via extends and
states only the roles it changes:
{
"name": "infrk-neon",
"extends": "obsidian",
"roles": {
"primary": "Pony",
"success": "Ice",
"info": "Mochi",
"keyword": "Zest"
}
}infrk -m llama3.2:3b --theme ./neon.json
Each role takes a Charmtone name (case-insensitive) or a #RRGGBB literal.
Unknown roles and unknown color names are hard errors naming the offending
key, rather than being silently ignored.
Brand: primary secondary accent keyword
Text: fg_base fg_subtle fg_more_subtle fg_most_subtle
Surfaces: bg_base bg_most_visible bg_less_visible bg_least_visible separator
Contrast pair: on_primary
Status: destructive error warning warning_subtle attention busy
info info_more_subtle info_most_subtle success success_more_subtle
success_most_subtle
Two roles carry meaning specific to this tool: prefill metrics render in
info and decode metrics in success, which is what keeps the two phases
distinguishable at a glance. Elapsed-time columns escalate through
success_more_subtle → warning → destructive as a request ages.
Inference has two distinct phases and they are reported separately throughout, because they scale differently and are bound by different hardware limits — prefill is compute-bound, decode is memory-bandwidth-bound.
- Prefill — processing the input prompt, ending at the first output token.
Prefill t/s =
prompt_tokens / TTFT. - Decode — generating output tokens, one at a time, after prefill.
Decode t/s =
output_tokens / (latency - TTFT).
Because the split is at TTFT, the two rates never mix: no prefill time is charged to decode, and no decode time to prefill.
- TTFT — time from request send to the first content delta. This is the prefill duration.
- Latency — full wall-clock request duration, prefill plus decode.
- Per-request vs aggregate — the per-model table's
Decode t/sis the average single-request rate, which stays roughly flat as you raise concurrency. The report'sDecode t/scolumn is the aggregate across all concurrent requests, which is what climbs. That aggregate number is the one to watch when tuning-c. - Recent RPS — over a 10-second sliding window, versus Overall RPS across the whole run.
Both counts come from the API's usage object — prompt_tokens for prefill,
completion_tokens for decode. Most servers send usage only in the final
stream chunk, so prefill totals update when each request completes rather than
live.
When a server omits usage entirely, decode tokens are counted as one per
streamed delta and prefill is estimated at ~4 characters per token. The
estimate is approximate; trust it for relative comparisons between levels, not
as an absolute token count.
With --no-stream there is no first delta to observe, so TTFT equals total
latency, prefill and decode rates are not meaningfully separable, and the live
preview stays empty.
infrk owes its shape to two projects:
- llama-bench
(llama.cpp) — the model for the benchmarking approach: the prefill/decode
split, the
-pcontext-size sweep with comma-separated values, and reporting a matrix of parameter combinations rather than a single number. - ollama_bench by @dkruyt — the inspiration for the live TUI: the concurrent-request dashboard, the in-flight request table, and the live token preview panes.
Vision benchmarking follows dots.ocr
by rednote-hilab: infrk uses its instruction prompts verbatim and reimplements
its smart_resize sizing rules so the reported image-token estimates match what
the model actually processes.
Theming follows the semantic-role approach Charm uses in Crush, with colors from Charm's Charmtone palette.
Synthetic document pages are rendered with Liberation Mono (Red Hat / Google),
bundled under the SIL Open Font License — see
assets/LiberationMono-LICENSE.txt.