Post your benchmark here 📊 — datapoints, A/Bs, cold→warm curves

[JustVugg]

This is the home for benchmark datapoints (they used to live in Issues — new ones belong here).

Template (same rigor as the README table):

Hardware:  CPU / RAM / disk (+ iobench number) / GPU if any
OS:        (native? WSL2? filesystem)
Commit:    git rev-parse --short HEAD
Model:     container + MTP head variant (int8 sizes verified?)
Command:   exact env vars + coli invocation
Warm-up:   cold? how many runs? median?
Result:    tok/s · expert hit % · RSS · profile split if you have it

What makes a datapoint great: an A/B (one knob changed), a cold→warm curve, or a config nobody has tried (see the README table for covered ground — first-of-a-kind hardware always gets a row). Negative results are welcome: "X did not help on my box" with numbers has saved this project from wrong turns at least four times.

Current table highlights to beat: 6.84 tok/s (6×5090 full residency) · 2.24 (M5 Max Metal tuned) · 1.23 (Gen5 NVMe + 5090) · 1.10 (Strix Halo) · 0.5 (native Windows warm).

[ebootheee]

Hardware:  Ryzen 9 9950X3D2 (16C/32T) / 121 GB DDR5 / Samsung 9100 PRO Gen5 NVMe -- iobench 11.48 GB/s O_DIRECT / 
  2x RTX 5090 (Gen4 x8 + x4)
  OS:        Ubuntu native, kernel 7.0.0-27, ext4 (no WSL2)
  Commit:    a032299 = 6d3ed7e + the three patches now in PR #270
  Model:     GLM-5.2 int4 (mateogrgic container), int8 MTP heads byte-verified
  Command:   MTP=0 DIRECT=1 PIPE=1 COLI_PIPE_BLOCK=1 PIPE_WORKERS=16 PREFETCH=1 \
             CUDA_DENSE=1 COLI_CUDA_ATTN=1 CUDA_RELEASE_HOST=1 COLI_NO_OMP_TUNE=1 \
             PIN=<model>/.coli_usage PIN_GB=55 COLI_CUDA_BATCH=2 \
             ./coli run --auto-tier --ram 92 --gpu 0,1 --vram 52 --repin 16 --ngen 1024
  Warm-up:   .coli_usage at ~3.5M selections; back-to-back runs, both reported
  Result:    2.2554 / 2.2505 tok/s @ 1024 tok · hit 79.0-79.8% · RSS ~82 GB
             profile: disk 149s service (0 wait) / matmul 117s / attention 69s

A/Bs from the same box, one knob each:

• PIPE=1 (workers were set, pipe was off -- see the implication fix in pipe: blocking pipe_wait (COLI_PIPE_BLOCK=1) + PIPE_WORKERS implies PIPE=1 #270): 1.98 -> 2.16 @ 192
• COLI_CUDA_BATCH=2: closed 2.17 -> 2.2554 @ 1024
• COLI_NO_OMP_TUNE=1: 0.41 -> 1.29 on stock main ([Bug]: ~3x throughput regression on x86+CUDA since 92011de — OMP hot-spin tuning vs new threads (I4_ACC512 exonerated; COLI_NO_OMP_TUNE=1 recovers and then some) #159) -- load-bearing
• negative: --ram 96 regressed (memory pressure), OMP team-size tuning never beat spin-off,
  and a 2-drive shard split LOST 5% (the async pipe already hides disk)

Research row (routing approximation, NOT for the main table): ROUTE_J=2 ROUTE_M=1 cache-aware routing -> 3.09 tok/s @ 92% hit, 15.3% expert substitution, MMLU-200 A/ indistinguishable from true routing (details in #161).

Trail: #120 -> #160 -> #215 -> PR #270. Won't catch Zac's 6.84 with two cards and a disk, but per-GPU it's a fight. :)

[VincentMarquez]

DGX Spark / GB10 — stock top-8 vs CACHE_ROUTE (+ O_DIRECT disk)

Two-tier discipline: stock full top-8 vs experimental CACHE_ROUTE. Metric is decode tok/s (generation window), not overall wall tok/s.

Hardware:  NVIDIA DGX Spark · GB10 (sm_121) / 20× ARM (X925+A725) / 121 GB unified
           local NVMe · iobench 4.25 GB/s buffered · 9.69 GB/s O_DIRECT (2.3×)
OS:        Ubuntu 24.04 · Linux 6.17 · aarch64 · native (not WSL)
Commit:    CACHE_ROUTE via PR #199 (62419af); engine stack as measured in notes
Model:     GLM-5.2 int4 (colibrì snap) · full K=8 · MTP off for speed cells
Command:   DIRECT=1 + CUDA unified/dense path · large pin/LRU
           Stock:  CACHE_ROUTE=0
           Exp:    CACHE_ROUTE=1 ROUTE_J=2 ROUTE_M=12  (optional ROUTE_AGREE=1)
Warm-up:   session warm LRU / multi-turn chat with :reset; timed PROFILE WARM=160 NGEN=48
Result (decode tok/s · full K=8):

  Tier A stock routing:     2.08–2.39 decode · hit ~82%
  Tier B CACHE_ROUTE early: 3.33 decode · hit 97% · swap 14%   (chat ladder peak)
                            2.67 decode · hit 93.5% · swap 15% (timed one-shot)
  Tier C later CUDA stack + CR (not CR alone): 5.14–6.17 decode band
                            fair CR on/off same stack: 2.98 → 5.14

Disk lever: DIRECT=1 / O_DIRECT 9.69 GB/s vs 4.25 buffered — flag louder on real NVMe;
            weaker lever on slow/VHDX-backed disks (measure both).

Why two tiers

Tier	What it isolates
A	Leaderboard-comparable stock full-k8 routing
B	CACHE_ROUTE alone on early stack (~2.4 → 3.33)
C	CR plus MLA/fuse/device-tier — do not call this “CR alone”

Quality A/B (limited — not full coli bench)

Protocol: SCORE log-likelihood · HellaSwag · n=20 · seed 1234 · same request file · full K=8.

Cell	acc_norm	acc	notes
CACHE_ROUTE=0	80% (16/20)	40%	stock
CACHE_ROUTE=1 J=2 M=12	70% (14/20)	35%	−10 pp
Same normalized pick	85%		2 flips stock✓→CR✗

Caveat: n=20 only. Primes multi-turn still produced correct first-20 list (eyeball). Prefer larger greedy-token agreement set for ROUTE_J/M defaults. Supports default off.

Vs EXPERT_BUDGET (blind drop): CR is the principled middle path — substitute toward resident experts inside top-M (arXiv:2412.00099), not drop by gate weight alone. Connects to expert-specialization (#207): which experts are safe to swap.

Links

• Notes / numbers: https://github.com/VincentMarquez/glm52-gb10-colibri
• CACHE_ROUTE PR: routing: opt-in CACHE_ROUTE max-rank (pin∪LRU prefer) #199
• Mechanism comment: Benchmark: NVIDIA DGX Spark (GB10) 121GB — 2.4 tok/s full-k8; 3.33 with experimental CACHE_ROUTE (Potential 4.2toks soon) #161 (comment)
• Quality snapshot JSON in that repo under results/

Happy to extend with greedy-token agreement on a small reasoning set (not just primes) when the box is free — that should help tune ROUTE_J / ROUTE_M.

[VincentMarquez]

Independent repro — thank you (@ebootheee)

This is exactly the cross-arch confirmation that was needed.

	GB10 (unified, our notes)	x86 dual 5090 (your box)
Stock full-k8	~2.39 decode	2.255 (1024-tok window)
J=2 M=12	3.33 (~+39%)	3.09 (~+42%)
Substitution	~14%	15.3%
Quality	HellaSwag n=20 canary only	MMLU-200: 59% vs 62%

Quality read: agree 100% — at n=200, ±~3.5 pp noise means no detectable difference, not “CR helps.” That is already a strong statement: ~15% expert substitution did not measurably dent MMLU. Full bench / --limit 1000 would be welcome as the real claim gate.

Discipline: also agree these stay off the main true-routing table (research rows only). Same two-tier rule we use on Spark: stock top-8 vs experimental CACHE_ROUTE; never mix Tier-C CUDA stacks into a “CR alone” row.

Upstream status: the routing-side path is already landed as an official opt-in (default off) in PR #199 (CACHE_ROUTE / ROUTE_J / ROUTE_M, optional ROUTE_AGREE + swap% on the stats line). No need for fork magic if you’re on a post-#199 tree — reimplementation from the write-up is still a great independent check.

@JustVugg — +1 to “research flag” framing: big README warning, substitution rate on the result line, benchmark rows separated. Happy to help fold any remaining docs/flags with the #215 / #270 stack if useful.

Notes updated: https://github.com/VincentMarquez/glm52-gb10-colibri (QUALITY + NUMBERS).

[woolcoxm]

Windows 11 native — GLM-5.2 744B int4 — RTX 5070 Ti + Core Ultra 9 185H + 32 GB RAM

Setup: MinGW-w64 build, NVMe Gen4, CPU-only AVX-VNNI matmul + CUDA dense/attention offload via runtime DLL (coli_cuda.dll).

Decode throughput — progressive optimization (32-token decode, DRAFT=0)

step	config added	tok/s	hit%	expert-disk	decode (32 tok)
0	stock EXPERT_BUDGET=4	0.33	9%	65.9s	97.4s
1	+ PIPE=1 (async expert load)	0.34	—	—	93.4s
2	+ RAM_GB=28 (larger cache cap 1→3)	0.39	—	—	81.5s
3	+ PILOT_REAL=1 (cross-layer prefetch)	0.42	—	—	77.1s
4	+ DIRECT=1 (O_DIRECT / FILE_FLAG_NO_BUFFERING)	0.63	72%	21.5s	50.6s
5	+ CUDA_DENSE=1 COLI_CUDA_ATTN=1 (GPU attention)	0.72	75%	18.4s	44.5s
6	+ COLI_CUDA_PIPE=2 (GPU resident pipeline at decode)	0.85	57%	18.2s	37.8s
7	+ ws_b cache reserve fix (cap 3→4, matches budget)	1.03	73%	12.4s	31.0s
8	+ CACHE_ROUTE=1 ROUTE_J=2 ROUTE_M=12 (cache-aware routing)	1.07–1.43	83%	8.4s	22–30s

Total: 3.2–4.3× over stock. Disk I/O reduced 8× (65.9s → 8.4s).

Full config

EXPERT_BUDGET=4 PIPE=1 RAM_GB=28 PILOT_REAL=1 DIRECT=1
COLI_CUDA=1 CUDA_DENSE=1 COLI_CUDA_ATTN=1 COLI_CUDA_PIPE=2 CUDA_EXPERT_GB=0
CACHE_ROUTE=1 ROUTE_J=2 ROUTE_M=12

Per-token breakdown (accurate, from non-pipe2 run to avoid overlap artifacts)

phase	ms/token	%	notes
Expert matmul (CPU serial)	386	36%	gate+up+down for ~5 experts/layer × 75 layers
Other (rmsnorm + routing + residual)	334	31%	156 rmsnorms + 75 router GEMMs + CACHE_ROUTE logic — mostly moved to GPU via pipe2
Expert disk I/O (PIPE-overlapped)	241	23%	~50 misses/token × 19MB, 82% hit rate
Attention (GPU)	111	10%	MLA absorb + projections on RTX 5070 Ti

Key findings

1. CACHE_ROUTE is the single biggest win (+37%): steers the MoE router to prefer cache-resident experts. route_agree=94.6% confirms minimal quality cost. This is the right fix for GLM-5.2's near-uniform routing (237 of 256 experts used per layer across 75 tokens — no "hot" experts to pin).

2. pipe2 resident pipeline unblocked by token-exactness proof: CUDARES = CUDAOFF = DIRECT token-for-token across 4 prompts. The resident pipeline is a pure execution-location change, not a math change. (CUDA greedy may differ from CPU on near-tie argmax — FP accumulation order drift, not a bug.)

3. The "other" bucket is the new frontier: 31% of decode is untracked CPU glue (rmsnorm, residual, router, top-K, gather/scatter). On a warm cache this is bigger than disk I/O. Fusing rmsnorm+residual and tightening the router/top-K path are hardware-independent wins.

4. Windows compat layer was essential: compat_fadvise WILLNEED cache-warmer (replaces no-op), PIPE default ON for Windows, FILE_FLAG_NO_BUFFERING for expert reads.

What didn't work (documented for the record)

• Windows mmap (COLI_MMAP): RSS bloat collapses cache budget (Windows-vs-Linux page-cache difference)
• CUDA expert matmul (CUDA_EXPERT_GB): sync-stall regression (16GB VRAM too small for expert tier)
• Async attention dispatch: overlap window too small (~0.1ms CPU vs 1.2ms GPU)
• ROUTE_M=24: faster but quality degrades (KL 2.78 vs 0.92)
• I4S=1 (IDOT at S=1): quantization overhead exceeds VNNI speedup
• pc_b reserve removal: memory pressure at 24GB RSS drops hit rate

Hardware details

component	spec
GPU	NVIDIA RTX 5070 Ti (17.1 GB VRAM, sm_120 Blackwell)
CPU	Intel Core Ultra 9 185H (6P+8E+2LP, 22 threads, AVX-VNNI)
RAM	32 GB DDR5
SSD	NVMe Gen4
VRAM usage	9.7 GB dense + 0.5 GB pipe2 scratch = 10.2 GB of 17.1 GB
RSS	~21 GB of 32 GB (16.7 dense + 5.8 expert cache + overhead)

Full diagnostic report with raw profiles, memory hierarchy analysis, and ranked optimization recommendations: DIAGNOSTIC.md on the windows-optimizations branch. PR #274, issues #256/#258/#273.

[allan-56]

3.6 tok/s on AMD EPYC 7713, 96 cpu avx2, 1 TB RAM for model, 1 TB RAM for full expert loaded

Hardware:  2× AMD EPYC 7713 64-Core Processor (Zen 3), 256 threads / 128 cores, 2 NUMA nodes, AVX2 kernels (idot: avx2),  4096 GB DDR4 (32×128GB Samsung DIMMs, DDR4-3200, running DDR4-2667), bare metal (2TB used), Micron 7450 NVMe SSD ( Cached read: 16.2 GB/s, Buffered read: 2.0 GB/s)
OS:        Ubuntu 22.04.3, 5.15.0-181 kernel, GCC 11.4.0
Commit:    62419af
Model:     mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp
Command:   OMP_NUM_THREADS=96 DRAFT=3 OMP_PROC_BIND=spread OMP_PLACES=cores COLI_NO_OMP_TUNE=1 PIN=~/colibri/models/glm52/.coli_usage PIN_GB=1000 COLI_MODEL=~/colibri/models/glm52 numactl --interleave=all ./coli run --ram 1000 --gpu none --temp 0 "test"
Warm-up:   .coli_usage
Best result:    3.60 tok/s

Run	Threads	NUMA	Bind	Draft	Warm RAM	Hit rate	Prefill	Decode tok/s	MTP acceptance	Notes
1	64	interleave	spread	1	213GB	93.5%	4.18s	1.57	65%	cold-ish cache
2	128	interleave	spread	1	213GB	100%	1.55s	3.55	65%	stable
3	256	interleave	spread	1	213GB	100%	25.97s	~0.15	65%	bad regression
4	96	interleave	spread	1	213GB	100%	1.58s	3.60	65%	best
5	112	interleave	spread	1	213GB	100%	1.51s	3.54	65%	similar
6	112	interleave	close	1	213GB	100%	1.49s	3.55	65%	same
7	96	interleave	close	1	213GB	100%	1.58s	3.58	65%	same
8	96	node0 only	spread	1	213GB	100%	1.32s	3.29	65%	slower decode
9	96	interleave	spread	3	227GB	97%	1.56s	1.56	38%	draft hurts

Best run:

== GLM C engine (glm_moe_dsa), cache=8 experts/layer | experts@8-bit dense@8-bit | idot: avx2 ==
loaded in 6.42s | resident dense: 9912.75 MB | layers=78 experts=256 | MTP ACTIVE (draft=1)
[MTP] active: native speculative decoding (draft=1)
[PIN] placement: 0 VRAM + 12088 RAM expert (228.6 GB warm) in 38s da /home/ubuntu/colibri/models/glm52/.coli_usage
[USAGE] expert history: 328416 selections (/home/ubuntu/colibri/models/glm52/.coli_usage)
[RAM_GB=1000.0] cap raised 8->256: budget allows it (projected peak 618.0 GB; set CAP_RAISE=0 to disable)
[stop] 3 stop tokens: 154820 154827 154829
prompt: 7 tokens | generating up to 1024 (EOS stop=154820) | n-gram draft=1
[gMASK]<sop><|user|>test<|assistant|><think></think>PROFILO PREFILL (1.58s):
PROFILE: expert-disk 0.000s service / 0.000s wait | expert-matmul 1.187s | attention 0.222s (including kvb 0.016s) | lm_head 0.007s | other 0.165s
ATTENTION: projection/RoPE 0.080s | score-softmax-value 0.010s | output projection 0.115s
I am working perfectly! How can I help you today?

*(If you
[t=16  RSS 227.26 GB  hit 100%  3.44 tok/s  1.50 tok/fw]
 are just testing the system, everything looks good! Let me know what else you
[t=32  RSS 227.26 GB  hit 100%  3.60 tok/s  1.63 tok/fw]
 would like to do.)*
---
prefill 7 tokens in 1.58s | decode 38 tokens in 10.71s (3.55 tok/s) | expert hit rate 100.0% | RSS 227.26 GB
experts loaded/token: 734.3 (per-layer 9.79 across 75; baseline topk=8) | TOPK=0 TOPP=0.00
speculation: 1.65 tokens/forward (23 forwards per 38 tokens) | MTP acceptance 65% (15/23)
PROFILE: expert-disk 0.000s service / 0.000s wait | expert-matmul 6.411s | attention 2.527s (including kvb 0.000s) | lm_head 0.000s | other 1.776s
ATTENTION: projection/RoPE 0.659s | score-softmax-value 0.804s | output projection 1.061s

Ref: #234

Please let me know if you think I can tune this any further.

[ZacharyZcR]

6× RTX 5090 full stack, current dev: 5.2 tok/s end-to-end through HTTP/SSE, TTFT 0.43 s

The strictest measurement we've taken of this rig: client-side, streaming over the OpenAI SSE API through an SSH tunnel — every layer of the real serving stack included, nothing engine-side about it.

Hardware: dual Xeon Silver 4510 (24C/48T, AVX-512 VNNI, 2 NUMA nodes) · 251 GB DDR5 · 6× RTX 5090 32 GB · NVMe
Engine: current dev (post #313/#274/#294/#232) · Config: CUDA_EXPERT_GB=auto (176.6 GB VRAM tier, 9,280 experts across 6 devices) + PIN_GB=all (rest pinned in RAM — disk 0) + CUDA_DENSE=1 COLI_CUDA_ATTN=1 COLI_CUDA_PIPE=2 COLI_CUDA_TC_W4A16=1 + COLI_NUMA=1 (#313, first production run) + OMP_NUM_THREADS=24 OMP_PROC_BIND=spread + DRAFT=0

Six consecutive 96-token turns (client-measured SSE chunk timing):

turn	TTFT	decode
1 (fresh prompt)	2.74 s	3.94 tok/s
2–3 (KV prefix hit, warming)	0.47 s	4.55 → 4.98
4–6 (plateau)	0.43–0.55 s	5.13 → 5.22 → 5.24 tok/s

Three things the curve shows:

1. The REPIN warm-up is textbook: 3.94 → 5.24 over six turns as the adaptive tier swaps hot experts into VRAM slots (every 16 tokens).
2. KV prefix reuse makes TTFT ~free on continued conversations: 0.43 s, with the new [API] KV slot 0 prefix 19/24, prefill 5 diagnostic (serve: emit the [API] KV prefix-reuse diagnostic in mux mode (#153) #248) confirming only the delta prefills.
3. This is ~20% above the same rig on last week's engine at the same E2E protocol-stack measurement — the delta is the accumulated dev work (resident pipeline at decode, NUMA interleave, kernel consistency). Engine-side decode-only on this rig remains 6.0–6.8 (README experiment log); the gap between that and 5.2 is the honest cost of HTTP+SSE+tunnel.

One bug found while measuring: after the recent telemetry merges, the server's colibri.stats response field comes back empty (the SSE side stream works; the JSON attach doesn't) — will send the fix.

[ZacharyZcR]

Full-residency decode on 6× RTX 5090 is CPU-bound, not GPU-bound — a direct overlap-window measurement

Following up on the earlier 5.2 tok/s datapoint with a mechanism result that surprised me. I instrumented moe() at decode to split the per-turn expert time into the GPU-group window vs the CPU rows, running them concurrently (issue the VRAM-resident experts' grouped GPU calls, then compute the RAM-tier/miss rows on the CPU while the GPUs work, then collect). Over a 512-token greedy run, full residency (CUDA_EXPERT_GB=auto PIN_GB=all, 9,276 experts in VRAM, 1,604 in RAM):

overlap window: pack+issue 3.20s | cpu-rows 44.55s | take(sync+acc) 0.63s

The GPU side of a token's experts costs ~3.8 s of wall time; the CPU side costs 44.5 s. The six 5090s finish their 9,276-expert share and then idle, waiting on the CPU to grind through the ~90 RAM-tier experts each token routes to (AVX-512 VNNI, 1–4 rows each). Independent confirmation from SM sampling during decode: 1.1% mean utilization. The cards are not the bottleneck and never were — at S=1 there simply isn't enough GPU work, and what work exists is dwarfed by the CPU tail.

Consequences for where speed lives on this class of host:

1. Concurrency and speculation don't help (measured separately: aggregate throughput is flat 1→8 streams because independent streams' expert sets barely intersect; speculation's S=4 verify batch is a net 3× loss). The MoE sparsity that makes the model fit is the same thing that starves small-batch GPU utilization.
2. The lever is the RAM tail, not the GPU. Every expert you can move from RAM into VRAM is one the CPU no longer computes. Two ways there: a streaming-weight GPU path for the RAM experts (upload-compute-discard on the idle cards — PCIe cost vs the 44 s CPU cost, worth measuring), or a smaller expert container so the whole set fits VRAM. The latter is why the int2/2.5-bit lattice work (Rotation-preconditioned quant for streamed experts (int2/int3 quality) worth considering? #81) matters for speed, not just disk: a ~200 GB container puts all 19,456 experts on six 32 GB cards and takes the CPU out of the decode loop entirely.

The COLI_GROUP_ASYNC=1 overlap that produced this measurement is up as #342 (opt-in, +6% end-to-end — small precisely because it hides GPU time that was never the bottleneck; its value is the diagnostic).