[Performance]: 2.82 tok/s — dual-socket Xeon 8368, 96 vCPU AVX-512 VNNI, 492 GB RAM, full expert residency (+ v1.0 serve-mode crash report)

[jellyfishmobile]

Commit

62419af (current main) — plus 97c756a (v1.0) for the serve bug section; perf reproduced on both

Hardware and storage

• 2× Intel Xeon Platinum 8368 (Ice Lake), 96 vCPU, 2 NUMA nodes — AVX-512 VNNI kernels active (idot: avx512-vnni)
• 492 GB DDR4 (16 channels total), QEMU/KVM guest
• Weights on ext4/LVM (virtio disk) — irrelevant after warm-up: planner reports 0.0 GB cold experts (352.9 GB warm in RAM)
• Model: mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp (383.7 GB, 144 shards)

This box fits the entire expert set in RAM — probably the closest thing in the wild to the README's "128–256 GB RAM + AVX-512 → ~5–15 tok/s" prediction row.

Software environment

Ubuntu 24.04, gcc 13.3.0, CPU-only build (no CUDA/BLAS). QEMU/KVM guest, kernel 6.8. Commits: perf measured on 97c756a (v1.0) and reproduced on 62419af (main); the serve bug section applies to 97c756a only.

Winning environment: OMP_NUM_THREADS=64 DRAFT=1 OMP_PROC_BIND=spread OMP_PLACES=cores COLI_NO_OMP_TUNE=1 PIN=<model>/.coli_usage PIN_GB=340, run under numactl --interleave=all, --gpu none.

Note: box co-hosts production vLLM services — colibri ran at nice -n 15; vLLM impact measured at +~10ms on short requests (negligible).

Build and benchmark commands

make glm   # gcc -O3 -march=native -fopenmp → idot: avx512-vnni

# benchmark generation (winning config):
OMP_NUM_THREADS=64 DRAFT=1 OMP_PROC_BIND=spread OMP_PLACES=cores \
COLI_NO_OMP_TUNE=1 PIN=/path/glm52_i4/.coli_usage PIN_GB=340 \
numactl --interleave=all ./coli run --gpu none --ngen 250 --temp 0 "<prompt>"

# quality:
./coli bench   # hellaswag/arc_challenge/mmlu, n=40 each (datasets+tokenizers pip-installed)

Results

All decode numbers: 250-token generations, temp 0, warm cache, PIN pre-placed from usage history. Full expert set resident in RAM (planner: 0.0 GB cold experts, 352.9 GB warm) — likely the closest real datapoint to the README's "128–256 GB RAM + AVX-512 → ~5–15 tok/s" prediction row.

| Config | tok/s |
|---|---|
| First run ever (cold) | 0.17 |
| 32 threads, interleave, PIN 340 GB, draft=3 | 1.64 |
| 64 threads, draft=3 | 2.39 |
| 96 threads, draft=3 | **0.54 — collapses** (oversubscription; box co-hosts vLLM) |
| 64 threads, **DRAFT=1** | 2.73 (MTP acceptance 46% → **82%**) |
| 64 threads, DRAFT=1, PROC_BIND spread | **2.82** |
| + `CACHE_ROUTE=1 ROUTE_J=2 ROUTE_M=12` (on 62419af) | 2.97 — only +6% here: hit rate already 99.8%, little for cache-aware routing to save; we reverted to exact routing |

## Why not the predicted 5–15 tok/s? Profiler: bandwidth + attention, not FLOPs

Representative 400-token run (draft=3):


expert-disk 12.3s | expert-matmul 131.3s | attention 98.8s | other 46.9s
ATTENTION: projection/RoPE 29.6s | score-softmax-value 51.2s | output projection 17.8s
experts loaded/token: 1198 (per-layer 15.98 vs baseline topk=8)  <- MTP verification union ~2x


- Disk is fully eliminated (4%), as the RAM row predicts — but decode is **not matmul-FLOP-bound**: expert GEMV streams weights from RAM at ~70 GB/s effective (NUMA interleave + random expert access). The memory system, not the kernel, is the wall.
- **Attention is 36% of decode** and the back-of-envelope model prices it at zero.
- MTP draft=3 loads ~2× experts per forward (verification union). On RAM-resident boxes **DRAFT=1 wins decisively** (82% acceptance, −17% expert traffic, +14% end-to-end). Suggestion: when the planner reports 0 GB cold experts, default DRAFT=1 or auto-tune depth from measured acceptance.

## Quality datapoint

`coli bench` (n=40 each): hellaswag 67.5 / arc 60.0 / mmlu 60.0 — mean acc_norm **62.5%**, an exact task-by-task replication of #108 on different hardware, supporting the protocol-artifact reading there. Generative spot-checks (multi-step math, spec-heavy code, JSON-constrained output, zh/en bilingual) at temp 0 were coherent and correct.

## Bug (v1.0 tag, 97c756a): `coli serve` crashes on every completion

`coli run` works, but any `/v1/chat/completions` via `coli serve` kills the engine:


missing tensor: model.layers.<N>.mlp.experts.-1.gate_proj.weight
[api] 127.0.0.1 - request failed: colibri engine exited unexpectedly


Layer varies per attempt; expert index is always **-1** (invalid router output in the serve dispatcher path only). Ruled out: DRAFT 0/1/3, `--topp/--topk 0`, `--auto-tier --gpu none --vram 0`, prompt length, memory limits. **Fixed between 97c756a and 62419af** — serve works on current main with identical env. A note for v1.0 users hitting `engine_error` might save them the debugging session.

Happy to run follow-up experiments on this box (thread sweeps, NUMA variants, dev features) — this hardware class seems to be the datapoint gap in the current benchmark set.

Baseline comparison

No response

[JustVugg]

Excellent report — and thanks for doing the legwork on the serve bug.

Perf: 2.82 tok/s on a 2×Xeon 8368 with the full set warm (0.0 GB cold, 352.9 GB in RAM), AVX-512 VNNI, under nice -n 15 alongside production vLLM — that's the closest thing in the wild to the README's "128–256 GB + AVX-512" row, exactly as you note. Please repost this in Discussions → Show and tell (#208) so it sits next to the EPYC 7713 (#234) and GB10 datapoints. One aside: you used PIN_GB=340 (an explicit cap), so you'd never have hit #229 — but I just fixed PIN_GB=all on dev to clamp to --ram instead of OOM-killing, in case you want to stop hand-tuning the number.

Serve crash (expert index -1, v1.0/97c756a): confirmed fixed between 97c756a and current main, matching your own bisect — the invalid router output in the serve dispatcher was resolved with the serve-path fixes. Nothing to do on main. I'll add a note for v1.0 users as you suggested.

I'd gladly take you up on the follow-up experiments — thread sweeps and NUMA variants on this class of box are exactly the gap. I'll ping you in #208 with a couple of dev knobs worth measuring.

[JustVugg]

Perf → Discussions #208; serve crash already fixed on main (reporter-confirmed bisect).

[biemster]

I have access to a very similar machine (2x Xeon Gold 6430 (sapphire rapids), 1TB DDR5), would the faster RAM on this machine make an interesting datapoint or will the difference be marginal? Also I can run the test bare metal and not as a KVM guest, although I don't know if that will make a difference.
I'll open a new [Performance] ticket if this is an interesting machine to test on after I figure out how to run the test.