Repository navigation
Home
You have four NVIDIA RTX PRO 6000 Blackwell Workstation Edition cards (96 GB GDDR7 each), no NVLink, and one question: how do I actually serve a large open-weight MoE model on this box? This wiki is the answer — everything you need to launch vLLM with the expert-balance (hotsplit) patches from this repository, step by step.
A single 384-expert model does not fit its experts in 384 GB of GPU memory and leave room for KV cache — not even on four 96 GB cards. The trick these recipes use: keep the hot experts (the ones your traffic actually routes to most) resident in GPU memory, and read the cold experts from pinned host memory over PCIe on demand. That is the expert balance, and the planner computes it from your own routing histograms — not a guess.
The result: big MoE models (100B+ parameter class) run on a workstation that costs a fraction of an enterprise-class server, at usable decode throughput.
- Four ready recipes — MiMo-V2.6-Pro, DeepSeek-V4.1-Flash, GLM-5.3, GLM-5.3-Flash — each with launch scripts, preflight checks, expert-balance calculator and verification tooling.
-
One shared planner (
hotsplit_plan.py) — greedy hot-set by routing share per byte, unit-tested, byte-identical across all four recipes. - Honest numbers — every claim comes with the method it was measured by. This hardware is PCIe 4.0 host slots (the cards are Gen 5, the platform is not), DDR4, GDDR7 VRAM. Numbers here respect that.
- Safety — API key required for any non-loopback bind, patch manifest verification before launch, no credentials in the tree.
- Quick Start — from a fresh box to a serving endpoint.
- How Expert Balance Works — the idea, the math, the knobs.
- Models & Geometry — which model fits which hot budget.
- FAQ — GLM-5.3-Flash blocker, UVA, NUMA, CUDA graphs, and more.
Owners of 4× RTX PRO 6000 workstations who want to run real models, not benchmarks. If you have this hardware and you have ever thought there must be a way to use all 384 GB — this is it.
English
Русский
中文