Models and Geometry

Models & Geometry

English | Русский | 中文

How much room each model needs on 4× RTX PRO 6000 (96 GB GDDR7, TP4). Read this before you set a hot budget — the previous version of this page published a hot-set size that did not fit on one card; the arithmetic below is what the planner actually does.

What the planner computes

balance.json gives layers, experts, topk, cell_mib_full (bytes of one full expert, one layer). The bank is:

full bank (all ranks) = layers × experts × cell_mib_full
per-rank bank (TP4)   = full bank / 4          # each expert is sharded across 4 ranks

The hot budget HOTSPLIT_HOT_GIB is per rank, in GiB of rank-local rows. The planner fills it greedily by routing share per byte. The number of hot experts is an output, not an input.

Per-model numbers

Values are computed from the model's config.json; cell_mib_full is rounded and the real tensor sizes are measured at load (see the recipe README). Treat these as planning numbers, not measurements.

MiMo-V2.6-Pro (reference recipe)

Property	Value
Layers × experts, top-k	69 × 384, top-8
Full bank (all ranks)	~492 GiB
Per-rank bank (TP4)	~123 GiB
Usable VRAM per card	~89 GiB (96 GB minus driver/context overhead)

The whole per-rank bank does not fit on one card. Hot + cold split is mandatory here. A safe starting hot budget leaves room for KV cache and CUDA-graph buffers on top of the hot rows:

hot budget ≈ usable VRAM − KV cache − reserve

For example with 8 GiB KV + 2 GiB reserve: 89 − 8 − 2 ≈ 79 GiB → the planner picks as many top experts as fit, roughly 240–250 of 384 at this geometry. The exact count depends on the routing histogram — run scripts/check-hotsplit-tp4.sh after a dry-run (HOTSPLIT_DRYRUN=1) and read the hotsplit plan line: it prints the budget and the resulting hot-cell count.

Earlier versions of this page cited "295 of 384 experts" as a default hot set. That number is an anchor from a previous single-GPU measurement (a 7360-slot pin-hot-experts split), not a per-rank VRAM budget. At 19 MiB × 295 ≈ 94 GiB per rank it does not fit alongside KV cache on a 96 GB card. Do not use it as a target.

DeepSeek-V4.1-Flash

Property	Value
Layers × experts, top-k	40 × 384, top-6
Full bank	~280 GiB
Per-rank bank (TP4)	~70 GiB
Suggested hot budget	52 GiB (ladder 47 / 52 / 57 GiB)

Fits with headroom: 52 GiB hot + KV + reserve is comfortable on 89 GiB.

GLM-5.3

Property	Value
Layers × experts, top-k	75 × 256, top-8
Full bank	~380 GiB
Per-rank bank (TP4)	~95 GiB
Suggested hot budget	36 GiB (ladder 31 / 36 / 41 GiB)

Per-rank bank is close to the card limit — 36 GiB hot leaves VRAM for KV cache.

GLM-5.3-Flash (glm5_next)

Property	Value
Layers × experts, top-k	42 × 288, top-8
Full bank	~160 GiB
Per-rank bank (TP4)	~40 GiB
Suggested hot budget	31 GiB (ladder 26 / 31 / 36 GiB)

The whole bank fits in VRAM — hot 31 GiB leaves room for KV cache. ⚠ Blocked on vanilla vLLM on SM120 — see FAQ.

Choosing the hot budget — safe defaults

Situation	Budget
First run, unknown traffic	recipe default (the anchor in tp4.env.example)
Long contexts, many concurrent requests	lower the budget, give the room to KV cache
Short contexts, single stream	raise it, watch the hotsplit mem log line

Rules of thumb:

• HOTSPLIT_HOT_GIB empty = "use the GPU-resident expert bytes the stock placement used" — NOT "fit the whole bank". The stock placement never exceeds VRAM; the empty default is safe but conservative.
• HOTSPLIT_VRAM_RESERVE_GIB=2 is a floor, not a promise. KV cache is allocated by vLLM after model load; long contexts eat it. For 128K context, plan for tens of GiB of KV, not 2.
• Always verify after launch: bash scripts/check-hotsplit-tp4.sh <container> <dynamic|static> and the hotsplit plan line in the logs.