FAQ

FAQ

English | Русский | 中文

Why do the expert weights go to host RAM instead of "just fitting"?

A 384-expert × 69-layer model has ~118 GiB of expert weights per TP rank. Four 96 GB cards give ~384 GB total, but you also need KV cache, CUDA graphs, activations and reserve. With no NVLink the only sane offload path is PCIe to pinned host RAM (UVA). The expert balance decides what stays resident.

What is UVA?

Unified Virtual Addressing: the GPU can read pinned host memory directly over PCIe without an explicit copy kernel. run-tp4-uva.sh launches vLLM with --offload-backend uva. Cold experts are UVA-resident; hot experts are real device memory.

Why is my GPU memory bandwidth not the bottleneck?

It often is not: cold hits are PCIe-bound (this host: PCIe 4.0 slots, ~25 GB/s per card). The planner minimises cold hits; dynamic mode pushes the hot hit-rate up further. Watch the hot/cold hit-rate log line.

GLM-5.3-Flash does not start on vLLM — what is wrong?

Known blocker. glm5_next uses rope-free MLA (qk_rope_head_dim=0) and vanilla vLLM fails on SM120 (RTX PRO 6000): assert "pe_dim must be 64 for fp8_ds_mla" (vLLM issues #53963, #55773). Fix candidates PR #55277 / #55778 / FlashInfer #4802 are not merged upstream yet. The recipe documents the blocker honestly; the balance math is ready and tested — the launch will work as soon as upstream lands the fix.

Do I need NVLink?

No. All four recipes are designed for PCIe-only TP4. NUMA-aware placement (HOTSPLIT_NUMA_POLICY = preferred / bind) helps keep host memory local to the GPU's PCIe root complex.

Static or dynamic?

• Static: predictable, zero runtime overhead, from your traffic histogram.
• Dynamic (default): adapts to live traffic, tiny cost per swap, better hit-rate over time. Measurable: hot/cold hit-rate per update.

Why does the launcher refuse to start (IMAGE / MODEL_REVISION / API_KEY)?

By design. Three parameters are mandatory:

• IMAGE must be pinned by @sha256:<digest> — floating tags are refused.
• MODEL_REVISION (40-char commit SHA) is required when the model is fetched by id.
• API_KEY is required even on 127.0.0.1 — only ALLOW_NO_AUTH=1 skips it, and only on loopback. The launcher will not "best effort" a misconfigured server. See the Quick Start.

What is the trust boundary (manifest, remote code, docker inspect)?

Three things to understand before the first run:

1. patches/MANIFEST.sha256 catches accidental edits and casual tampering, but the manifest lives in the same tree as the patches. An attacker with write access to the repo can replace both. Pin the git revision you reviewed (git rev-parse HEAD) and diff against it.
2. --trust-remote-code lets the model run its own Python. Pinning MODEL_REVISION prevents silent changes on later fetches but does not make the code itself safe. Review the model's remote code before the first run.
3. The API key travels in the container env (VLLM_API_KEY). Anyone who can run docker inspect can read it. Docker access is root-equivalent; restrict it.

Is the API exposed?

By default BIND=127.0.0.1 (loopback only). For LAN serving set BIND=0.0.0.0 and API_KEY — the launchers refuse a non-loopback bind without a key (ALLOW_NO_AUTH=1 is only accepted on loopback).

How do I know it actually works after launch?

scripts/check-hotsplit-tp4.sh <container> <dynamic|static> — every rank must report dynamic_hook=ACTIVE (dynamic) and hotsplit done. Then smoke-test /v1/chat/completions. Full measurement procedures (TTFT/TPOT, PCIe traffic, race checks under compute-sanitizer): see the recipe's README ("Verify" section) and the check scripts.

Can I contribute?

Yes — issues and PRs are welcome at sikkatu/recipes. The contract: state exactly what ran and on what, keep patch sha256 pins, list known limits, and mark honestly what is verified and what is not. If you publish anything derived from this repository, reference it (see the Attribution section of the README).