Quick Start

Quick Start — 4× RTX PRO 6000

English | Русский | 中文

From a fresh Debian 12 machine to a serving endpoint. Read the three mandatory parameters first — the launcher refuses to start without them, and that is by design.

Target OS: Debian 12 (bookworm). Commands also work on Ubuntu 22.04/24.04 with apt.

The three things the launcher will not start without

Parameter	Why	How to get it
IMAGE	Must be pinned by sha256 digest — floating tags are refused	step 3
MODEL_REVISION	Required when the model is fetched by id (not from a local dir)	the model's commit SHA
API_KEY	Required even on 127.0.0.1 — only ALLOW_NO_AUTH=1 skips it, and only on loopback	openssl rand -hex 24

0. Check the machine

• 4× NVIDIA RTX PRO 6000 (96 GB GDDR7), nvidia-smi shows all four
• ≥ 1 TB RAM (cold expert banks live in host memory), ≥ 2 TB NVMe for model + container
• Debian 12, root or sudo

1. Install Docker + NVIDIA Container Toolkit

# Docker (Debian 12)
sudo apt-get update && sudo apt-get install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
  https://download.docker.com/linux/debian $(. /etc/os-release && echo $VERSION_CODENAME) stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update && sudo apt-get install -y docker-ce docker-ce-cli containerd.io

# NVIDIA Container Toolkit
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list > /dev/null
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker && sudo systemctl restart docker

# verify: must print GPU information inside a container
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

2. Get the recipes

git clone https://github.com/sikkatu/recipes.git ~/recipes
cd ~/recipes/rtx-pro-6000/mimo-v2.6-pro-hotsplit

All further commands run from this directory — no repeated cd guessing.

3. Pin the container image (mandatory)

The launcher refuses floating tags. Resolve the digest once and use it everywhere:

export IMAGE="vllm/vllm-openai:<tag>"   # use the tag your recipe README pins
# NOTE: the image is UNVERIFIED on target hardware -- hotsplit replaces model files
# inside vLLM, so the exact vLLM build matters. Treat the first boot as an experiment.
docker pull "$IMAGE"
export IMAGE="$(docker inspect --format='{{index .RepoDigests 0}}' "$IMAGE")"
echo "$IMAGE"   # must end in @sha256:<digest>

4. Get the model

Run these commands on the Linux host (the machine with the four GPUs), NOT inside a docker container. The model directory is a host path that the launcher later mounts read-only.

Two terms, two different things:

Name	What it is	Example
MODEL_ID	the model's address on Hugging Face	XiaomiMiMo/MiMo-V2.6-Pro-MOPD
MODEL_DIR	a local directory on this host holding the downloaded files	/models/MiMo-V2.6-Pro-MOPD
MODEL_REVISION	the exact commit SHA of the model version — used for both download and launch	f3a9... (40 hex chars)

MODEL_REVISION identifies the version. Getting the SHA does not download the model. Note: with MODEL_DIR the launcher mounts the directory and does not verify its contents against a SHA -- MODEL_REVISION then only records which version you downloaded from. For reproducible local runs, keep the revision string next to the files and verify integrity separately. With MODEL_ID the revision is enforced by the launcher at fetch time. the the download step below. Use the same SHA when you launch, so the server runs exactly the files you verified.

Option A — download the whole model repo to a local directory (recommended)

# 1. install the Hugging Face CLI in its own venv (not the system Python)
python3 -m venv "$HOME/.venvs/hf-download"
source "$HOME/.venvs/hf-download/bin/activate"
python -m pip install --upgrade huggingface_hub

# 2. model id and target directory
export MODEL_ID="XiaomiMiMo/MiMo-V2.6-Pro-MOPD"
export MODEL_DIR="/models/MiMo-V2.6-Pro-MOPD"

# 3. create the directory and let the current user write to it
sudo mkdir -p "$MODEL_DIR"
sudo chown "$USER:$(id -gn)" "$MODEL_DIR"

# 4. resolve the exact commit SHA of the current model version
export MODEL_REVISION="$(
  curl -fsSL "https://huggingface.co/api/models/$MODEL_ID" |
  python3 -c 'import json,sys; print(json.load(sys.stdin)["sha"])'
)"
if ! [[ "$MODEL_REVISION" =~ ^[0-9a-f]{40}$ ]]; then
  echo "ERROR: could not obtain a valid model revision SHA" >&2
  exit 1
fi
echo "Model:     $MODEL_ID"
echo "Revision:  $MODEL_REVISION"
echo "Directory: $MODEL_DIR"

# 5. check disk space BEFORE downloading (these models are hundreds of GB)
df -h "$(dirname "$MODEL_DIR")"

# 6. download the entire model repository at that exact revision
hf download "$MODEL_ID" \
  --revision "$MODEL_REVISION" \
  --local-dir "$MODEL_DIR"

# 7. verify the download (config, weight index, all shards, tokenizer, revision)
test -s "$MODEL_DIR/config.json" || { echo "ERROR: config.json missing" >&2; exit 1; }
find "$MODEL_DIR" -maxdepth 1 -type f -name '*.safetensors' -print -quit | grep -q . \
  || { echo "ERROR: no safetensors files found" >&2; exit 1; }
# every shard listed in the weight index must exist and be non-empty
if ls "$MODEL_DIR"/*.index.json >/dev/null 2>&1; then
  python3 -c '
import json, sys, pathlib
idx = next(pathlib.Path(sys.argv[1]).glob("*.index.json"))
shards = sorted(set(json.load(open(idx))["weight_map"].values()))
missing = [s for s in shards if not (pathlib.Path(sys.argv[1]) / s).is_file()
           or (pathlib.Path(sys.argv[1]) / s).stat().st_size == 0]
if missing:
    print("ERROR: missing or empty shards:", missing, file=sys.stderr); sys.exit(1)
print(f"shards OK: {len(shards)} files listed in {idx.name}")
' "$MODEL_DIR" || exit 1
fi
# tokenizer files (at least one of the standard names)
for f in tokenizer.json tokenizer_config.json; do
  test -s "$MODEL_DIR/$f" || { echo "ERROR: tokenizer file $f missing" >&2; exit 1; }
done
# record the revision next to the files -- MODEL_DIR launches do NOT verify it at load time
echo "$MODEL_REVISION" > "$MODEL_DIR/REVISION.txt"
echo "Download completed:"
ls -lah "$MODEL_DIR"

hf download with no file list downloads the whole repo (weights, config.json, tokenizer, index) — better than picking files by hand and forgetting one. If the model needs authentication, run hf auth login first. Public models need no login.

These checks verify config.json, every shard listed in the weight index (existence and non-zero size), and the tokenizer files. They do not hash the bytes: for byte-level integrity run sha256sum against the hashes published by the model author, if any. With MODEL_DIR the launcher does not compare the directory against MODEL_REVISION at load time -- the saved REVISION.txt is a record you keep yourself; verify it when you need reproducibility.

Option B — let the launcher download into the Hugging Face cache

The launcher can also fetch by MODEL_ID into the default HF cache (~/.cache/huggingface), but then MODEL_REVISION (a 40-hex SHA) is mandatory — the launcher refuses a branch name. Get the SHA the same way (step 4 above), export MODEL_ID and MODEL_REVISION instead of MODEL_DIR, and start the server. The cached files live under ~/.cache/huggingface/hub/models--<org>--<name>/snapshots/<SHA>/ — that snapshot directory is what the server loads. Note the cache is not wiped by rollback (see manage.sh clean-cache).

5. Generate an API key (mandatory)

export API_KEY="$(openssl rand -hex 24)"

For loopback-only debug you may instead run with ALLOW_NO_AUTH=1 BIND=127.0.0.1 — but a key is required for every other bind.

6. First run — safe baseline (stock vLLM + UVA)

Start with the unchanged baseline and a small context:

export IMAGE="vllm/vllm-openai:<tag>@sha256:<64-hex-digest>"   # from step 3
export API_KEY="$(openssl rand -hex 24)"                       # from step 5
export MAX_MODEL_LEN=16384 CPU_OFFLOAD_GB=96
bash scripts/run-tp4-uva.sh
bash scripts/check-tp4-api.sh   # waits for /v1/models, sends one chat request (uses $API_KEY)

Only when the baseline is healthy, try hotsplit (it is not verified on a 4-GPU target host yet):

# stop the UVA baseline first -- both use --network host and the same port
docker stop mimo-pro-tp4-uva && docker rm mimo-pro-tp4-uva
export HOTSPLIT_DYNAMIC=0          # static first
bash scripts/run-tp4-hotsplit.sh
bash scripts/check-hotsplit-tp4.sh mimo-pro-tp4-hotsplit static

Then the dynamic experiment (separate run):

docker stop mimo-pro-tp4-hotsplit && docker rm mimo-pro-tp4-hotsplit
HOTSPLIT_DYNAMIC=1 bash scripts/run-tp4-hotsplit.sh
bash scripts/check-hotsplit-tp4.sh mimo-pro-tp4-hotsplit dynamic expect-progress

7. How to read the check script

• check-hotsplit-tp4.sh <container> static — expects dynamic_hook=INACTIVE on all ranks; a missing hotsplit done per rank is a FAIL.
• check-hotsplit-tp4.sh <container> dynamic — expects dynamic_hook=ACTIVE on every rank. After real traffic add expect-progress (3rd arg or EXPECT_PROGRESS=1): zero dynamic progress then becomes a FAIL with non-zero exit, not a silent WARN. After real traffic the hotsplit dynamic update count must be non-zero; before generation a zero count is normal — it proves nothing either way.

Day-to-day commands (scripts/manage.sh)

After the first successful launch you do not need to remember docker flags and env variables:

./manage.sh status      # containers, GPUs, API reachability — one screen
./manage.sh logs        # follow the server log
./manage.sh stop        # stop and remove the server container
./manage.sh start       # start again (HOTSPLIT_DYNAMIC from env or tp4.env)
./manage.sh restart     # stop + start, picks up a changed tp4.env
./manage.sh update      # git pull + restart
./manage.sh rollback    # container-level rollback (removes the container only)
./manage.sh clean-cache # show the HF cache size, delete only after explicit confirmation
./manage.sh check       # hotsplit verification, mode follows HOTSPLIT_DYNAMIC

manage.sh is a thin wrapper over the recipe launchers — it sources ./tp4.env if you created one (copy tp4.env.example), and environment values you already exported win over the file.

Known sharp edges (from real review)

• Hot budget (HOTSPLIT_HOT_GIB) is per rank. Empty = "use the GPU-resident expert bytes the stock placement used", not "fit the whole bank" — a whole bank does not fit on one card. See Models & Geometry for the arithmetic.
• HOTSPLIT_VRAM_RESERVE_GIB=2 is a floor, not a promise. Long contexts and concurrency need more — measure with the recipe's RUNNING.md procedures.
• The model patch (PATCH_MODEL_FILES=1) is verified against patches/MANIFEST.sha256 before launch — do not edit patch files in place.

Full measurement procedures: the recipe's RUNNING.md and the FAQ.