v2.0.0-beta.10

Engine

• Data-parallel profiles build and restore on vLLM's DP frontend, which runs a
  /collective_rpc call on every engine replica but answers with the first
  replica's workers only. With dp > 1 warmctl calls its worker methods
  through the adapter's warmctl_rpc, which publishes each rank's result as a
  file in the agent's IO directory, and requires one file for each rank
  (#23). TP-only profiles and their snapshots keep the plain RPC.
• Engines run with NCCL_GRAPH_REGISTER=0: NCCL's registration of the user
  buffers of graph-captured collectives goes through cuInterpose's memory
  adapter, and on the r615 user-mode driver the first replay of a larger
  decode graph hung every rank (#27). Without the registration the DEP8
  engine runs, checkpoints and restores; the registration only matters for
  large buffers.

Profiles

• deepseek-v41-flash-1017-b200-dep8: the DEP8 recipe on 8×B200 (100 GiB of
  FP8 KV per GPU; 46 GiB of weights per rank). In the catalog for driver
  615: 809 GiB, restores in 84 s from the page cache, 38K output tokens/s at
  1K/1K once restored.
• deepseek-v41-flash-1017-b300-tp8: DeepSeek-V4.1-Flash-1017 on 8×B300, one
  TP8 engine with expert parallelism, 256 sequences, 16K prefill chunks, 128 GiB
  of FP8 KV per GPU and DSpark with 5 draft tokens.
• deepseek-v41-flash-1017-b300-dep8: the same model for throughput, after
  vLLM's B300 recipe: DEP8 (eight data-parallel attention replicas, EP with
  weight filtering), FlashMLA mega-attention with an MXFP4 sparse indexer,
  DeepGEMM mega-MoE, DSpark with block rejection sampling. Its image is the
  recipe's (vLLM 0.31) with libcuda 615 in place of 580
  (registry.warmctl.io/warmctl/vllm-openai:0.31-dsv41-b300-cu13compat615,
  deploy/images/vllm-cu13compat615): an r580 libcuda cannot restore the
  mega-MoE multicast buffers after cuda-checkpoint.

Docs

• docs/interpose.md: multicast objects restore only on a user-mode driver
  of r595 or newer (ai-dynamo/snapshot#510); the image's forward-compatible
  libcuda counts, the host kernel driver may stay on 580.
• The hardware table has a b300-275gb row (NVIDIA B300 SXM6, sm_103).

v2.0.0-beta.9

v2.0.0-beta.9

v2.0.0-beta.8

v2.0.0-beta.8

v2.0.0-beta.7

v2.0.0-beta.7

v2.0.0-beta.6

v2.0.0-beta.6

v2.0.0-beta.5

The first public release. Restore in a Kubernetes pod with the same
warmctl up NAME, multi-GPU engines restore under NVIDIA driver 610, and a
cold restore from a file share reads the snapshot in parallel (GLM-5.3-Flash
on 8×H100: 75 s → 57 s).

Kubernetes (prototype)

• up picks where to restore. In the warmctl runtime image (marker
  /opt/warmctl/RUNTIME) it restores in place, in its own container; otherwise
  it asks Docker for a receiver container. With neither, it stops with an
  error. --mode host|in-place overrides the choice.
• In place, warmctl prepares its privileged container as a receiver: tmpfs at
  /tmp and /dev/shm, the snapshot's extension and cuInterpose mounted
  read-only, the GPUs the device plugin assigned. The agent runs in a UTS
  namespace of its own with the host name the engine had. The pod needs no
  volumes; a node-local /var/lib/warmctl only keeps pulled snapshots.
• The preflight refuses hostNetwork and hostPID, a missing capability or
  seccomp profile, a read-only /proc/sys and an image built from another
  vLLM base. warmctl empties /workspace and its run directory only when they
  are empty or marked by an earlier restore; foreign data stops the restore
  untouched.
• The gateway listens on :8080 from the start: /health answers 503 while
  pulling and restoring and 200 after the canary, so a readinessProbe keeps
  traffic away until the engine is warm.
• The runtime image is registry.warmctl.io/warmctl/runtime, a tag per
  release and :latest; make pod-image builds it for another registry or
  vLLM base. deploy/kubernetes holds its Dockerfile, a Deployment and
  Service, a node-preparation DaemonSet and a KEDA ScaledObject on
  vllm:num_requests_waiting. Design and open risks:
  docs/kubernetes.md.
• Tested on CPU in pod-like containers (make e2e-test), not yet on a GPU
  node of a Kubernetes cluster.

Driver 610

• Multi-GPU engines under driver branch 595 and later run on the host's
  libcuda: build puts the host's library directory first in
  LD_LIBRARY_PATH, because the vLLM image's forward-compatible libcuda
  cannot checkpoint processes that share GPU memory under a 610 kernel
  module. Single-GPU engines keep the image's libcuda.
• Published for 610.57.04: eight single-GPU H100 snapshots, gpt-oss-120b on
  2×H100 and GLM-5.3-Flash on 8×H100.

Restore from a file share

• up reads the snapshot ahead from the start of the restore: the CRIU
  images first, then the weights round-robin across ranks, with 32 parallel
  readers capped by available memory (WARMCTL_PREFETCH_WORKERS overrides).
  Each range is also handed to the kernel with posix_fadvise(WILLNEED), and
  up logs how the prefetch went. GLM-5.3-Flash on 8×H100 restored cold from
  a virtiofs share in 56.7 s instead of 75.1 s; a warm restore stays at
  40–44 s.

Known issues

• Under driver 610, MiMo-V2.6-Flash and DeepSeek-V4.1-Flash do not start on
  the host's libcuda: their kernels ship as CUDA 13.4 PTX, which needs a newer
  libcuda to compile.
• The first restore right after a host boots can fail in CRIU (the restored
  root task is killed after about a second); later restores pass.
• warmctl pull onto a virtiofs share writes at about 20 MB/s.
• GLM-5.3-Flash at TP4 on B200 fails in a Triton kernel of its linear
  attention under 32 users with 4K random-token prompts; TP8 and ordinary
  text work.

Install: unpack warmctl-v2.0.0-beta.5-linux-amd64.tar.gz and keep
warmctl-linux-amd64 and cuinterpose/ side by side (see the README's
quickstart). The Kubernetes runtime image is
registry.warmctl.io/warmctl/runtime:v2.0.0-beta.5.

sha256 5f67e717ca3ca6b3b2d10b9e7d9692210872f0383200ca77e97e27cf9d676d22 warmctl-v2.0.0-beta.5-linux-amd64.tar.gz