v2.0.0-beta.18

Snapshots carry fewer zero pages, and restores copy fewer.

• TensorRails DMTCP v5.0.0-warmctl.2: a checkpoint leaves out resident
  memory that holds only zeros, and restart readers share the large runs
  between them. Baked with warmctl and restored on one RTX 3090, Qwen3-0.6B's
  process images fell from 8,582 to 6,065 MiB and time to /health from
  4.6–5.1 s to 4.0–4.4 s, with the same answers. Pinned CUDA host memory
  already left its zeros out, so Qwen3.8-Flash-Next's CPU-side embedding
  table is unchanged. Bake new snapshots with this release; images restart
  under either DMTCP version.
• up's stage copy (a snapshot on FUSE or stored with --compress) skips
  all-zero pages: the staged files keep holes there. More than half of a
  process image is zero pages (57% of GLM-5.3-Flash's CRIU images), and
  writing them took half of the stage. Staging those images into memory
  went from 3.96 to 7.23 GB/s from a compressed snapshot and to 8.48 GB/s
  from a raw one; weights, which have no zero pages, stage as before.

v2.0.0-beta.17

Large local DMTCP snapshots restore faster with huge pages and parallel
CPU-memory reads. On one H200, Qwen3.8-Flash-Next-NVFP4 reached health in
20.21–23.43 s across four runs (median 20.83 s, first streaming token
20.92 s), versus 77.36/82.14 s with the ordinary reader on the same host
and snapshot. These were local restores with hot file cache and sufficient
RAM, excluding provisioning and snapshot transfer; this is not a <=20 s SLA.

• up supplies restart-only huge-page advice and up to 16 parallel readers,
  bounded by Go's container-aware CPU budget. Explicit
  DMTCP_RESTART_HUGEPAGES and DMTCP_RESTART_READERS values override the
  defaults. Private mmap remains disabled by default because it shifted
  work into CUDA and worsened full recovery time.
• Linux archives now include the separately released TensorRails DMTCP
  v5.0.0-warmctl.1 runtime and fdkeep plugin for amd64 and arm64. The
  runtime Dockerfile downloads the same packages with source-pinned SHA-256
  checksums, with no DMTCP compilation during the warmctl build. Packages
  require Debian 12-compatible glibc/libstdc++; macOS remains client-only.
• The CLI finds its versioned runtime beside its own executable, including
  custom install prefixes. The installer preserves runtime bytes and mtimes,
  retains older versions, and refuses to replace a different build at the
  same version. Bake new snapshots after upgrading; old snapshots still
  need their original libraries and paths.
• Release archives and the installer are tested natively on both Linux
  architectures before publication. ARM64 is CPU-tested; GPU serving and
  the parallel reader remain x86-64-only qualifications.
• All 20 GPU experiments matched weight fingerprints and passed streaming
  and tool-call tests; 640 short requests succeeded. The numerical canary
  remains unqualified: the donor already varied before checkpoint. Its
  tolerance is unchanged. See docs/dmtcp.md for the phase breakdown.

Native Linux runtime requirements: glibc >= 2.36, compatible libstdc++ and libatomic. On minimal Debian/Ubuntu hosts, install libstdc++6 and libatomic1. ARM64 is CPU checkpoint/restore tested; GPU serving qualification is x86-64 only.

Fresh H200 replicas from a network volume

Two fresh Runpod pods using the published binaries returned the first external streaming token in 52.14 s and 52.94 s from the pod-create API request, including 5.57/7.28 s of test-only client-cache advice. An ordinary fresh pod with host cache unchanged took 32.93 s. Each passed weight fingerprints, streaming, tool calls and 32 concurrent short requests, then was deleted.

The measured route uses a cached stock Runpod image, a prepared Python tree and one verified snapshot on a high-performance network volume, 16 prefetch workers and at least 27 advertised vCPUs (22.95-core quota in these runs). The snapshot was built locally, verified, copied with checked/fsynced files and verified on the volume before donor shutdown. Receivers neither rebuild it nor reload the original safetensors.

These measurements are not a latency SLA: the final pods reused the same physical host; targeted cache advice cannot clear storage-server or shared-library caches. Numerical canary equivalence remains unqualified at the unchanged 0.001 tolerance. Publication failure recovery remains tracked in issue #67.

See the startup recipe, phase timings and measurement boundaries. The runtime images were also published at registry.warmctl.io/warmctl/runtime:v2.0.0-beta.17 and :v2.0.0-beta.17-vllm032; they passed CPU smoke tests. GPU timing above qualifies the stock-image route, not custom-image pull latency. No newer CLI release is needed for this deployment tuning.

v2.0.0-beta.16

v2.0.0-beta.16

v2.0.0-beta.15

v2.0.0-beta.15

v2.0.0-beta.14

v2.0.0-beta.14

v2.0.0-beta.13

v2.0.0-beta.13

v2.0.0-beta.12

v2.0.0-beta.12

v2.0.0-beta.11

v2.0.0-beta.11

v2.0.0-beta.10

Engine

• Data-parallel profiles build and restore on vLLM's DP frontend, which runs a
  /collective_rpc call on every engine replica but answers with the first
  replica's workers only. With dp > 1 warmctl calls its worker methods
  through the adapter's warmctl_rpc, which publishes each rank's result as a
  file in the agent's IO directory, and requires one file for each rank
  (#23). TP-only profiles and their snapshots keep the plain RPC.
• Engines run with NCCL_GRAPH_REGISTER=0: NCCL's registration of the user
  buffers of graph-captured collectives goes through cuInterpose's memory
  adapter, and on the r615 user-mode driver the first replay of a larger
  decode graph hung every rank (#27). Without the registration the DEP8
  engine runs, checkpoints and restores; the registration only matters for
  large buffers.

Profiles

• deepseek-v41-flash-1017-b200-dep8: the DEP8 recipe on 8×B200 (100 GiB of
  FP8 KV per GPU; 46 GiB of weights per rank). In the catalog for driver
  615: 809 GiB, restores in 84 s from the page cache, 38K output tokens/s at
  1K/1K once restored.
• deepseek-v41-flash-1017-b300-tp8: DeepSeek-V4.1-Flash-1017 on 8×B300, one
  TP8 engine with expert parallelism, 256 sequences, 16K prefill chunks, 128 GiB
  of FP8 KV per GPU and DSpark with 5 draft tokens.
• deepseek-v41-flash-1017-b300-dep8: the same model for throughput, after
  vLLM's B300 recipe: DEP8 (eight data-parallel attention replicas, EP with
  weight filtering), FlashMLA mega-attention with an MXFP4 sparse indexer,
  DeepGEMM mega-MoE, DSpark with block rejection sampling. Its image is the
  recipe's (vLLM 0.31) with libcuda 615 in place of 580
  (registry.warmctl.io/warmctl/vllm-openai:0.31-dsv41-b300-cu13compat615,
  deploy/images/vllm-cu13compat615): an r580 libcuda cannot restore the
  mega-MoE multicast buffers after cuda-checkpoint.

Docs

• docs/interpose.md: multicast objects restore only on a user-mode driver
  of r595 or newer (ai-dynamo/snapshot#510); the image's forward-compatible
  libcuda counts, the host kernel driver may stay on 580.
• The hardware table has a b300-275gb row (NVIDIA B300 SXM6, sm_103).

v2.0.0-beta.9

v2.0.0-beta.9