Repository navigation
v2.0.0-beta.17
Pre-releaseLarge local DMTCP snapshots restore faster with huge pages and parallel
CPU-memory reads. On one H200, Qwen3.8-Flash-Next-NVFP4 reached health in
20.21–23.43 s across four runs (median 20.83 s, first streaming token
20.92 s), versus 77.36/82.14 s with the ordinary reader on the same host
and snapshot. These were local restores with hot file cache and sufficient
RAM, excluding provisioning and snapshot transfer; this is not a <=20 s SLA.
upsupplies restart-only huge-page advice and up to 16 parallel readers,
bounded by Go's container-aware CPU budget. Explicit
DMTCP_RESTART_HUGEPAGESandDMTCP_RESTART_READERSvalues override the
defaults. Private mmap remains disabled by default because it shifted
work into CUDA and worsened full recovery time.- Linux archives now include the separately released TensorRails DMTCP
v5.0.0-warmctl.1runtime and fdkeep plugin for amd64 and arm64. The
runtime Dockerfile downloads the same packages with source-pinned SHA-256
checksums, with no DMTCP compilation during the warmctl build. Packages
require Debian 12-compatible glibc/libstdc++; macOS remains client-only. - The CLI finds its versioned runtime beside its own executable, including
custom install prefixes. The installer preserves runtime bytes and mtimes,
retains older versions, and refuses to replace a different build at the
same version. Bake new snapshots after upgrading; old snapshots still
need their original libraries and paths. - Release archives and the installer are tested natively on both Linux
architectures before publication. ARM64 is CPU-tested; GPU serving and
the parallel reader remain x86-64-only qualifications. - All 20 GPU experiments matched weight fingerprints and passed streaming
and tool-call tests; 640 short requests succeeded. The numerical canary
remains unqualified: the donor already varied before checkpoint. Its
tolerance is unchanged. See docs/dmtcp.md for the phase breakdown.
Native Linux runtime requirements: glibc >= 2.36, compatible libstdc++ and libatomic. On minimal Debian/Ubuntu hosts, install libstdc++6 and libatomic1. ARM64 is CPU checkpoint/restore tested; GPU serving qualification is x86-64 only.
Fresh H200 replicas from a network volume
Two fresh Runpod pods using the published binaries returned the first external streaming token in 52.14 s and 52.94 s from the pod-create API request, including 5.57/7.28 s of test-only client-cache advice. An ordinary fresh pod with host cache unchanged took 32.93 s. Each passed weight fingerprints, streaming, tool calls and 32 concurrent short requests, then was deleted.
The measured route uses a cached stock Runpod image, a prepared Python tree and one verified snapshot on a high-performance network volume, 16 prefetch workers and at least 27 advertised vCPUs (22.95-core quota in these runs). The snapshot was built locally, verified, copied with checked/fsynced files and verified on the volume before donor shutdown. Receivers neither rebuild it nor reload the original safetensors.
These measurements are not a latency SLA: the final pods reused the same physical host; targeted cache advice cannot clear storage-server or shared-library caches. Numerical canary equivalence remains unqualified at the unchanged 0.001 tolerance. Publication failure recovery remains tracked in issue #67.
See the startup recipe, phase timings and measurement boundaries. The runtime images were also published at registry.warmctl.io/warmctl/runtime:v2.0.0-beta.17 and :v2.0.0-beta.17-vllm032; they passed CPU smoke tests. GPU timing above qualifies the stock-image route, not custom-image pull latency. No newer CLI release is needed for this deployment tuning.