Skip to content

v0.3.0

Latest

Choose a tag to compare

@danbar2 danbar2 released this 08 Nov 19:05
· 1 commit to release/0.3 since this release
8ed2227

Snapshot v0.3.0 - Release Notes

Snapshot v0.3.0 adds S3-compatible object storage as a snapshot store, next to the ReadWriteMany PVC. It also adds lifecycle hooks to the workload contract, and upstream engines are adopting the contract: vLLM has merged its own checkpoint and restore endpoints, which will be available in v0.34, and TensorRT-LLM has merged the memory management the contract relies on, in v1.3.0rc26 and later. Restored containers end with the restored workload, failures name the step they came from, and a restore is refused before CRIU runs when the destination cannot take it.

Summary

S3: Object Storage as a Snapshot Store Option

Snapshot can now also keep snapshots in any S3-compatible object store. Set storage.type: s3, point storage.s3 at an endpoint, region, bucket and prefix, and give it credentials from a Secret. PageBroker uploads each snapshot and publishes an index of its files, and a restore validates the index and downloads the files with Model Streamer. A PVC remains the default and is unchanged. See Storage (#416, #663, #694).

Workload Contract: Lifecycle Hooks & Upstream Engine Support

The workload contract is what a workload does to be snapshot-ready: signal when it is safe to checkpoint, and wait for the restore. v0.3.0 adds lifecycle hooks, so a container can name its own signals, a file or an HTTP call, in place of an entrypoint that writes sentinel files (#746). vLLM now implements these steps itself, with POST /checkpoint/prepare, GET /checkpoint/status and POST /checkpoint/restore in its Python and Rust frontends (vllm-project/vllm#62570). It is merged on vLLM main and will be available in v0.34. TensorRT-LLM does not have native endpoints yet. The sleep and wakeup calls and the CUDA graph address preservation that the contract relies on are merged and in v1.3.0rc26 and later, and its lifecycle controls are an open pull request (NVIDIA/TensorRT-LLM#21598).

Restore: Proxy, Named Failures & Early Refusals

Restored containers now end when the restored workload ends, through a restore proxy that runs as PID 1 and needs containerd 2.2 or newer (#504). A failed checkpoint or restore says which step it came from (#587), and after a failed checkpoint the node agent brings the suspended workload back instead of killing it (#791). A checkpoint or restore on a MIG slice, a restore onto a CPU that lacks an instruction set the source had, and a restore without the AppArmor or SELinux labels a snapshot needs are refused before CRIU runs (#318).

Baseline

Component Requirement
Nodes Linux kernel 5.13 or newer, x86_64 or arm64
Container runtime containerd or CRI-O; the restore proxy needs containerd 2.2 or newer with NRI v0.10
NVIDIA GPU Operator 26.3 or newer, with MIG disabled
NVIDIA driver Kernel-mode driver 580 or newer; CUDA user-mode driver 615 or newer to restore a multi-GPU workload that holds multicast groups
Snapshot store A ReadWriteMany storage class, or an S3-compatible bucket (storage.type: s3)
Engines in the published benchmarks vLLM 0.31.0, SGLang 0.5.21, TensorRT-LLM 1.3.0rc24
Engines that implement the steps themselves A vLLM build that includes vllm-project/vllm#62570, which will be available in v0.34

Upgrade Notes

These matter when moving from v0.2.0.

  • Two Helm values were renamed. config.checkpoint.checkpointTimeoutSeconds is now config.checkpoint.timeoutSeconds (default 3600), and config.restore.restoreTimeoutSeconds is now config.restore.timeoutSeconds (default 7200). The chart refuses to render while an old name is set. The rendered ConfigMap carries both spellings during this release, so a v0.2.0 agent that restarts before the DaemonSet roll reaches its node still starts (#794).

    Migrate: Rename any of the two values you set. A node agent ConfigMap you manage yourself needs the new names: without restore.timeoutSeconds the agent does not start, and without checkpoint.timeoutSeconds the checkpoint timeout falls back to 3600.

  • Restore containers run the restore proxy. Snapshot replaces the command of every restore destination container with /snapshot-control/snapshot-restore-proxy. The node agent sets it through NRI when the container is created, and fails the restore if the proxy is not PID 1. Restore Pods on nodes with the proxy keep the source's command, which never runs, and are expected to use restartPolicy: Never: a restored container that starts again ends in CrashLoopBackOff. The node agent mounts the runtime's NRI socket directory, /var/run/nri (#504, #537, #633).

    Migrate: Set restartPolicy: Never on restore Pods. Where clusters share nodes, such as vCluster tenants, set daemonset.nriHostNamespace so each installation uses its own NRI namespace. On nodes without NRI v0.10 or newer, restore Pods still need an inert command. SNAPSHOT_RESTORE_STANDBY is deprecated and the DYN_SNAPSHOT_RESTORE_STANDBY alias is removed in the next release.

  • New CRD fields. PodSnapshot, SnapshotJob and PodSnapshotContent gain containers, which holds the lifecycle hooks. PodSnapshotContent also gains spec.storage.storeID and status.storage.artifacts, and SnapshotJob gains spec.nonTargetArtifacts. Deploy the matching CRDs, operator and node agent together (#734, #471, #489).
  • Failure messages changed. They open with the step they came from, as while <step>: <cause>, on the Pod's nvidia.com/Restored condition, the PodSnapshotContent Failed condition, the restore events and the node agent log. The log also carries a structured step key. Other messages and the CRD field descriptions say checkpoint and snapshot where they said capture. No condition type, reason or label changed (#587, #690, #794).

    Migrate: Update alerts and log filters that match the previous message text.

  • Restoring into an AppArmor or SELinux container needs labels in the snapshot. A restore without them is refused up front under the new lsm-labels check, which nvidia.com/snapshot-skip-compat-check does not waive. Snapshots taken by earlier node agent versions on AppArmor nodes do not have the labels (#607, #708).

    Migrate: Take those snapshots again.

  • New refusals. A restore onto a CPU missing an instruction set is refused under cpu-features. A snapshot taken before the upgrade recorded no CPU flags and is not checked (#545, #546, #547). A checkpoint or restore on a MIG slice is refused under mig-slice. A checkpoint fails before anything is locked, leaving the source process running, and nvidia.com/snapshot-skip-compat-check waives the restore side (#318).
  • A restored process sees its own /proc. CRIU now mounts procfs from the restored process tree, so lookups by PID work after a restore. A snapshot taken before the upgrade still restores with the placeholder's procfs (#787).

    Migrate: Take snapshots again to get the new behavior.

  • Failed non-target output is kept unless you opt in. Omitting SnapshotJob.spec.nonTargetArtifacts, or setting Retain, preserves what non-target containers wrote to a shared PVC. The old writer annotation is still accepted but no longer grants automatic cleanup (#489).

    Migrate: To clean up after a failed job, declare the writers and set onFailure: Cleanup in spec.nonTargetArtifacts.

  • PageBroker staging is capped below the node agent's memory limit. The default staging volume is now 56Gi against the 64Gi limit, so staging can no longer push the node agent into an OOM kill. A cap that leaves no room is lowered to the limit less 8Gi instead of failing the install (#640).
  • Using S3 needs matching images and one more value. The S3 backend is in the node agent and PageBroker images of this release, not v0.2.0, so use the chart, node agent and PageBroker images from the same release. storage.type: s3 requires operator.bindNewContents: true; the chart fails the render without it. PVC installs are unchanged (#663).

Features & Improvements

S3 Storage

  • S3 Store Configuration: Added storage.type: s3 with storage.s3.endpoint, region, bucket and prefix, and credentials from a shared-credentials-file Secret through storage.s3.credentialsSecretRef. The chart no longer assumes a PVC when S3 is selected (#663).
  • Connection Settings: Added storage.s3.addressing (path or virtual), storage.s3.allowHttp, a CA bundle through storage.s3.caBundleConfigMapRef, pageBroker.s3.limits and pageBroker.transferEngine.modelStreamer. Changing one restarts PageBroker on each node (#694).
  • Upload and Restore Engines: Added batch uploads through rclone and a native S3 restore engine on Model Streamer, with concurrent restores, cancellation and recovery after a failed session, and made the Model Streamer runtime settings configurable in PageBroker (#258, #410, #469).
  • Published Index: Published an index of each snapshot's files, paths, sizes and permissions, retried an uncertain publication without uploading again, and validated the index and object sizes before restoring (#416).
  • Store Binding: Bound every new snapshot to its store with spec.storage.storeID, recorded published artifacts in status.storage.artifacts, and refused to restore or clean up against a different store (#471, #475, #492).
  • Safe Cleanup: Added a coordination contract so two cleanup operations never touch the same artifact at once, and wired PVC maintenance to it (#474).
  • Staging Path: Fixed S3 restores being refused because PageBroker staged them under a different directory than the node agent accepts (#723).
  • Faster Restore Preparation: Skipped per-file existence and size requests before an S3 restore. A missing or short object now fails during download with STORAGE_ERROR (#728).

Workload Contract

  • Checkpoint Readiness Probe: Added containers[].checkpointReadinessProbe to PodSnapshot and SnapshotJob. A file that must exist in the source container, or an HTTP GET to the Pod IP that must return a 2xx status, replaces the Ready condition as the signal to checkpoint, with initialDelaySeconds, periodSeconds, timeoutSeconds and successThreshold. A file on a volume other than an emptyDir, or an HTTP probe on a host-network Pod, fails the checkpoint with InvalidCheckpointReadinessProbe (#734, #735, #736).
  • Post-Restore Action: Added containers[].lifecycle.postRestore. After the restore, the node agent creates a file or sends a POST with an empty body every periodSeconds until it succeeds, and a restore whose action is refused or does not succeed in time fails (#738, #746).
  • vLLM Checkpoint Lifecycle: Merged POST /checkpoint/prepare, GET /checkpoint/status, POST /checkpoint/restore and --checkpoint-prepare-on-startup into vLLM, in the Python and Rust frontends. The routes need --enable-sleep-mode. The status route returns 200 once the engine is prepared, 503 until then and 500 after a failure, and the restore route can be called again. They are on main and will be available in v0.34, and the usage guides still start vLLM through app.py (vllm-project/vllm#62570). Nightly end-to-end coverage against vLLM's top-of-tree images is not in place yet (#732).
  • vLLM Communicator Hooks: Added process-checkpoint lifecycle hooks for communicators, starting with FlashInfer. They are in vLLM v0.27.0 and later (vllm-project/vllm#46877).
  • vLLM Proposal: Opened the proposal for who owns what in an externally orchestrated checkpoint and restore (vllm-project/vllm#60422).
  • TensorRT-LLM Sleep and Wakeup: Added sleep and wakeup to the MPI executor, for one rank and for several, the calls the workload contract names for parking and restoring KV cache memory. They are in v1.3.0rc26 and later (NVIDIA/TensorRT-LLM#14052, NVIDIA/TensorRT-LLM#14636).
  • TensorRT-LLM Graph Addresses: Preserved the virtual addresses of CUDA graphs for MoE all-to-all and MNNVL all-reduce across a restore. They are in v1.3.0rc26 and later (NVIDIA/TensorRT-LLM#16632, NVIDIA/TensorRT-LLM#16635).

Restore

  • Restore Proxy: Added the proxy as PID 1 in restore containers, with a fallback for runtimes without NRI v0.10 (#504, #537, #633).
  • Security Labels: Passed the restore container's AppArmor profile or SELinux label to CRIU, mounted the node's securityfs read-only into the node agent so CRIU can detect AppArmor, and checked labels from a recorded value instead of scanning every CRIU image on each restore (#607, #708).
  • Root Filesystem Diff: Kept files that the checkpointed processes held open when a rootfs exclusion matched them, so regenerable caches can be excluded safely, and extracted the diff with parallel read-ahead (#602, #724).
  • Restore Ordering: Prepared restore participants with GPU data first (#672).
  • Parallel Ranks: Restored the ranks of a multi-GPU workload in parallel, where cuda-restore had restored them one after another (#757).
  • GPU Listing: Ran nvidia-smi once per restore instead of twice, which saves one 107 to 193 ms run on an eight-GPU B200 node (#712).

Compatibility Checks

  • MIG Slices: Refused a checkpoint or restore on a MIG slice, naming the slice on the PodSnapshotContent Failed condition (#318).
  • CPU Features: Recorded the source CPU's instruction sets on each snapshot and refused a restore onto a CPU that lacks one (#545, #546, #547).

Failure Handling

  • Step Names: Named the step a checkpoint or restore failure came from, including the steps inside nsrestore, and gave a restore whose nsrestore does not start the reason nsenter reported (#587, #690, #586).
  • Failure Reasons: Gave a failed checkpoint a reason that tells a node momentarily out of staging (StagingUnavailable) from a request that will never be accepted (BrokerRejected), plus StagedDataLost and CheckpointTimeout (#685).
  • Recovered Workloads: After a failed checkpoint, brought the suspended source workload back, with its GPUs restored and unlocked, and killed it only when its GPU state could not be recovered or CRIU had not released it (#791).
  • Kept Work: Kept the staged data after a completed checkpoint for a commit failure that may still clear, instead of releasing it (#642).
  • Staging Shortfall: Refused a checkpoint the node cannot stage before suspending the workload, as ImageTooLarge with the shortfall (#687).
  • Resilience: Recovered from a panic in one work item (#641), and let PageBroker skip a GPU that fails setup instead of exiting and stopping every operation on the node (#714).

Operator & Chart

  • Unscheduled Source Pods: Waited for an unscheduled source Pod as Pending and recorded the reason SourcePodUnscheduled on the PodSnapshot, so the wait is visible without reading the operator log (#697).
  • Operator Cache: Kept only the Pod fields the controllers read in the operator's cache. The operator had cached every Pod in full and was killed for running out of memory at 256Mi on a cluster of about 4,000 Pods. On 3,581 Pods the cached data went from 28.0 MiB to 10.0 MiB of JSON (#761).
  • Node Affinity: Accepted daemonset.affinity.nodeAffinity and added the agent's architecture requirement to every required node selector term, so a term such as example.com/pool DoesNotExist keeps the agent off those nodes (#799).

Bug Fixes

  • Missing Snapshots: A restore of a Ready snapshot whose artifact is missing now fails with RestoreFailed instead of retrying forever and leaving the restore Pod and its GPU blocked (#674).
  • Out-of-Memory Kills: Capped PageBroker staging below the node agent's memory limit (#640).
  • Restored /proc: Fixed a restored process seeing the placeholder's procfs, which broke lookups by PID, such as UCX shared-memory endpoints, after a restore (#787).
  • Device Plugin Markers: Fixed restores onto a GPU other than the source GPU being refused with mount: source /run/nvidia-container-devices/GPU-<uuid>, target missing when the NVIDIA device plugin uses the volume-mounts device list strategy. A checkpoint now skips the marker (#814).
  • Multi-GPU: Kept cudaMalloc semantics for converted allocations (#673).
  • Restore Start: Started restores when NRI reports the destination container (#704).
  • Builds: Fetched PageBroker dependency sources in the install layer (#681) and updated the Go toolchain to 1.27.3 (#668).

Testing & CI

  • GPU tests for multi-GPU sharing, peer carriers and coordinator orchestration, and a CI job that runs the CPU and GPU suites (#338, #566, #578, #579, #580, #581, #582).
  • End-to-end coverage for SnapshotJob partial-success recovery, CPU restore failure and scheduling, graceful shutdown of a restored workload, and the framework checkpoint and restore test over S3 on the GPU cluster (#518, #515, #713, #727).
  • CI runners are pinned to Ubuntu 24.04, and a branch's first image build can start from the main cache (#667, #747).

Documentation

  • Lifecycle Hooks: Documented in the API reference, the workload contract, the architecture reference, troubleshooting and limitations (#746).
  • Storage: Started the storage best practices guide, with a backend comparison and a slow-restore walkthrough (#775).
  • Naming: Reworded the user-facing pages, the CRD field descriptions, the chart README and the condition and failure messages to say checkpoint and snapshot where they said capture (#794).
  • Architecture: the architecture reference and the multi-GPU design page (#683, #337), and documented and exercised multi-GPU workloads in the examples (#456).
  • Benchmarks: extended the published benchmarks to SGLang, TensorRT-LLM and multiple GPUs (#661).
  • Troubleshooting: organized by checkpoint and restore step (#588).
  • Contributor Guides: documentation and naming guides for contributors and agents (#699, #745).
  • Proposals: SNEP-411, application contracts for checkpoint and restore (#437).

Known Issues

None recorded.

Contributors

86 merged PRs from 18 contributors. Thank you to everyone who contributed to this release, including @galletas1712 and @hhzhang16, who landed the vLLM and TensorRT-LLM changes above, and @leeZardNav, who opened the vLLM proposal. Welcome to the 7 first-time contributors:

Full Changelog: v0.2.0...v0.3.0