Repository navigation
Snapshot v0.3.0 - Release Notes
Snapshot v0.3.0 adds S3-compatible object storage as a snapshot store, next to the ReadWriteMany PVC. It also adds lifecycle hooks to the workload contract, and upstream engines are adopting the contract: vLLM has merged its own checkpoint and restore endpoints, which will be available in v0.34, and TensorRT-LLM has merged the memory management the contract relies on, in v1.3.0rc26 and later. Restored containers end with the restored workload, failures name the step they came from, and a restore is refused before CRIU runs when the destination cannot take it.
Summary
S3: Object Storage as a Snapshot Store Option
Snapshot can now also keep snapshots in any S3-compatible object store. Set storage.type: s3, point storage.s3 at an endpoint, region, bucket and prefix, and give it credentials from a Secret. PageBroker uploads each snapshot and publishes an index of its files, and a restore validates the index and downloads the files with Model Streamer. A PVC remains the default and is unchanged. See Storage (#416, #663, #694).
Workload Contract: Lifecycle Hooks & Upstream Engine Support
The workload contract is what a workload does to be snapshot-ready: signal when it is safe to checkpoint, and wait for the restore. v0.3.0 adds lifecycle hooks, so a container can name its own signals, a file or an HTTP call, in place of an entrypoint that writes sentinel files (#746). vLLM now implements these steps itself, with POST /checkpoint/prepare, GET /checkpoint/status and POST /checkpoint/restore in its Python and Rust frontends (vllm-project/vllm#62570). It is merged on vLLM main and will be available in v0.34. TensorRT-LLM does not have native endpoints yet. The sleep and wakeup calls and the CUDA graph address preservation that the contract relies on are merged and in v1.3.0rc26 and later, and its lifecycle controls are an open pull request (NVIDIA/TensorRT-LLM#21598).
Restore: Proxy, Named Failures & Early Refusals
Restored containers now end when the restored workload ends, through a restore proxy that runs as PID 1 and needs containerd 2.2 or newer (#504). A failed checkpoint or restore says which step it came from (#587), and after a failed checkpoint the node agent brings the suspended workload back instead of killing it (#791). A checkpoint or restore on a MIG slice, a restore onto a CPU that lacks an instruction set the source had, and a restore without the AppArmor or SELinux labels a snapshot needs are refused before CRIU runs (#318).
Baseline
| Component | Requirement |
|---|---|
| Nodes | Linux kernel 5.13 or newer, x86_64 or arm64 |
| Container runtime | containerd or CRI-O; the restore proxy needs containerd 2.2 or newer with NRI v0.10 |
| NVIDIA GPU Operator | 26.3 or newer, with MIG disabled |
| NVIDIA driver | Kernel-mode driver 580 or newer; CUDA user-mode driver 615 or newer to restore a multi-GPU workload that holds multicast groups |
| Snapshot store | A ReadWriteMany storage class, or an S3-compatible bucket (storage.type: s3) |
| Engines in the published benchmarks | vLLM 0.31.0, SGLang 0.5.21, TensorRT-LLM 1.3.0rc24 |
| Engines that implement the steps themselves | A vLLM build that includes vllm-project/vllm#62570, which will be available in v0.34 |
Upgrade Notes
These matter when moving from v0.2.0.
- Two Helm values were renamed.
config.checkpoint.checkpointTimeoutSecondsis nowconfig.checkpoint.timeoutSeconds(default3600), andconfig.restore.restoreTimeoutSecondsis nowconfig.restore.timeoutSeconds(default7200). The chart refuses to render while an old name is set. The rendered ConfigMap carries both spellings during this release, so a v0.2.0 agent that restarts before the DaemonSet roll reaches its node still starts (#794).Migrate: Rename any of the two values you set. A node agent ConfigMap you manage yourself needs the new names: without
restore.timeoutSecondsthe agent does not start, and withoutcheckpoint.timeoutSecondsthe checkpoint timeout falls back to3600. - Restore containers run the restore proxy. Snapshot replaces the command of every restore destination container with
/snapshot-control/snapshot-restore-proxy. The node agent sets it through NRI when the container is created, and fails the restore if the proxy is not PID 1. Restore Pods on nodes with the proxy keep the source's command, which never runs, and are expected to userestartPolicy: Never: a restored container that starts again ends inCrashLoopBackOff. The node agent mounts the runtime's NRI socket directory,/var/run/nri(#504, #537, #633).Migrate: Set
restartPolicy: Neveron restore Pods. Where clusters share nodes, such as vCluster tenants, setdaemonset.nriHostNamespaceso each installation uses its own NRI namespace. On nodes without NRI v0.10 or newer, restore Pods still need an inert command.SNAPSHOT_RESTORE_STANDBYis deprecated and theDYN_SNAPSHOT_RESTORE_STANDBYalias is removed in the next release. - New CRD fields.
PodSnapshot,SnapshotJobandPodSnapshotContentgaincontainers, which holds the lifecycle hooks.PodSnapshotContentalso gainsspec.storage.storeIDandstatus.storage.artifacts, andSnapshotJobgainsspec.nonTargetArtifacts. Deploy the matching CRDs, operator and node agent together (#734, #471, #489). - Failure messages changed. They open with the step they came from, as
while <step>: <cause>, on the Pod'snvidia.com/Restoredcondition, thePodSnapshotContentFailedcondition, the restore events and the node agent log. The log also carries a structuredstepkey. Other messages and the CRD field descriptions say checkpoint and snapshot where they said capture. No condition type, reason or label changed (#587, #690, #794).Migrate: Update alerts and log filters that match the previous message text.
- Restoring into an AppArmor or SELinux container needs labels in the snapshot. A restore without them is refused up front under the new
lsm-labelscheck, whichnvidia.com/snapshot-skip-compat-checkdoes not waive. Snapshots taken by earlier node agent versions on AppArmor nodes do not have the labels (#607, #708).Migrate: Take those snapshots again.
- New refusals. A restore onto a CPU missing an instruction set is refused under
cpu-features. A snapshot taken before the upgrade recorded no CPU flags and is not checked (#545, #546, #547). A checkpoint or restore on a MIG slice is refused undermig-slice. A checkpoint fails before anything is locked, leaving the source process running, andnvidia.com/snapshot-skip-compat-checkwaives the restore side (#318). - A restored process sees its own
/proc. CRIU now mounts procfs from the restored process tree, so lookups by PID work after a restore. A snapshot taken before the upgrade still restores with the placeholder's procfs (#787).Migrate: Take snapshots again to get the new behavior.
- Failed non-target output is kept unless you opt in. Omitting
SnapshotJob.spec.nonTargetArtifacts, or settingRetain, preserves what non-target containers wrote to a shared PVC. The old writer annotation is still accepted but no longer grants automatic cleanup (#489).Migrate: To clean up after a failed job, declare the writers and set
onFailure: Cleanupinspec.nonTargetArtifacts. - PageBroker staging is capped below the node agent's memory limit. The default staging volume is now 56Gi against the 64Gi limit, so staging can no longer push the node agent into an OOM kill. A cap that leaves no room is lowered to the limit less 8Gi instead of failing the install (#640).
- Using S3 needs matching images and one more value. The S3 backend is in the node agent and PageBroker images of this release, not v0.2.0, so use the chart, node agent and PageBroker images from the same release.
storage.type: s3requiresoperator.bindNewContents: true; the chart fails the render without it. PVC installs are unchanged (#663).
Features & Improvements
S3 Storage
- S3 Store Configuration: Added
storage.type: s3withstorage.s3.endpoint,region,bucketandprefix, and credentials from a shared-credentials-file Secret throughstorage.s3.credentialsSecretRef. The chart no longer assumes a PVC when S3 is selected (#663). - Connection Settings: Added
storage.s3.addressing(pathorvirtual),storage.s3.allowHttp, a CA bundle throughstorage.s3.caBundleConfigMapRef,pageBroker.s3.limitsandpageBroker.transferEngine.modelStreamer. Changing one restarts PageBroker on each node (#694). - Upload and Restore Engines: Added batch uploads through rclone and a native S3 restore engine on Model Streamer, with concurrent restores, cancellation and recovery after a failed session, and made the Model Streamer runtime settings configurable in PageBroker (#258, #410, #469).
- Published Index: Published an index of each snapshot's files, paths, sizes and permissions, retried an uncertain publication without uploading again, and validated the index and object sizes before restoring (#416).
- Store Binding: Bound every new snapshot to its store with
spec.storage.storeID, recorded published artifacts instatus.storage.artifacts, and refused to restore or clean up against a different store (#471, #475, #492). - Safe Cleanup: Added a coordination contract so two cleanup operations never touch the same artifact at once, and wired PVC maintenance to it (#474).
- Staging Path: Fixed S3 restores being refused because PageBroker staged them under a different directory than the node agent accepts (#723).
- Faster Restore Preparation: Skipped per-file existence and size requests before an S3 restore. A missing or short object now fails during download with
STORAGE_ERROR(#728).
Workload Contract
- Checkpoint Readiness Probe: Added
containers[].checkpointReadinessProbetoPodSnapshotandSnapshotJob. A file that must exist in the source container, or an HTTP GET to the Pod IP that must return a 2xx status, replaces the Ready condition as the signal to checkpoint, withinitialDelaySeconds,periodSeconds,timeoutSecondsandsuccessThreshold. A file on a volume other than anemptyDir, or an HTTP probe on a host-network Pod, fails the checkpoint withInvalidCheckpointReadinessProbe(#734, #735, #736). - Post-Restore Action: Added
containers[].lifecycle.postRestore. After the restore, the node agent creates a file or sends a POST with an empty body everyperiodSecondsuntil it succeeds, and a restore whose action is refused or does not succeed in time fails (#738, #746). - vLLM Checkpoint Lifecycle: Merged
POST /checkpoint/prepare,GET /checkpoint/status,POST /checkpoint/restoreand--checkpoint-prepare-on-startupinto vLLM, in the Python and Rust frontends. The routes need--enable-sleep-mode. The status route returns 200 once the engine is prepared, 503 until then and 500 after a failure, and the restore route can be called again. They are onmainand will be available in v0.34, and the usage guides still start vLLM throughapp.py(vllm-project/vllm#62570). Nightly end-to-end coverage against vLLM's top-of-tree images is not in place yet (#732). - vLLM Communicator Hooks: Added process-checkpoint lifecycle hooks for communicators, starting with FlashInfer. They are in vLLM v0.27.0 and later (vllm-project/vllm#46877).
- vLLM Proposal: Opened the proposal for who owns what in an externally orchestrated checkpoint and restore (vllm-project/vllm#60422).
- TensorRT-LLM Sleep and Wakeup: Added sleep and wakeup to the MPI executor, for one rank and for several, the calls the workload contract names for parking and restoring KV cache memory. They are in v1.3.0rc26 and later (NVIDIA/TensorRT-LLM#14052, NVIDIA/TensorRT-LLM#14636).
- TensorRT-LLM Graph Addresses: Preserved the virtual addresses of CUDA graphs for MoE all-to-all and MNNVL all-reduce across a restore. They are in v1.3.0rc26 and later (NVIDIA/TensorRT-LLM#16632, NVIDIA/TensorRT-LLM#16635).
Restore
- Restore Proxy: Added the proxy as PID 1 in restore containers, with a fallback for runtimes without NRI v0.10 (#504, #537, #633).
- Security Labels: Passed the restore container's AppArmor profile or SELinux label to CRIU, mounted the node's securityfs read-only into the node agent so CRIU can detect AppArmor, and checked labels from a recorded value instead of scanning every CRIU image on each restore (#607, #708).
- Root Filesystem Diff: Kept files that the checkpointed processes held open when a rootfs exclusion matched them, so regenerable caches can be excluded safely, and extracted the diff with parallel read-ahead (#602, #724).
- Restore Ordering: Prepared restore participants with GPU data first (#672).
- Parallel Ranks: Restored the ranks of a multi-GPU workload in parallel, where
cuda-restorehad restored them one after another (#757). - GPU Listing: Ran
nvidia-smionce per restore instead of twice, which saves one 107 to 193 ms run on an eight-GPU B200 node (#712).
Compatibility Checks
- MIG Slices: Refused a checkpoint or restore on a MIG slice, naming the slice on the
PodSnapshotContentFailedcondition (#318). - CPU Features: Recorded the source CPU's instruction sets on each snapshot and refused a restore onto a CPU that lacks one (#545, #546, #547).
Failure Handling
- Step Names: Named the step a checkpoint or restore failure came from, including the steps inside
nsrestore, and gave a restore whosensrestoredoes not start the reasonnsenterreported (#587, #690, #586). - Failure Reasons: Gave a failed checkpoint a reason that tells a node momentarily out of staging (
StagingUnavailable) from a request that will never be accepted (BrokerRejected), plusStagedDataLostandCheckpointTimeout(#685). - Recovered Workloads: After a failed checkpoint, brought the suspended source workload back, with its GPUs restored and unlocked, and killed it only when its GPU state could not be recovered or CRIU had not released it (#791).
- Kept Work: Kept the staged data after a completed checkpoint for a commit failure that may still clear, instead of releasing it (#642).
- Staging Shortfall: Refused a checkpoint the node cannot stage before suspending the workload, as
ImageTooLargewith the shortfall (#687). - Resilience: Recovered from a panic in one work item (#641), and let PageBroker skip a GPU that fails setup instead of exiting and stopping every operation on the node (#714).
Operator & Chart
- Unscheduled Source Pods: Waited for an unscheduled source Pod as Pending and recorded the reason
SourcePodUnscheduledon thePodSnapshot, so the wait is visible without reading the operator log (#697). - Operator Cache: Kept only the Pod fields the controllers read in the operator's cache. The operator had cached every Pod in full and was killed for running out of memory at 256Mi on a cluster of about 4,000 Pods. On 3,581 Pods the cached data went from 28.0 MiB to 10.0 MiB of JSON (#761).
- Node Affinity: Accepted
daemonset.affinity.nodeAffinityand added the agent's architecture requirement to every required node selector term, so a term such asexample.com/pool DoesNotExistkeeps the agent off those nodes (#799).
Bug Fixes
- Missing Snapshots: A restore of a
Readysnapshot whose artifact is missing now fails withRestoreFailedinstead of retrying forever and leaving the restore Pod and its GPU blocked (#674). - Out-of-Memory Kills: Capped PageBroker staging below the node agent's memory limit (#640).
- Restored
/proc: Fixed a restored process seeing the placeholder's procfs, which broke lookups by PID, such as UCX shared-memory endpoints, after a restore (#787). - Device Plugin Markers: Fixed restores onto a GPU other than the source GPU being refused with
mount: source /run/nvidia-container-devices/GPU-<uuid>, target missingwhen the NVIDIA device plugin uses thevolume-mountsdevice list strategy. A checkpoint now skips the marker (#814). - Multi-GPU: Kept
cudaMallocsemantics for converted allocations (#673). - Restore Start: Started restores when NRI reports the destination container (#704).
- Builds: Fetched PageBroker dependency sources in the install layer (#681) and updated the Go toolchain to 1.27.3 (#668).
Testing & CI
- GPU tests for multi-GPU sharing, peer carriers and coordinator orchestration, and a CI job that runs the CPU and GPU suites (#338, #566, #578, #579, #580, #581, #582).
- End-to-end coverage for SnapshotJob partial-success recovery, CPU restore failure and scheduling, graceful shutdown of a restored workload, and the framework checkpoint and restore test over S3 on the GPU cluster (#518, #515, #713, #727).
- CI runners are pinned to Ubuntu 24.04, and a branch's first image build can start from the
maincache (#667, #747).
Documentation
- Lifecycle Hooks: Documented in the API reference, the workload contract, the architecture reference, troubleshooting and limitations (#746).
- Storage: Started the storage best practices guide, with a backend comparison and a slow-restore walkthrough (#775).
- Naming: Reworded the user-facing pages, the CRD field descriptions, the chart README and the condition and failure messages to say checkpoint and snapshot where they said capture (#794).
- Architecture: the architecture reference and the multi-GPU design page (#683, #337), and documented and exercised multi-GPU workloads in the examples (#456).
- Benchmarks: extended the published benchmarks to SGLang, TensorRT-LLM and multiple GPUs (#661).
- Troubleshooting: organized by checkpoint and restore step (#588).
- Contributor Guides: documentation and naming guides for contributors and agents (#699, #745).
- Proposals: SNEP-411, application contracts for checkpoint and restore (#437).
Known Issues
None recorded.
Contributors
86 merged PRs from 18 contributors. Thank you to everyone who contributed to this release, including @galletas1712 and @hhzhang16, who landed the vLLM and TensorRT-LLM changes above, and @leeZardNav, who opened the vLLM proposal. Welcome to the 7 first-time contributors:
- @nv-tusharma pinned the CI runners to Ubuntu 24.04 (#667).
- @zanes-ops prepared restore participants with GPU data first (#672).
- @tryambak2019 added end-to-end coverage for SnapshotJob partial-success recovery (#518).
- @Iceber made the operator wait as Pending for an unscheduled source Pod (#697).
- @youssef-lyceum restored the ranks of a multi-GPU workload in parallel (#757).
- @aleksandr-inherent cut the operator's Pod cache to the fields its controllers read (#761).
- @skpulipaka26 let the chart accept
daemonset.affinity.nodeAffinity(#799).
Full Changelog: v0.2.0...v0.3.0