Mode-Aware GPU Checkpoint

한국어 문서: README.ko.md

One Kubernetes operator that checkpoints GPU workloads through either of two mechanisms — system level (GCR interceptor + CRIUgpu) or application level (FluidCR) — chosen by a single field, not by a separate resource kind.

Status: builds clean; never run against a cluster. go vet and go build pass. No container image has been published and no checkpoint has been taken with this code. See Status.

---

Why one operator instead of two

The two checkpoint paths look different at the front but converge at the back:

[system]        interceptor freeze  ->  data.blob        ┐
                                                          ├─>  kubelet Checkpoint API  ->  tar
[application]   fluidcr-ctrl -> state_dict -> lock       ┘     POST https://<node>:10250
                                                                /checkpoint/<ns>/<pod>/<container>

Everything after that call — snapshot retrieval, upload to storage, status reporting — is identical. Splitting the system in two would duplicate all of it.

Three consequences shape this design:

mode is a field, never a kind	An AppCheckpoint kind would differ from GPUCheckpoint by exactly one derivable field (fluidcrEndpoint, which is <podIP>:8298). One field is not worth a second CRD, controller, RBAC set and webhook match.
One agent, two quiescers	Both paths need nodes/checkpoint create — the strongest permission in play. Separating them issues the same node-level credential twice and buys no real isolation.
Mode exclusivity gets one enforcement point	With two independent controllers nothing stops both from firing on the same Pod. With one agent it is a single check before Prepare.

---

Custom Resources

Two kinds. mode selects the quiescer; everything else is shared.

WorkloadCheckpoint — what you write

apiVersion: gpu-cr.io/v1alpha1
kind: WorkloadCheckpoint
metadata:
  name: train-system
spec:
  workloadRef: {kind: Deployment, name: cuda-train}
  container: cuda-app
  mode: system                    # system | application
  storageRef:
    type: nfs                     # local | nfs | s3 | http
    endpoint: "10.178.0.14"       # "none" if there is no external server
    path: /mnt/nfs/gcr
  schedule: "0 */6 * * *"         # "" = run once

GPUCheckpoint — what the controller generates

One per Pod. mode, storageRef and schedule are stamped down from the parent so the node agent can act on this object alone — no cross-object read, so less RBAC and a simpler reconcile.

Its status carries the entire handover to the restore side:

status:
  mode: system
  checkpointURI: "nfs://10.178.0.14/mnt/nfs/gcr/cuda-train-7c9d8-abc-1786340472.tar"
  sourcePodUID:  "c8a67622-b7bf-4ee4-a236-26a909f79aa9"

Three values. Two things deliberately absent:

• data.blob location — same stem as the tar with a .blob suffix. Derived, not stored.
• a second storage descriptor — checkpointURI already carries scheme, host and path.

storageRef composes into the URI like this:

nfs   + 10.178.0.14 + /mnt/nfs/gcr           ->  nfs://10.178.0.14/mnt/nfs/gcr/<pod>-<ts>.tar
local + none        + /var/lib/gcr-checkpoint ->  hostpath:///var/lib/gcr-checkpoint/<pod>-<ts>.tar

type: local keeps the checkpoint on the node that produced it, so it cannot be restored elsewhere. Use nfs, s3 or http if the workload may move.

---

What each mode actually does

system — GCR interceptor + CRIUgpu

GPU state is split in two on purpose:

what	who handles it	where it lands
GPU data buffers (the large part)	interceptor copies to host, then munmaps	data.blob
GPU control state + CPU process	CRIU + NVIDIA cuda_plugin	.tar

Buffers are pulled out before the dump because leaving them in would blow up CRIU's dump size and time. The cost is that a complete checkpoint is tar + blob — the tar alone cannot be restored.

Sequence: signal the interceptor to freeze → wait for data.blob → call the kubelet Checkpoint API → signal remap so the original Pod keeps running.

application — FluidCR

FluidCR patches PyTorch through a PEP-451 import hook. On SIGUSR1 the worker saves its state_dict and exits 99; the launcher buffers it and writes /checkpoint/<PID>/lock.

Sequence: POST <podIP>:8298/checkpoint → wait for the lock file → call the kubelet Checkpoint API. The GPU is already torn down at dump time, so there is no blob and the checkpoint is roughly 63% smaller than the system-mode one.

---

Prerequisites

The checkpoint side sits on top of a runtime that is already verified. Get these in place first.

	requirement	check
Runtime	CRI-O with the merged GPU restore patches	crio --version
CRIU	4.2.x with cuda_plugin.so	ls /usr/lib/criu/cuda_plugin.so
crun	built against the same libcriu	crun --version | grep CRIU
Driver	NVIDIA 570+	nvidia-smi --query-gpu=driver_version --format=csv
kubelet	ContainerCheckpoint feature gate on	kubectl get --raw /api/v1/nodes/<node>/proxy/configz | grep -i checkpoint
Device plugin	NVIDIA device plugin present	kubectl -n kube-system get ds | grep nvidia
application mode only	FluidCR inside the image	pip show fluidcr in the Pod

One trap worth knowing before you start

/etc/criu/default.conf must not contain tcp-close if you intend to checkpoint anything holding live TCP connections — it closes them at dump time and the restore comes back broken. Note that some setup scripts write it back in, so check after any node rebuild.

---

Install

kubectl apply -f config/namespace.yaml
kubectl apply -f config/crd/
kubectl apply -f config/rbac/

kubectl get crd | grep gpu-cr.io
# gpucheckpoints.gpu-cr.io        2026-08-12T…
# workloadcheckpoints.gpu-cr.io   2026-08-12T…

The controller Deployment and agent DaemonSet are not published yet — see Status.

---

Walkthrough

system mode

kubectl apply -f config/samples/workloadcheckpoint-system.yaml

kubectl get wckpt train-system
# NAME           MODE     WORKLOAD     PHASE     SCHEDULE      AGE
# train-system   system   cuda-train   Running   0 */6 * * *   5s

# one child per Pod
kubectl get gckpt -l gpu-cr.io/workload-checkpoint=train-system

When a child reaches Completed, read the three values the restore side needs:

kubectl get gckpt <name> -o jsonpath='{.status.mode}{"\n"}{.status.checkpointURI}{"\n"}{.status.sourcePodUID}{"\n"}'

Confirm both artifacts exist — this is the step people skip:

ls -la /mnt/nfs/gcr/cuda-train-7c9d8-abc-1786340472.tar
ls -la /mnt/nfs/gcr/cuda-train-7c9d8-abc-1786340472.blob

And confirm the original Pod resumed rather than died:

kubectl logs cuda-train-7c9d8-abc | grep -E 'freeze:|remap:'
# [gcr][engine] remap: external blob -> VA + H2D; 0 failed

application mode

kubectl apply -f config/samples/workloadcheckpoint-application.yaml

The one signal that matters is the lock file appearing before the snapshot:

kubectl exec <pod> -- ls -la /checkpoint/*/
# latest.pt   lock          <- both present = safe to snapshot

Restoring is expected to resume at the checkpointed step, not step 0. That is the application-mode equivalent of a byte-identical checksum.

---

Verifying a checkpoint is genuinely correct

"The command returned 0" is not verification. Two checks are.

system mode — the tensor survived. Hash a GPU buffer before checkpoint and after restore; the digests must be identical. Anything else means the buffer was reallocated rather than restored.

application mode — training resumed. The first step printed after restore must be near the checkpointed step. epoch 0 step 0 means the state was thrown away and the container merely restarted.

---

Troubleshooting

symptom	cause	fix
Error: No mapping for /var/lib/gcr-data mountpoint	CRIU requires an ext-mount-map entry for every external bind mount; GCR's hostPaths are external	the merged CRI-O generates these — check you are on the patched runtime
Restore never starts, no CRIU log	crun and libcriu built against different versions, so the CRIU config never reaches CRIU	rebuild crun against the installed libcriu
Pod restores but GPU memory is stale	the blob remap step was skipped	check remap: in the Pod log; confirm data.blob was staged
Restored container has no /dev/nvidia*	the runtime skips NVIDIA devices unconditionally when CDI is assumed	requires the conditional CDI guard — again, the patched runtime
Checkpoint fails -52 "Connected TCP socket"	dump ran while a TCP connection was ESTABLISHED and tcp-close is absent	expected trade-off; checkpoint when idle, or accept tcp-close and lose live connections
data.blob missing, system mode	the interceptor never got the freeze signal	check LD_PRELOAD and that /var/lib/gpu-cr/run is mounted

---

Status

make build passes (go vet clean, both binaries produced). What has not happened: no image built, no deployment, no checkpoint taken. Everything below marked "runtime-unverified" is code that compiles but has never executed.

component	state
CRD definitions (config/crd/)	✅ written
RBAC (config/rbac/)	✅ written
Sample manifests (config/samples/)	✅ written, validated against the CRD schemas
API types + deepcopy (api/v1alpha1/)	✅ compiles
WorkloadCheckpoint controller	✅ compiles, ⬜ runtime-unverified
Node agent + Quiescer interface	✅ compiles, ⬜ runtime-unverified
gcrQuiescer	✅ compiles, ⬜ runtime-unverified
fluidcrQuiescer	✅ compiles, ⬜ runtime-unverified
kubelet + FluidCR control clients	✅ ported from FluidCR (see NOTICE)
Deployment / DaemonSet manifests	✅ written
Container images	⬜ not built or published
End-to-end run on a cluster	⬜ never executed
Merged CRI-O runtime (dependency)	✅ implemented and measured, separate repository

Known gaps

• storageRef.type s3 and http return "not implemented" rather than silently dropping a checkpoint. local and nfs work as a filesystem copy, which is why the agent mounts the storage root.
• schedule supports "" (one-shot) and "@every 6h". Cron syntax needs a parser dependency that has not been added.
• Mode exclusivity is not enforced yet — nothing rejects a Pod that carries both annotation families.

Build

make build        # verified: go vet clean, both binaries produced
make test
make docker-build REGISTRY=<your-registry> TAG=<tag>
make deploy

---

Design notes

• mode is a value, so it is a field. Storage is a different kind of data, so it could have been a kind — it stayed inline because three fields do not justify the indirection.
• The agent is a DaemonSet for two reasons only: the kubelet Checkpoint endpoint is per-node, and system mode needs hostPath access to /var/lib/gcr-data and the interceptor socket.
• The restore side reads three annotations and nothing else. Keeping that contract at three is why credentials, when a storage backend needs them, are injected into the Pod as a Secret rather than passed as a fourth annotation.

License

Apache License 2.0 — see LICENSE.