nvproxy: `frontendFD`s are present during `runsc checkpoint` despite being closed

[mattnappo]

Description

Hello! I am currently using cuda-checkpoint within gVisor, and runsc checkpoint is failing with nvproxy.frontendFD is not saveable. From my understanding, this error means that a /dev/nvidia#, /dev/nvidiactl, or /dev/nvidia-uvm file descriptor is open at the time of a gVisor snapshot. However, running lsof -nP inside the container after cuda-checkpoint indicates that none of these file descriptors are currently open. Why is there a discrepancy between lsof and the nvproxy.frontendFDs set? Is it possible that gVisor has a bug where it holds on to these frontendFD file descriptors even if they're closed by the guest?

Steps to reproduce

I came across this issue when attempting to checkpoint/restore sglang.

Step 1: Build the image from this Dockerfile

FROM nvidia/cuda:12.9.1-devel-ubuntu22.04

ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y \
    wget \
    git \
    python3 \
    python3-pip \
    lsof \
    && rm -rf /var/lib/apt/lists/*

# Install CUDA Toolkit (nvcc)
RUN arch="x86_64" && \
    distro="debian12" && \
    filename="cuda-keyring_1.1-1_all.deb" && \
    cuda_keyring_url="https://developer.download.nvidia.com/compute/cuda/repos/${distro}/${arch}/${filename}" && \
    wget ${cuda_keyring_url} && \
    dpkg -i ${filename} && \
    rm -f ${filename} && \
    apt-get update && \
    apt-get install -y cuda-nvcc-12-9 && \
    rm -rf /var/lib/apt/lists/*

RUN apt-get update -y && apt-get upgrade -y
RUN apt-get install numactl -y

# Clone and install sglang
RUN pip install --upgrade pip setuptools
RUN git clone https://github.com/sgl-project/sglang.git /sglang && \
    cd /sglang && \
    pip install -e "python[all]"

RUN wget https://github.com/NVIDIA/cuda-checkpoint/raw/refs/heads/main/bin/x86_64_Linux/cuda-checkpoint /usr/bin/cuda-checkpoint

CMD ["python3", "-c", "import sglang.srt.entrypoints.engine; import time; import os; print(f'pid: {os.getpid()}', flush=True); time.sleep(1000)"]

Step 2: Prepare the container
Follow the instructions here to create an OCI bundle and config. Make make sure to setup GPU support. This guide is useful.

Step 3: Running the container
Run the container: runsc -nvproxy run -bundle bundle/ sglang-container

Then, once the PID is printed, run cuda-checkpoint inside the container: runsc exec sglang-container cuda-checkpoint --toggle --pid 1. This step should succeed.

You can now verify that no handles to /dev/nvidia* are present: runsc exec sglang-container lsof -nP.

Step 4: Checkpoint the container
This step fails: checkpoint -image-path checkpointed_img/ sglang-container with the error message pasted below. I've also uploaded the container's lsof -nP output, grepped for cuda|nvidia. I'm also seeing lots of the following, and I'm not sure if it's related.

object.go:260 [ 1689: 1689] nvproxy: freeing object with unknown handle 0xc1d09e26:0x5c000006
frontend.go:76 [ 1707: 1707] nvproxy: failed to open nvidia0: no such file or directory

Log output

• runsc_boot_logs.txt
• lsof_post_cuda_checkpoint.txt

runsc version

docker version (if using docker)

uname

oracle linux 9.6, kernel 5.15.0-309.180.4.el9uek.x86_64

kubectl (if using Kubernetes)

N/A

repo state (if built from source)

No response

runsc debug logs (if available)

See the above file upload.

[nixprime]

Minor diff to your Dockerfile:

33c33,34
< RUN wget https://github.com/NVIDIA/cuda-checkpoint/raw/refs/heads/main/bin/x86_64_Linux/cuda-checkpoint /usr/bin/cuda-checkpoint
---
> RUN wget -O /usr/bin/cuda-checkpoint https://github.com/NVIDIA/cuda-checkpoint/raw/refs/heads/main/bin/x86_64_Linux/cuda-checkpoint
> RUN chmod 0755 /usr/bin/cuda-checkpoint

I ran this image using Docker rather than raw runsc, but I see there being many processes other than PID 1 with /dev/nvidia* FDs open:

root@6493efaba133:/# cuda-checkpoint --get-state --pid 1
checkpointed
root@6493efaba133:/# cuda-checkpoint --get-state --pid 44
running
root@6493efaba133:/# head -n 1 /proc/44/status
Name:   python3
root@6493efaba133:/# lsof -nP | grep /dev/nvidia
python3  44               root  mem       CHR              195,0               20 /dev/nvidia0
python3  44               root  mem       CHR            195,255               18 /dev/nvidiactl
...

[ayushr2]

@mattnappo What driver version are you using? Try 570+, as that has NVML support, without which cuda-checkpoint won't close all GPU device FDs.

[mattnappo]

I’m on 575

…

On Wed, Sep 17, 2025 at 7:01 PM Ayush Ranjan ***@***.***> wrote: *ayushr2* left a comment (google/gvisor#12144) <#12144 (comment)> @mattnappo <https://github.com/mattnappo> What driver version are you using? Try 570+, as that has NVML support <https://github.com/NVIDIA/cuda-checkpoint?tab=readme-ov-file#570-features>, without which cuda-checkpoint won't close all GPU device FDs. — Reply to this email directly, view it on GitHub <#12144 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE7GYU6DYR66EFVMETN3JPL3THR6FAVCNFSM6AAAAACGZRCSQ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTGMBUG44TIOJRHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[ayushr2]

@mattnappo As Jamie pointed out above, seems like PID 44 is also a CUDA process, you need to run cuda-checkpoint on that as well. In general, you need to toggle all CUDA pids. PID 1 may spawn multiple CUDA sub-processes.

[mattnappo]

Even after running cuda-checkpoint on these additional PIDs (it seems that only some of them can be checkpointed), I get the following gvisor error when creating the checkpoint:

I0918 22:25:03.973655  2648933 main.go:213] **************** gVisor ****************
D0918 22:25:03.973684  2648933 state_file.go:76] Load container, rootDir: "/var/run/runsc", id: {SandboxID: ContainerID:sglang-repro}, opts: {Exact:false SkipCheck:false TryLock:false RootContainer:false}
D0918 22:25:03.974797  2648933 sandbox.go:1997] ContainerRuntimeState, sandbox: "sglang-repro", cid: "sglang-repro"
D0918 22:25:03.974821  2648933 sandbox.go:768] Connecting to sandbox "sglang-repro"
D0918 22:25:03.974892  2648933 urpc.go:592] urpc: successfully marshalled 72 bytes.
D0918 22:25:03.975115  2648933 urpc.go:635] urpc: unmarshal success.
D0918 22:25:03.975154  2648933 sandbox.go:2002] ContainerRuntimeState, sandbox: "sglang-repro", cid: "sglang-repro", state: 2
D0918 22:25:03.975217  2648933 container.go:729] Checkpoint container, cid: sglang-repro
D0918 22:25:03.975232  2648933 sandbox.go:1470] Checkpoint sandbox "sglang-repro", imagePath "checkpointed_img", opts {Compression:flate-best-speed Resume:false Direct:false ExcludeCommittedZeroPages:false SaveRestoreExecArgv: SaveRestoreExecTimeout:10m0s SaveRestoreExecContainerID:sglang-repro}
D0918 22:25:03.975272  2648933 sandbox.go:768] Connecting to sandbox "sglang-repro"
D0918 22:25:03.975342  2648933 urpc.go:592] urpc: successfully marshalled 308 bytes.
W0918 22:25:08.063391  2648933 util.go:64] FATAL ERROR: checkpoint failed: checkpointing container "sglang-repro": urpc method "containerManager.Checkpoint" failed: EOF

[EtiennePerot]

Please provide the *.boot.txt logs, as the runsc checkpoint logs don't contain a useful error here.

[mattnappo]

boot.txt