Docker driver: --gpu allocates GPU but driver libraries (libnvidia-ml.so) not visible inside sandbox

[PhenixStar]

Summary

On the Docker compute driver, openshell sandbox create --gpu 1 successfully allocates a GPU (CDI device selected, container starts, supervisor relay connects, the command runs over SSH), but the NVIDIA driver userspace libraries are not present inside the sandbox. nvidia-smi fails with:

NVIDIA-SMI couldn't find libnvidia-ml.so library in your system.

find / -name 'libnvidia-ml.so*' inside the sandbox returns nothing, even though the exact same CDI device injects the libraries correctly under raw docker run. The nvidia-smi binary is injected, but none of the driver .so libraries are.

Environment

• OpenShell: 0.0.76 (snap, latest/stable)
• Docker: docker snap 29.3.1 (rev 3505)
• snapd: 2.76
• NVIDIA driver: 535.309.01 (8× Tesla P40)
• Kernel: 6.8.0-124-generic
• OS: Ubuntu 22.04.5 LTS
• Gateway: docker driver, gateway_jwt enabled, allow_unauthenticated_users = true (local single-user, loopback)

What works — raw docker run with the same CDI device

$ docker run --rm --device nvidia.com/gpu=all ubuntu:22.04 nvidia-smi -L
GPU 0: Tesla P40 (UUID: GPU-84ae8966-...)
... all 8 GPUs listed ...

CDI spec: /var/snap/docker/current/etc/cdi/nvidia.yaml (auto-generated by the docker snap); dockerd cdi-spec-dirs includes that path.

What fails — inside an OpenShell sandbox

Base image built with the sandbox contract satisfied (glibc 2.39 via ubuntu:24.04, iproute2, a sandbox user/group):

$ openshell sandbox create --gpu 1 --from <that image> --no-keep --no-tty \
    -- sh -lc 'ldconfig; find / -name "libnvidia-ml.so*"; nvidia-smi -L'
Created sandbox: ...
  Requesting compute...
  Sandbox allocated
  Starting sandbox Container created
  Starting sandbox Waiting for supervisor relay
NVIDIA-SMI couldn't find libnvidia-ml.so library in your system.
# (find returned no libnvidia-ml.so anywhere in the sandbox)

Diagnosis

The CDI spec injects the driver libraries via bind-mounts sourced from the docker snap's hostfs view (/var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/...) plus createContainer hooks (nvidia-cdi-hook) that create the libnvidia-ml.so.1 symlink and update ldcache. Under raw docker run this all works.

Inside an OpenShell sandbox, the supervisor performs network-namespace + mount isolation, and the CDI-injected library bind-mounts do not survive into the sandbox's mount namespace. The device nodes (/dev/nvidia*) and the nvidia-smi binary mount appear, but the driver library mounts / ldcache entries do not.

Things that do not help

• NVIDIA_DRIVER_CAPABILITIES=all / NVIDIA_VISIBLE_DEVICES=all baked into the image
• Running ldconfig inside the sandbox (the libraries are genuinely absent, not just uncached)

Hypothesis / question

This looks specific to the docker snap, whose CDI mounts are sourced from /var/lib/snapd/hostfs/.... Does the sandbox supervisor's mount isolation drop mounts that originate outside the container's expected roots? Is the Docker --gpu path expected to work with the confined docker snap, or only with system-installed Docker (now supported on snapd 2.76)? Happy to test any patch or a system-Docker configuration.

[elezar]

This looks specific to the docker snap, whose CDI mounts are sourced from /var/lib/snapd/hostfs/....

I think this is the key. The supervisor adds the expected library paths from the host to the read-only baseline here. This does not include the driver paths under /var/lib/snapd/hostfs.

To confirm: locate the driver files in the docker container running using docker run natively, and create a policy that allows readonly access to some parent of these files.

Note that #1606 would address this since we would adjust our sandbox policy based on what is actually being injected.