Add CDI support to --devices

[klueska]

Description

Feature Request:
Add support to the --device flag to process fully-qualified CDI device names in addition to standard device node paths.

Support for this was added in podman v4.1.0 in May 2022 and it would be good to have feature parity with this in docker.

Background:
The Container Device Interface (CDI) is a CNCF sponsored initiative to standardize the way in which complex devices are exposed to containers. It is based on the Container Networking Interface (CNI) model and specification.

With CDI, a "device" is more than just a single device node under /dev.

Instead, CDI defines a device as a high-level concept mapped to specific set of OCI runtime spec modifications. These modifications include not just the inclusion of device nodes, but also filesystem mounts, environment variables, and container lifecycle hooks.

Vendors are responsible for writing CDI specs that define their devices in terms of these modifications, and CDI-enabled container runtimes are responsible for reading these specs and making the proper modifications when requested to do to.

The list of container runtimes that already include support for CDI are:

• podman
• cri-o
• containerd

Moreover, a new feature in Kubernetes called Dynamic Resource Allocation (DRA) uses CDI under the hood to do its device injection (with the assumption that a CDI-enabled runtime such as cri-o or containerd is there to support it).

As a concrete example, consider the following abbreviated CDI spec for injecting an NVIDIA GPU into a container:

---
cdiVersion: 0.4.0
kind: nvidia.com/gpu
devices:
- name: gpu0
  containerEdits:
   deviceNodes:
   - path: /dev/nvidia0
   deviceNodes:
   - path: /dev/nvidiactl
   mounts:
   - containerPath: /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.460.91.03
     hostPath: /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.460.91.03
     options: [ro, nosuid, nodev, bind]
   - containerPath: /usr/lib/x86_64-linux-gnu/libcuda.so.460.91.03
     hostPath: /usr/lib/x86_64-linux-gnu/libcuda.so.460.91.03
     options: [ro, nosuid, nodev, bind]
   - containerPath: /usr/bin/nvidia-smi
     hostPath: /usr/bin/nvidia-smi
     options: [ro, nosuid, nodev, bind]
   - containerPath: /usr/bin/nvidia-persistenced
     hostPath: /usr/bin/nvidia-persistenced
     options: [ro, nosuid, nodev, bind]
   - containerPath: /var/run/nvidia-persistenced/socket
     hostPath: /var/run/nvidia-persistenced/socket
     options: [ro, nosuid, nodev, bind]
   hooks:
   - hookName: createContainer
     path: /usr/bin/nvidia-ctk
     args:
     - /usr/bin/nvidia-ctk
     - hook
     - update-ldcache
     - --folder
     - /usr/lib/x86_64-linux-gnu

This spec defines a device whose name is gpu0 associated with vendor nvidia.com and device class gpu -- resulting in a fully-qualified CDI device name of nvidia.com/gpu=gpu0.

Referencing this fully-qualified device name and passing it to a CDI-enabled runtime would ensure that not only does the /dev/nvidia0 device node get injected into a container, but also the required /dev/nvidiactl control device, as well as a set of host-level libraries and hooks to make working with NVIDIA devices in containers easier.

Note: For those of you familiar with nvidia-docker and / or the nvidia-container-toolkit, this effectively obsoletes the need for for these tools, because everything these tools have traditionally been responsible for can now be encoded in a CDI spec.

As such, saving the above spec under /etc/cdi/nvidia.yaml and running podman as below results in the container starting with the desired modifications:

$ podman --version
podman version 4.1.0

$ podman run --device nvidia.com/gpu=gpu0 ubuntu:20.04 nvidia-smi -L
GPU 0: Tesla T4 (UUID: GPU-2e5daa54-7530-ede9-5b96-d4e31fbbe7f8)

[klueska]

/cc @elezar

[klueska]

/cc @thaJeztah

[Clockwork-Muse]

For those of you familiar with nvidia-docker and / or the nvidia-container-toolkit, this effectively obsoletes the need for for these tools, because everything these tools have traditionally been responsible for can now be encoded in a CDI spec.

While it's certainly possible to create the spec-file manually, the tool nvidia is writing to generate it (nvidia-ctk) is currently in nvidia-container-toolkit-base, so it's probably still a necessary package.
Note that even without CDI support, it seems like nvidia-container-toolkit (1.12 rc3+, anyways) and its dependencies seems to be the only package required, so nvidia-docker2/the runtime package doesn't seem to be required anyways.

One additional benefit to CDI support is that it enables using a non-nvidia base image in GPU-related scenarios (although you may still be required to download/install some packages depending on you use case, like X11 libraries). For instance, with podman 4.3.1+ and nvidia-container-toolkit 1.12+ the command podman run --rm --device nvidia.com/gpu=gpu0 docker.io/ubuntu:latest nvidia-smi generates the expected output.
(Actually, in accordance with the note above, with nvidia-container-toolkit 1.12+ you can run docker run --rm --gpus all docker.io/ubuntu:latest nvidia-smi and get the expected output there too, but without the actual CDI spec support there are odd holes for things like EGL that cause multiple use-cases to fail.)

[corhere]

CDI support would need to be added to Docker Engine first; this repo is merely for the main command-line client.

We discussed this proposal in yesterday's Moby maintainer call. The main concerns about adding CDI support to Docker Engine centered around the visibility of the edits applied to a container's spec. Since it mutates a container's OCI spec directly, Docker Engine would be unaware of what is being injected and therefore e.g. docker container inspect output would not reflect the injected bind mounts, environment variables, etc. How would a user troubleshoot an issue with a container caused by e.g. a CDI spec injecting a shared library linked against glibc into a musl-based container? What happens if a CDI device's container edits conflict with user-configured environment variables, mount points, capabilities, etc?

We do not want to overload the --device flag (and its Docker API counterpart) with CDI semantics as that could introduce ambiguities. Currently a device means exactly one thing: projecting a device-special file into the container on Linux, or a device class on Windows. Given the lack of validation in the current implementation, it would be trivial to craft a (technically non-conforming) CDI spec file which defines a device with the same name as a device-special file path. There would be no way to tell ahead of time whether docker run --device /dev/foo would project the device file /dev/foo into the container or apply the CDI transformations for vendor.com/bar=/dev/foo to the container. And it would be really hard to spot in a bug report that --device /dev/foo was interpreted as a CDI device name and not a device file path, which would waste our time and the user's. A user should have to be explicit and unambiguous in their request to use a CDI device with their container, in both the API and CLI. (We've been burned by this before: compare the --volume flag vs. --mount.)

Not everything has to be solved before we would accept PRs, so long as the feature is gated behind an experimental flag and there are plans to address all our concerns in a reasonable amount of time. If you don't want to jump straight into opening a PR, I would suggest opening a Feature Request issue on https://github.com/moby/moby with an outline of the daemon configuration and Engine API changes you would like to make to enable CDI support.

[neersighted]

Related: moby/moby#36987

[klueska]

@corhere

Thanks for taking the time to write such a detailed response.

We discussed this proposal in yesterday's Moby maintainer call.

Glad to hear that this was discussed in the last meeting. Does it make sense for someone from my team to attend the next meeting to discuss things further?

The main concerns about adding CDI support to Docker Engine centered around the visibility of the edits applied to a container's spec. Since it mutates a container's OCI spec directly, Docker Engine would be unaware of what is being injected and therefore e.g. docker container inspect output would not reflect the injected bind mounts, environment variables, etc. How would a user troubleshoot an issue with a container caused by e.g. a CDI spec injecting a shared library linked against glibc into a musl-based container?

The expectation would be that docker would read the CDI device specs itself and extract this information to include in its inspect output. The CDI API library has functions to make this easy (or if any are missing they could easily be added).

One thing also worth pointing out is that what you describe is already a problem with --gpus. When --gpus is passed, it ultimately results in a hook being invoked that bind mounts a set of device nodes, library mounts, etc. (as well as setting up cgroups for those devices). This all happens behind the back of docker (and containerd for that matter), resulting in lots of confusion and a multitude of bug reports.

At least with CDI you now have a well-defined format for gaining visibility into what will ultimately be injected by the CDI machinery under the hood and can react to it accordingly.

What happens if a CDI device's container edits conflict with user-configured environment variables, mount points, capabilities, etc?

I would expect that any user-configured fields would override what is specified in the CDI spec for a device. I would need to double check how the CDI libraries handle this case though (and also hear what the CDI developers themselves have to say about this @elezar @klihub @bart0sh).

We do not want to overload the --device flag (and its Docker API counterpart) with CDI semantics as that could introduce ambiguities. Currently a device means exactly one thing: projecting a device-special file into the container on Linux, or a device class on Windows. Given the lack of validation in the current implementation, it would be trivial to craft a (technically non-conforming) CDI spec file which defines a device with the same name as a device-special file path. There would be no way to tell ahead of time whether docker run --device /dev/foo would project the device file /dev/foo into the container or apply the CDI transformations for vendor.com/bar=/dev/foo to the container. And it would be really hard to spot in a bug report that --device /dev/foo was interpreted as a CDI device name and not a device file path, which would waste our time and the user's. A user should have to be explicit and unambiguous in their request to use a CDI device with their container, in both the API and CLI. (We've been burned by this before: compare the --volume flag vs. --mount.)

I think this needs to be discussed further. From my perspective, it would, in fact, be added as an extension to --device ideally. It would be good to understand if anything would change your mind here or if this is a hard no.

As a starting point, validation, for example, could always be added to the CDI libs to ensure that a kind never starts a /.

Not everything has to be solved before we would accept PRs, so long as the feature is gated behind an experimental flag and there are plans to address all our concerns in a reasonable amount of time. If you don't want to jump straight into opening a PR, I would suggest opening a Feature Request issue on https://github.com/moby/moby with an outline of the daemon configuration and Engine API changes you would like to make to enable CDI support.

I think we will start with a feature request that outlines the proposed changes. Thanks again for your help on this.