feat: Support NVIDIA k8s-device-plugin with CDI

[LandonTClipp]

Description

gVisor supports two paths for injecting NVIDIA devices:

1. Through the legacy OCI pre-start hook created by
   nvidia-container-runtime-hook (legacy)
2. Through CDI annotations.

When gVisor detected GPUs were being attached through the legacy hook
method, it would correctly run the nvidia-container-cli to modify the
container's rootfs to create the NVIDIA library symlinks that client
applications need (through ldconfig) to speak to the host driver. When
the NVIDIA k8s-device-plugin is used in CDI mode, it injects the client
libraries but not the symlinks. The only step that was missing was the
symlink creation with nvidia-container-cli.

This commit modifies the GPU detection logic to check both the CDI
annotations for the presence of GPUs and the legacy hook-based method.
If GPUs are detected in the CDI annotations, the code path to run the
symlinking step is performed.

The normal limitations apply in this scenario regarding
nvidia-container-cli, namely that the container's rootfs must be
writable. If it's not writable, like when using a readonly image
distribution mechanism (EROFS for example), this will not work. This
limitation applied in the legacy scenario anyway, so there is no
regression.

Doesn't k8s-device-plugin already do symlinking?

You'll notice in k8s-device-plugin, it already has hooks for creating these symlinks. For example:

  "containerEdits": {
    "hooks": [
      {
        "hookName": "createContainer",
        "path": "/usr/bin/nvidia-ctk",
        "args": [
          "nvidia-ctk",
          "hook",
          "create-symlinks",
          "--link",
          "libnvidia-opticalflow.so.1::/usr/lib/x86_64-linux-gnu/libnvidia-opticalflow.so",
          "--link",
          "libGLX_nvidia.so.580.126.20::/usr/lib/x86_64-linux-gnu/libGLX_indirect.so.0",
          "--link",
          "libcuda.so.1::/usr/lib/x86_64-linux-gnu/libcuda.so"
        ]
      },
    ],

However, gVisor explicitly does not run createContainer hooks because they're not supported.

This is why gVisor must run nvidia-container-cli configure manually to create the symlinks and update the ldconfig cache.

[LandonTClipp]

@ayushr2 one major question I have is whether this will break GKE's infrastructure. I had to make an assumption that GKE environments are somehow performing the client library symlinking that is missing. In this case, it might make sense to introduce a config parameter that toggles the behavior in this PR and keep it disabled by default. You'll have to let me know what you think.

[ayushr2]

Yeah I think this might break GKE because specutils.GPUFunctionalityRequested() is true for GKE. I am not sure if nvidia-container-cli is present in GKE Nodes.

If the GPU devices are already bind-mounted into the container filesystem, then we should probably not run nvidia-container-cli configure again, that bind mounts devices again. All we want to do is emulate the createContainer hook (by only invoking nvidia-ctk to create symlinks), like we emulate the prestart hook.

[LandonTClipp]

I will look into narrowing the scope of what we're doing to the rootfs with nvidia-ctk. I did test the current method on our systems and it works as expected but I did suspect I was using too broad of a hammer.

[ayushr2]

Also, I think you might be able to get away with just passing --nvproxy-docker runsc flag. It forces the nvidia prestart hook code path even when the OCI spec doesn't have that hook. So you don't need this PR.

But I still think the right thing to do is to emulate nvidia-ctk invocation in gVisor for symlink creation. Because the pre-start hook is doing extra work.

[LandonTClipp]

The issues I have with the GPUFunctionalityRequestedViaHook code path was that it would mount all of the /dev/nvidiaN devices from the host, not just the one that was requested by CDI. This is why I started investigating if we can just use raw CDI for everything because the k8s-device-plugin gives us a CDI spec file, that if adhered to correctly, gives a correct working environment inside the container.

The issues I discovered were multi-layered:

1. gVisor would correctly mount the client library as specified in the CDI spec file. Example:

      {
        "hostPath": "/usr/lib/x86_64-linux-gnu/libcuda.so.580.126.20",
        "containerPath": "/usr/lib/x86_64-linux-gnu/libcuda.so.580.126.20",
        "options": [
          "ro",
          "nosuid",
          "nodev",
          "bind"
        ]
      },
   

   But it would not create the libcuda.so -> libcuda.so.1 symlink that is supposed to be created by this createContainer hook:

      {
        "hookName": "createContainer",
        "path": "/usr/bin/nvidia-ctk",
        "args": [
          "nvidia-ctk",
          "hook",
          "create-symlinks",
          "--link",
          "libGLX_nvidia.so.580.126.20::/usr/lib/x86_64-linux-gnu/libGLX_indirect.so.0",
          "--link",
          "libcuda.so.1::/usr/lib/x86_64-linux-gnu/libcuda.so",
          "--link",
          "libnvidia-opticalflow.so.1::/usr/lib/x86_64-linux-gnu/libnvidia-opticalflow.so"
        ]
   

   Because createContainer hooks were explicitly not run.

2. After the libcuda.so -> libcuda.so.1 symlink is created, you're supposed to also update the ldcache. This step completes the symlink chain so that libcuda.so.1 -> libcuda.so.580.126.20. This step also wasn't being run because it's a createContainer hook.

3. The reason why createContainer hooks were disabled appears to be because nobody had figured out how to make the symlinking and ldcache dance work with the gofer (filesystem proxy) and the sentry (thing that manages the VFS). I have a working solution internally that I will submit as a PR soon.

[LandonTClipp]

Closed in favor of #13034