docker sees all GPUs despite access configured via Slurm (cgroups)

[mil-ad]

We want to use nvidia-docker inside our slurm jobs. Inside the slurm job only the GPUs we have requested are visible when we look at nvidia-smi outputs. I was expecting if I do docker run --gpus all nvidia-smi I would also see the same number GPUs. However within the container nvidia-smi shows all gpus available on that machine. The container seems to have the correct CUDA_VISIBLE_DEVICES environment variable but not the correct permissions.

Could it be related to the fact that by default docker needs sudo and we've created a user group so that we don't need sudo?

(BTW I noticed some things are different in Fedora but we're using Ubuntu.)

[flx42]

By default, docker containers are part of the Docker daemon's cgroups (since they are spawned by the daemon that is not a child of your job allocation). So the device cgroups of your job is not being applied.

[trn84]

Is there anything planned on this?

[flx42]

@trn84 not sure what could be done, that's the design of Docker. You can try to play around with --cgroup-parent, or try to spawn an unprivileged docker daemon in the job context, but that's about it.

You can look at this page for an overview of options:
https://slurm.schedmd.com/containers.html

[trn84]

yes, I was thinking about the cgroup-parent coming from the slurm context. There is a tool called socker that was going in that direction but did not work for me.

I am using slurm+enroot+pyxis which seems to work but is kinda a hassle with the additional container infrastructure...

[elgalu]

@trn84 what was your experience these months with slurm+enroot+pyxis so far?

[rkindi]

Not sure if it's relevant since I am a slurm and docker noob, but just wanted to share what worked for my use case. I'm not using nvidia-docker, but am using docker for some gpu work and was able to use the following:

docker run -it --gpus "device=$(scontrol show job -d $SLURM_JOB_ID | grep -oP '(?<=GRES=gpu:1\(IDX:)[0-9]+(?=\))')" my_image:latest

The command gets the physical id of the gpu (I only request 1 gpu for my use case) using scontrol show job -d and passes it into the docker run command. Of course, this command can be made more generic if you are using multiple gpus.

For more context on scontrol show job -d, please see https://stackoverflow.com/a/68232406/21405443

[adamnovak]

Our Slurm cluster seems to set a SLURM_STEP_GPUS environment variable, but doesn't have scontrol available on the nodes. The environment variable gives a list of GPU numbers, without device IDs. So the workaround that seems to work for me is to not use --gpus, but to use the NVIDIA_VISIBLE_DEVICES magic container environment variable, which also takes a list of GPU numbers. So:

docker run --rm --runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=$SLURM_STEP_GPUS nvidia/cuda:12.1.1-base-ubuntu22.04 bash -c "nvidia-smi"

seems to work for me.

[elezar]

The --gpus flag does allow for specific devices to be specified, but requires a different syntax. Specifying an integer after the --gpus flag specifies the number (i.e. COUNT) of GPUs to make available and not specific devices.

The NVIDIA_VISIBLE_DEVICES envrionment variable accepts a comma-speparated list of device indices, uuids, or PCI busIDs.

[adamnovak]

I tried --gpus=\"device=$SLURM_STEP_GPUS\" as suggested at https://docs.docker.com/config/containers/resource_constraints/#expose-gpus-for-use and that also seemed to work.

It would be great if Slurm and Docker and the nvidia runtime developed some way to pass this information around behind the scenes, or at least made it easy and well-documented for users to make the necessary connection. Maybe a "${SLURM_DOCKER_FLAGS[@]}" could be provided by Slurm, ready to be added to a docker run command?

[klueska]

Closing for now as the originally posted issue seems to be resolved. @adamnovak feel free to open a separate issue for the feature request in your last comment.

[loopback-kr]

After creating the nvidia.yaml file with nvidia-ctk, try running podman with the --device=nvidia.com/gpu=all option.

Here is an ansible playbook task:

- name: Create hook script
  copy:
    content: |
      #!/bin/bash
      if [ -e /usr/bin/nvidia-ctk ]; then
        if [ -e /var/run/cdi/nvidia.yaml ]; then
          sleep $(( (RANDOM % 6) + 2 ))
          nvidia-ctk cdi generate --output=/var/run/cdi/nvidia.yaml > /dev/null 2>&1
        else
          nvidia-ctk cdi generate --output=/var/run/cdi/nvidia.yaml > /dev/null 2>&1
        fi
      fi
      exit 0
    dest: /etc/containers/pre-exec-hooks/000_nvidia_cdi.sh
    mode: '0555'

Here is the desired output

hyunseoki@Master:~/projects$ srun -p V100 --gpus 4 --pty -w gpu100 nvidia-smi -L
GPU 0: Tesla V100-PCIE-32GB (UUID: GPU-8767d3bc-143f-825c-8181-ae0def9ddf35)
GPU 1: Tesla V100-PCIE-32GB (UUID: GPU-a17836ca-ea76-b9cc-2f9c-6550a1c07111)
GPU 2: Tesla V100-PCIE-32GB (UUID: GPU-328ae301-380b-18f3-eda5-f114e782f650)
GPU 3: Tesla V100-PCIE-32GB (UUID: GPU-f532d676-1c20-7437-8878-c25d330b3b52)
hyunseoki@Master:~/projects$ srun -p V100 --gpus 1 --pty -w gpu100 nvidia-smi -L
GPU 0: Tesla V100-PCIE-32GB (UUID: GPU-8767d3bc-143f-825c-8181-ae0def9ddf35)
hyunseoki@Master:~/projects$ srun -p V100 --gpus 1 --pty -w gpu100 docker run -it --rm --device=nvidia.com/gpu=all ubuntu nvidia-smi -L
GPU 0: Tesla V100-PCIE-32GB (UUID: GPU-8767d3bc-143f-825c-8181-ae0def9ddf35)