Add support for devices with "service create"

[flx42]

Initially reported: moby/moby#24865, but I realized it actually belongs here. Feel free to close the other one if you want. Content of the original issue copied below.

Related: #1030

Currently, it's not possible to add devices with docker service create, there is no equivalent for docker run --device=/dev/foo.

I'm an author of nvidia-docker with @3XX0 and we need to add devices files (the GPUs) and volumes to the starting containers in order to enable GPU apps as services.
See the discussion here: moby/moby#23917 (comment) (summarized below).

We figured out how to add a volume provided by a volume plugin:

$ docker service create --mount type=volume,source=nvidia_driver_367.35,target=/usr/local/nvidia,volume-driver=nvidia-docker [...]

But there is no solution for devices, @cpuguy83 and @justincormack suggested using --mount type=bind. But it doesn't seem to work, it's probably like doing a mknod but without the proper device cgroup whitelisting.

$ docker service create --mount type=bind,source=/dev/nvidiactl,target=/dev/nvidiactl ubuntu:14.04 sh -c 'echo foo > /dev/nvidiactl'
$ docker logs stupefied_kilby.1.2445ld28x6ooo0rjns26ezsfg
sh: 1: cannot create /dev/nvidiactl: Operation not permitted

It's probably equivalent to this:

$ docker run -ti ubuntu:14.04                      
root@76d4bb08b07c:/# mknod -m 666 /dev/nvidiactl c 195 255
root@76d4bb08b07c:/# echo foo > /dev/nvidiactl
bash: /dev/nvidiactl: Operation not permitted

Whereas the following works (invalid arg is normal, but no permission error):

$ docker run -ti --device /dev/nvidiactl ubuntu:14.04
root@ea53a1b96226:/# echo foo > /dev/nvidiactl
bash: echo: write error: Invalid argument

[stevvooe]

@flx42 For the container runtime, devices require special handling (a mknod syscall), so mounts won't work. We'll probably have to add some sort of support for this. (cc @crosbymichael)

Ideally, we'd like to be able to schedule over devices, as well.

[cpuguy83]

@stevvooe Already have device support in the runtime, just not exposed in swarm.

[flx42]

Ideally, we'd like to be able to schedule over devices, as well.

This question was raised here: moby/moby#24750
But the discussion was redirected here: moby/moby#23917, in order to have a single discussion thread.

[flx42]

@stevvooe I quickly hacked a solution, it's not too difficult:
flx42@a82b9fb
This is not a PR yet, would you be interested if I do one? Or are the swarmkit features frozen right now before 1.12?
The next step would be to also modify the engine API.

[flx42]

Forgot to mention that I can now run GPU containers by mimicking what nvidia-docker does:

./bin/swarmctl service create --device /dev/nvidia-uvm --device /dev/nvidiactl --device /dev/nvidia0 --bind /var/lib/nvidia-docker/volumes/nvidia_driver/367.35:/usr/local/nvidia --image nvidia/digits:4.0 --name digits

[stevvooe]

@flx42 I took a quick peak and the PR looks like a decent start. I am not sure about representing these as cluster-level resources for container startup. From an orchestration perspective, we have to match these up with announced resources at the node level, which might be okay. It might be better on ContainerSpec, but I'm not sure yet.

Go ahead and file as a [WIP] PR.

[flx42]

@stevvooe Yeah, that's the biggest discussion point for sure.

In engine-api, devices are resources:
https://github.com/docker/engine-api/blob/master/types/container/host_config.go#L249

But in swarmkit, resources are so far "fungible" objects like CPU shares and memory, with a base value and a limit. A device doesn't really fit that definition. For GPU apps we have devices that must be shared (/dev/nvidiactl) and devices that could be exclusively acquired (like /dev/nvidia0).

I decided to initially put devices into resources because there is already a function in swarmkit that creates a engine-api Resource object from a swarm Resource object:
https://github.com/docker/swarmkit/blob/master/agent/exec/container/container.go#L301-L324
This method would also need to access the container spec.

I will file a PR soon to continue the discussion.

[stevvooe]

@flx42 Great!

We really aren't planning on following the same resource model from HostConfig for SwarmKit. In this case, we are instructing the container to mount these devices, which is specific to a container runtime. Other runtimes may not have a container or devices. Thus, I would err on ContainerSpec.

Now, I would like to see scheduling of fungible GPUs but that might a wholly separate flow, keeping the initial support narrow. Such services would require manual constraint and device assignment, but you still achieve the goal.

Let's discuss this in the context of the PR.

[aluzzardi]

Thanks @flx42 - I think GPU is definitly something we want to support medium term.

/cc @mgoelzer

[flx42]

Thanks @aluzzardi, PR created, it's quite basic.

[mlhales]

The --device option is really import for my use case too. I am trying to use swarm to manage 50 Raspberry Pi's to do computer vision, but I need to be able to access /dev/video0 to capture images. Without this option, I'm stuck, and have to manage them without swarm, which is painful.

[stevvooe]

@mlhales We need someone who is willing to workout the issues with --device in a clustered environment and support that solution, rather than just a drive by PR. If you or a colleague want to take this on, that would be great, but this isn't as simple as adding --device.

[StefanScherer]

Using --device=/dev/gpiomem would be great on a RPi swarm to access GPIO on each node without privileged mode.

[nazar-pc]

Using --device=/dev/fuse would be great for mounting FUSE, which isn't currently possible.

[StefanScherer]

We found an easier way for Blinkt! LED strip to use sysfs. Now we can run Blinkt! in docker swarm mode without privileges.

[mathiasimmer]

@StefanScherer is it a proper alternative for using e.g. --device=/dev/mem to access GPIO on a RPi ? Would love to see an example if you would care to share :)

[StefanScherer]

@mathiasimmer For the use-case with Blinkt! LED strip there are only eight RGB LED's. So using sysfs it not time critical for these few LED's. If you want to drive hundreds of them you still need faster GPIO access to have a higher clock rate. But for Blinkt! we have forked the Node.js module and adjusted in in this branch https://github.com/sealsystems/node-blinkt/tree/sysfs.
A sample application can be found as well and how to use this forked module as dependency in an own package.json.

[aluzzardi]

/cc @cyli

[stevvooe]

@aluzzardi I think we should resurrect the --device patch. I don't think there is anything in the pipeline that is sophisticated enough to handle proper, cluster-level dynamic resource allocation. Looking back at this issue, there isn't necessarily a model that will work well in all cases (mostly because no one here can seem to enumerate them).

We can always add logic in the scheduler to prevent device contention in the future.

[cyli]

Attempt to add devices to the container spec and plugin spec here: #1964

I've no objection to the --device flag - cc @diogomonica ?

[diogomonica]

--device allows any service to escalate privileges. Why would we add this w/out profiles on services?

[cyli]

@diogomonica I thought profiles mainly covered capabilities, etc?

[diogomonica]

@cyli well, if we believe "devices" are easy enough to understand for easy user acceptance then we might not need them, but we should look critically at adding anything that allows escalation of privileges of a container to the cmd-line before we have agood way of informing everything the service will need from a security perspective to the user.

[brubbel]

Also following this. Very interested in access to character devices (/dev/bus/usb/...) in a docker swarm.
To help some others until this is supported by docker, a workaround for swarm + usb:

1. On the (linux) host(s), create a udev rule which creates a symlink to your device (in my case an ftdi device). e.g. /etc/udev/rules.d/99-libftdi.rules
   SUBSYSTEMS=="usb", ATTRS{idVendor}=="xxxx", ATTRS{idProduct}=="xxxx", GROUP="dialout", MODE="0666", SYMLINK+="my_ftdi", RUN+="/usr/bin/setupdockerusb.sh"
   Then reload udev rules:
   sudo udevadm control --reload-rules
   Upon connect of the usb device, the udev manager will create a symlink /dev/my_ftdi -> /dev/bus/usb/xxx/xxx and execute /usr/bin/setupdockerusb.sh

2. The /usr/bin/setupdockerusb.sh (ref)
   This script sets the character device permissions on (the first) container with given image name.

#!/bin/bash
USBDEV=`readlink -f /dev/my_ftdi`
read minor major < <(stat -c '%T %t' $USBDEV)
if [[ -z $minor || -z $major ]]; then
    echo 'Device not found'
    exit
fi
dminor=$((0x${minor}))
dmajor=$((0x${major}))
CID=`docker ps --no-trunc -q --filter ancestor=my/imagename|head -1`
if [[ -z $CID ]]; then
    echo 'CID not found'
    exit
fi
echo 'Setting permissions'
echo "c $dmajor:$dminor rwm" > /sys/fs/cgroup/devices/docker/$CID/devices.allow

3. Create the docker swarm with following options:
   docker service create [...] --mount type=bind,source=/dev/bus/usb,target=/dev/bus/usb [...]

4. Event listener (systemd service):
   Waits for a container to be started and sets permissions. Run with root permissions on host.

#!/bin/bash
docker events --filter 'event=start'| \
while read line; do
    /usr/bin/setupdockerusb.sh
done

[allfro]

We NEED device mapping for swarms. I'd hate to switch over to Kubernetes for something as trivial as mapping common devices such as /dev/tun across a cluster. We beg you Docker!

[cpuguy83]

Maybe stop begging someone else to write features you need?
That is why there is exactly one person working on this repo... in their spare time.

[allfro]

@cpuguy83 isn't swarmkit developed by Docker corp which is also commercially sold as part of Docker EE?

[cpuguy83]

@allfro No. Docker sold off the EE stuff to Mirantis... but even before then Swarmkit had very little support.

[mdegans]

The docs say to use:

deploy:
  resources:
    reservations:
      devices:
        - driver: nvidia
          count: 1
          capabilities: [gpu]

But unsurprisingly this is broken in swarm.

Time to move to k8s.

[mdegans]

Maybe stop begging someone else to write features you need? That is why there is exactly one person working on this repo... in their spare time.

Maybe they can maintain their stuff? How long has GPU support been broken in swarm?

[prologic]

I know this is a 6 year old issue, but is there actually an open PR for this that just needs a bit of attention? Maybe I could help finish the code required to support this? 🤔

[Stefan592]

Solution does not work for Debian11 Bullseye.
Is there a new workaround for this?

#1244 (comment)

[MohammedNoureldin]

Hey, @allfro
Have you found a solution? I need the exact same usage like you (for tun device). Did you switch to another solution or have you figured out a workaround?

[radeksh]

How can I help to finish that feature?

[pjalusic]

I really like workaround from @BretFisher #1244 (comment) and here is how I adapted it for nodes that require a device:

• connect that special container to stack network to make it behave like part of the stack (shutting down, redeploying). Otherwise it will keep running forever and require manual removal
• extract that service to separate docker-compose.yml and save it on each node that requires a device. Run command will be less messy

Putting it all together, your services will have to change from this:

services:
  my-service-starter:
    image: docker
    command: 'docker run --name <name> --device /dev/bus/usb -e TOKEN=1234 -p 5000:5000 <image>'
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    deploy:
      placement:
        constraints:
          - node.labels.device_required == true

to this:

services:
  my-service-handler:
    image: docker
    command: 'docker-compose -f /docker-compose.yml up'
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /home/ubuntu/docker-compose.yml:/docker-compose.yml
    deploy:
      placement:
        constraints:
          - node.labels.device_required == true

networks:
  default:
    name: my_network
    driver: overlay
    attachable: true

(on manager)
and

services:
  <name>:
    image: <image>
    restart: always
    container_name: <name>
    devices:
      - /dev/bus/usb
    environment:
      - TOKEN=1234
    ports:
      - 5000:5000

networks:
  default:
    name: my_network
    external: true

(/home/ubuntu/docker-compose.yml on nodes that require a device)

[bighb69738]

Hi @pjalusic ,

services:
  <name>:
    image: <image>
    restart: always
    container_name: <name>
    devices:
      - /dev/bus/usb
    environment:
      - TOKEN=1234
    ports:
      - 5000:5000

networks:
  default:
    name: my_network
    external: true

But in the worker node, I need to depend on another servcie from manager node.
Could you give me a example for the docker-compose.yml of worker node to add "depends_on" tag.

[allfro]

I developed a plugin in the end that allows me to map devices to containers: https://github.com/allfro/device-volume-driver. Hope it helps others. Unfortunately, it only works on systems that use cgroup v1 (alpine). I am looking for some help to develop the cgroupv2 support into the plugin. It works really well and I've used it to containerize x11 desktops that require access to fuse and the vmware graphics devices.

cc: @MohammedNoureldin

[zikaeroh]

After planning to redo my home server setup with swarm (so I can have multiple nodes), I discovered that this wasn't supported, and I needed it for VAAPI.

After looking through things, it seemed to me like this was a plumbing (and developer-hour) problem. Basing things on a previous PR series which added ulimit support to swarm, here is a chain of PRs which add devices in the most boring way; just plumbing it through the API as-is, no special management or API. Just what docker already supported outside of swarm.

• Add support for devices #3106
• Add support for devices in swarm moby#44695
• Add support for devices in swarm docker/cli#3930

I'm sure I've missed something, and I don't quite know how to get everything building together to test this (I typically run things from my package manager's installed docker), but maybe someone is willing to try the above out.

[vadd98]

Hi, I'm trying the workaround in #1244 (comment) and it indeed works, but when I remove the stack the handler is successfully removed, while the privileged container in docker-compose.yml continue running and has to be killed manually using docker kill.

Any idea on what could be the issue?

[coltonbh]

The docs say to use:

deploy:
  resources:
    reservations:
      devices:
        - driver: nvidia
          count: 1
          capabilities: [gpu]

But unsurprisingly this is broken in swarm.

Time to move to k8s.

These docs are referencing the API for docker compose, not swarm (services or stacks). The correct (and functioning) API for a stack is:

services:
  my-gpu-service:
    ...
    deploy:
      resources:
        reservations:
          generic_resources:
            - discrete_resource_spec:
              kind: "NVIDIA-GPU"
              value: 2

This works if you've registered your GPUs in the /etc/docker/daemon.json file.

For anyone looking for device support for NVIDIA GPUs using Swarm I did a quick write up here summarizing two solutions. My write up was heavily inspired by the original gist I found on the subject here.

[zeppmg]

Hello,
Thanks for the tip. Anyway in my case it doesn't work. After having investigated step by step, I've realized that I don't have a /sys/fs/cgroup/devices folder on any of my swarm nodes. Does anyone have an idea of where this can come from ?

sudo ls /sys/fs/cgroup
cgroup.controllers      cgroup.stat             cpuset.cpus.effective  dev-mqueue.mount  io.pressure       memory.stat                    sys-kernel-debug.mount
cgroup.max.depth        cgroup.subtree_control  cpuset.mems.effective  init.scope        io.stat           -.mount                        sys-kernel-tracing.mount
cgroup.max.descendants  cgroup.threads          cpu.stat               io.cost.model     memory.numa_stat  sys-fs-fuse-connections.mount  system.slice
cgroup.procs            cpu.pressure            dev-hugepages.mount    io.cost.qos       memory.pressure   sys-kernel-config.mount        user.slice

[reisholmes]

Hello, Thanks for the tip. Anyway in my case it doesn't work. After having investigated step by step, I've realized that I don't have a /sys/fs/cgroup/devices folder on any of my swarm nodes. Does anyone have an idea of where this can come from ?

sudo ls /sys/fs/cgroup
cgroup.controllers      cgroup.stat             cpuset.cpus.effective  dev-mqueue.mount  io.pressure       memory.stat                    sys-kernel-debug.mount
cgroup.max.depth        cgroup.subtree_control  cpuset.mems.effective  init.scope        io.stat           -.mount                        sys-kernel-tracing.mount
cgroup.max.descendants  cgroup.threads          cpu.stat               io.cost.model     memory.numa_stat  sys-fs-fuse-connections.mount  system.slice
cgroup.procs            cpu.pressure            dev-hugepages.mount    io.cost.qos       memory.pressure   sys-kernel-config.mount        user.slice

Also in this same situation. I was using this solution to passthrough the iGPU driver to PLEX on a dockerswarm host for hardware transcoding: https://pastebin.com/XY7GP18T

I had some new hardware which required running the latest version of ubuntu to recognise it but this uses the cgroups v2. At the moment I reverted back to using cgroups v1 via these instructions to get this working again: https://sleeplessbeastie.eu/2021/09/10/how-to-enable-control-group-v2/
Key bit:
$ sudo sed -i -e 's/^GRUB_CMDLINE_LINUX=""/GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=1"/' /etc/default/grub $ sudo update-grub $ sudo reboot

I will experiment with moving to cgroups v2 and a combination of generic resource advertising the iGPU to the service as soon as I have time via these two hints as outlined by @coltonbh :

https://gist.github.com/coltonbh/374c415517dbeb4a6aa92f462b9eb287
https://docs.docker.com/compose/gpu-support/#enabling-gpu-access-to-service-containers

If anyone has any idea how to correctly advertise a quicksync driver to a cgroup v2 using dockerswarm it would be highly appreciated. Alternatively, I guess I could migrate to kubernetes ;)

[jvrobert]

I'm getting strong "the perfect is the enemy of the good" vibes from this issue. Strongly in favor of just passing through the devices options and letting buyer beware.

[allfro]

I've written this hack and tried it with plex and it seems to work: https://github.com/allfro/device-mapping-manager. Essentially it runs a privileged container which listens for docker create events and inspects the mount points. If a mount is within the /dev folder, it will walk the mount path for character and block devices and apply the necessary device rules to make the devices available. This doesn't work with fuse yet because the default apparmor profile blocks mounts (ugh!) but it does work with graphics cards and other devices that don't require operations that are blocked by Docker's apparmor profile. It is inspired by the previous comments.

[allfro]

If anyone has any idea how to correctly advertise a quicksync driver to a cgroup v2 using dockerswarm it would be highly appreciated. Alternatively, I guess I could migrate to kubernetes ;)

@reisholmes check this out: https://github.com/allfro/device-mapping-manager