Mounting FUSE should be possible with Docker Swarm Mode

[nazar-pc]

Description
I'd like to mount filesystem using FUSE inside container when it starts, but this doesn't seem to be possible in Docker Swarm Mode.
In particular, I want to mount ceph-fuse.
Ideally, following should work (it actually works with version: '2' after scaling Ceph's nodes to 3+ instances):

# docker-compose.yml
version: '3.1'
services:
  consul:
    image: nazarpc/webserver:consul
    restart: always

  ceph-mon:
    image: nazarpc/webserver:ceph
    command: mon
    restart: always

  ceph-osd:
    image: nazarpc/webserver:ceph
    command: osd
    restart: always

  ceph-mds:
    image: nazarpc/webserver:ceph
    command: mds
    restart: always

  nginx:
    image: nazarpc/webserver:nginx
    restart: always
    environment:
      CEPHFS_MOUNT: 1
# In order to access FUSE
    devices:
      - /dev/fuse:/dev/fuse
# Because https://github.com/docker/docker/issues/16429
    security_opt:
      - "apparmor:unconfined"
# In order to mount CEPHFS
    cap_add:
      - SYS_ADMIN

However, both devices and cap_add are not allowed with v3 of docker-compose.yml file format, which in turn seems to be limited by Docker Engine itself.

Steps to reproduce the issue:

1. Mount ceph-fuse in container running with Docker Swarm Mode

Describe the results you received:
Mounting fails

Describe the results you expected:
Mounting should be possible somehow

Output of docker version:

Client:
 Version:      1.13.1
 API version:  1.26
 Go version:   go1.7.5
 Git commit:   092cba3
 Built:        Wed Feb  8 06:57:21 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.13.1
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   092cba3
 Built:        Wed Feb  8 06:57:21 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 28
Server Version: 1.13.1
Storage Driver: btrfs
 Build Version: Btrfs v4.7.3
 Library Version: 101
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1
runc version: 9df8b306d01f59d3a8029be411de015b7304dd8f
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.10.0-8-generic
Operating System: Ubuntu Zesty Zapus (development branch)
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 15.57 GiB
Name: nazar-pc
ID: 3RNU:FHWJ:O265:QXAO:Y276:CZXJ:Z3XR:LP6O:POQW:MRRC:PK44:BP2K
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

[nazar-pc]

Just to clarify a little bit on what I'm trying to achieve: the goal is to have distributed storage implemented inside of containers that doesn't need privileged access to Docker Engine and doesn't need basically any manual configuration in simplest cases besides generic declarative docker-compose.yml.

I've seen Infinit acquisition, but it is not integrated deeply and as easy to use as overlay for networking yet.

[justincormack]

It might be easier to use a volume plugin, eg see the example sshfs fused based volume plugin here https://github.com/vieux/docker-volume-sshfs as an example, writing a ceph plugin should be fairly simple.

Adding device support is on the roadmap see the functionality parity issue for details #25303

[nazar-pc]

It is not easier to use volume plugin, since Ceph needs to be configured upfront manually and is not a part of services declaration in contrast to simple docker-compose up -d.
Writing and maintaining custom plugin is not easy too, especially for someone who doesn't know the Go language.

Do you think this issue should be closed in favor of generic --devices and --cap_add service options support you've linked?

[justincormack]

@nazar-pc I think you will find that writing a volume plugin is much easier. They do not need to be written in Go, they just communicate over a socket. This is exactly the use case they are designed for. You just need to do docker plugin install to install them and then your compose yaml file can just specify that you want to use the ceph volume driver.

If you use the method you suggest you are granting cap_sys_admin to all services, which is almost the same as --privileged.

[thaJeztah]

Yes, let's close this in favor of #25885, and #25303, which are already tracking the requirements