docker service create doesn't allow --privileged flag

[ghost]

Output of docker version:

Client:
 Version:      1.12.0-rc4
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   e4a0dbc
 Built:        Wed Jul 13 03:39:43 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.0-rc4
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   e4a0dbc
 Built:        Wed Jul 13 03:39:43 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 54
Server Version: 1.12.0-rc4
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 71
 Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge null host overlay
Swarm: active
 NodeID: 33ops9juo9ea1twbfq2dyt89y
 IsManager: Yes
 Managers: 2
 Nodes: 5
 CACertHash: sha256:cef0da32ea05dd1038a5b8ae1a3a6956b6a5efa2d2fcad535a696dd568220197
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 3.13.0-86-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 94.42 GiB
Name: irvm-ggallag
ID: WA3H:N54J:H7F3:CQV6:74ZX:IWIZ:U6XG:2VCB:45LP:LDD5:FHB6:7CWZ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
 127.0.0.0/8

Additional environment details (AWS, VirtualBox, physical, etc.):
Ubuntu 14.04 VM under KVM running Docker enginer 1.12 RC4

Steps to reproduce the issue:

1. docker service create
2. inside docker image NFS mount

Describe the results you received:
I can run "docker run --privileged" to allow an NFS mount from within my container, however there is no way to pass this --privileged flag to "docker service" and if I do not pass the --privileged flag, the contain would error internally on the mount like:

mount: permission denied

Describe the results you expected:
I should be able to have my container mount an NFS server from within it. I do not want to do this externally or via a docker volume, for example, I am trying to drive a huge number of parallel containers running NFS mounts and I/O individually.

Additional information you deem important (e.g. issue happens only occasionally):

[justincormack]

I think there is a whole set of issues for these features on service create, we should probably make an issue listing them all.

I think the plan was to discuss what should be added, once 1.12 is released.

[thaJeztah]

Correct, I was planning to create a tracking issue for that

[gerred]

I could really use this for 1.12 as well. If this is an area I could jump in and issue a PR, I'm happy to get started on it.

[thaJeztah]

We need to decide first; services are not "containers", so not all options can be / should be copied to service create

[gerred]

@thaJeztah Another consideration - I have different needs between a replicated service and a global one. If a jobs service type is introduced, which has been discussed, those needs might be different too.

The global one I may expect to have more flags/options around, just given the nature of "other things" I might be doing with them (monitoring, networking, running containers, etc.). I suppose I could have a global service that mounts the docker socket that then runs a privileged container on each node, but that seems messy (now my the tasks in my global service are managing the lifecycle of a container on each engine separately).

Hopefully that helps with some of that discussion.

[ghost]

If services != containers, why do you pass an image name to the create command? Seems like you would rather pass something like a manifest (maybe exactly like the docker-compose.yml file ?).

For this issue, if it is a PR, I'll phrase it in user story format: As a user of swarm, I want to create services and containers which run under privileged mode. How do I do this?

I'm happy to help any way that I can!

[gerred]

@frellus You're right - stacks/DABs are really what I need but they're also really early and don't have a service type option associated with them yet. There's also some other little nits there I need to write a more specific issue around in compose. Ultimately it's all still a bit of a chicken and egg problem - in one I get privileged, in the other I get service types. :) It'll all shake out, for now just reporting my uses to help provide as much data as I can! 😄

[padyx]

I am very interested in this, because as far as I know the Oracle DB cannot run in a container without either the --privileged option or specifying the --shm flag when running a container.

Given that they are both not supported yet in API 1.24 for services AFAICS, it would be impossible to replace a Docker Swarm (standalone) by Docker 1.12 Swarm to run such services.

Edit: Oracle just published Docker files https://github.com/oracle/docker-images/tree/master/OracleDatabase, so this big hurdle is resolved for us.

[bboreham]

Drive-by observation: you should do --cap-add before --privileged, to encourage people to be more granular in what they need. #25885 relates.

[b0ch3nski]

I'd really like to see it implemented soon - there are more solutions that require --privileged flag to function properly, e.g. cAdvisor which I'm using for containers performance monitoring.

[acaranta]

If I may add ... --privileged and/or --device* are quite critical for the case you need to run containers using GPU/CUDA calculations ....
Placement rules to allow these kind of containers to run on specific hosts can be used ... but not being able to actually use the GPU ..... kind of renders the swarmmode useless for us ... :(

[AkihiroSuda]

Just FYI, linking the PR for supporting "device" to this issue:
moby/swarmkit#1244
moby/swarmkit#1355

Even though there would be --device, I think --privileged is still attractive (e.g. for DinD)

[frimdo]

+1

--cap-add and --cap-drop Is a must. --privileged would be nice. Are there any plans implementing it?

[fjammes]

Missing --cap-add to use swarm mode in production, my management push me to move towards kubernetes if this option is not added soon. Do you have some plan and agenda for adding this feature please?

[dalefwillis]

+1

--cap-add would be a huge help!

[seiferteric]

Working on kind of a workaround. It will run your privileged app in a secondary container by mounting /var/run/docker.sock in your service and proxying tcp connections back to the service container with socat and unix sockets. Still needs some work though.

[calh]

I'd also like to see the --cap-add on docker services. I've written a workaround, similar to @seiferteric's if anyone would like to try it out: https://github.com/calh/docker_priv_proxy

I'm using signal traps, socat, and the docker socket to pair swarm mode service containers with a local privileged mode container. It seems to work well so far!

[seiferteric]

Just another use case, I want to run keepalived with vrrp on a swarm and it needs net=host and --cap-add=NET_ADMIN, so cap-add would be great.

[aderbique]

Pleeeeaaaaase add this feature in :(

[olljanat]

Pleeeeaaaaase don't send useless messages here. It just slowdown the process as lot of people get notified and that time is away from actual implementation.

My plan is to get #38380 released as part of 19.03 and then actual Swarm side changes on version which comes after that (19.06 I guess). Anyone who want to help plz test that and tell comments on PR so then hopefully we can stay on that schedule.

[ntimo]

This issue is open since 2016 is there a chance that this gets added to Docker in the next years.... ? This is a must have feature, for running Docker in Docker on a swarm. Please Docker team.

Sorry but this is very frustrating for me. I can't switch to a HA setup because of this :(

[AkihiroSuda]

@ntimo #24862 (comment)

[olljanat]

#39173 was merged so this feature will ship as part of Docker 19.06 / 19.09 (not sure actual version yet).

Please, look my question about CLI side implementation on #25885 (comment)
and here you can also comment use cases which actually would need --privileged switch or can all those be handled by defining all needed capabilities? (nothing prevents user to list all of them)

[olljanat]

FYI #25885 (comment)

[SySfRaMe]

Hi
I to seek for the possiblity to attach a device such as the /dev/ttyACM0 to a stack deployed service.
I created a script to run based on crontab every 5 min. (on the host of every node)
If the script see´s my special device attached. Then it will raise a node flag.
This flag is then used as a constraint to what node the service is allowed:

If lets say node1 failes then i can move the usb-dongle to node2 and the service is started there.

If you are interested in the cron script then look here:
https://github.com/SySfRaMe/Docker_ZwaveHW_Flag

[kaysond]

I am trying to run ip netns exec <namespace> iptables -L from within a swarm container, and it seems that without --privileged, I'm unable to do so. I'm guessing its because /proc, /sys, cgroup etc are mounted as RO. Does anyone know of a workaround for this?

If not, it seems my container must have --privileged in swarm.

[olljanat]

Note on Docker 20.10.x you can use --cap-add parameter with swarm services. --privileged switch still is not available (and will no be added because it makes too easy to create unsecure configuration) but nothing prevents you to list all capabilities you need.

To allow iptables comnand inside of container you need to use --cap-add NET_ADMIN

@thaJeztah I think that we can actually close this issue?

[kaysond]

@olljanat the problem in my case is different. i already have --cap-add NET_ADMIN, which allows me to run iptables. the problem is ip netns exec runs mount --make-rslave /, which fails due to the read-only /proc /sys, etc mounts. so i can only run iptables on the default namespace

[olljanat]

@kaysond on that case plz create new issue with all asked details.

[beornf]

@kaysond I also want to run ip netns exec scoped commands in a swarm container. A more direct way to make the mounts writable would be to add the parameter --security-opt apparmor=unconfined. There's an open issue #41371 for adding the --security-opt option for services.

[kaysond]

@beornf a workaround that I was able to use is nsenter --net=/var/run/netns/<namespace> <command>. I used host networking and set up a bind mount: /var/run/docker/netns:/var/run/netns, then used NET_ADMIN and SYS_ADMIN cap_add's on the service.

It similarly runs a process in the specified namespace, it just doesn't do all of the remounting that ip netns exec does. Luckily, it doesn't seem that many of the stock linux networking utilities actually need that remounting.

[dimisjim]

@beornf a workaround that I was able to use is nsenter --net=/var/run/netns/<namespace> <command>. I used host networking and set up a bind mount: /var/run/docker/netns:/var/run/netns, then used NET_ADMIN and SYS_ADMIN cap_add's on the service.

It similarly runs a process in the specified namespace, it just doesn't do all of the remounting that ip netns exec does. Luckily, it doesn't seem that many of the stock linux networking utilities actually need that remounting.

Could you provide a sample compose yaml demonstrating this?

[kaysond]

@dimisjim

Could you provide a sample compose yaml demonstrating this?

It's just

volumes:
  - /var/run/docker/netns:/var/run/netns