New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker service create doesn't allow --privileged flag #24862

Open
frellus opened this Issue Jul 20, 2016 · 89 comments

Comments

Projects
None yet
@frellus

frellus commented Jul 20, 2016

Output of docker version:

Client:
 Version:      1.12.0-rc4
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   e4a0dbc
 Built:        Wed Jul 13 03:39:43 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.0-rc4
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   e4a0dbc
 Built:        Wed Jul 13 03:39:43 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 54
Server Version: 1.12.0-rc4
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 71
 Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge null host overlay
Swarm: active
 NodeID: 33ops9juo9ea1twbfq2dyt89y
 IsManager: Yes
 Managers: 2
 Nodes: 5
 CACertHash: sha256:cef0da32ea05dd1038a5b8ae1a3a6956b6a5efa2d2fcad535a696dd568220197
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 3.13.0-86-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 94.42 GiB
Name: irvm-ggallag
ID: WA3H:N54J:H7F3:CQV6:74ZX:IWIZ:U6XG:2VCB:45LP:LDD5:FHB6:7CWZ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
 127.0.0.0/8

Additional environment details (AWS, VirtualBox, physical, etc.):
Ubuntu 14.04 VM under KVM running Docker enginer 1.12 RC4

Steps to reproduce the issue:

  1. docker service create
  2. inside docker image NFS mount

Describe the results you received:
I can run "docker run --privileged" to allow an NFS mount from within my container, however there is no way to pass this --privileged flag to "docker service" and if I do not pass the --privileged flag, the contain would error internally on the mount like:

mount: permission denied

Describe the results you expected:
I should be able to have my container mount an NFS server from within it. I do not want to do this externally or via a docker volume, for example, I am trying to drive a huge number of parallel containers running NFS mounts and I/O individually.

Additional information you deem important (e.g. issue happens only occasionally):

@justincormack

This comment has been minimized.

Contributor

justincormack commented Jul 22, 2016

I think there is a whole set of issues for these features on service create, we should probably make an issue listing them all.

I think the plan was to discuss what should be added, once 1.12 is released.

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Jul 22, 2016

Correct, I was planning to create a tracking issue for that

@gerred

This comment has been minimized.

gerred commented Jul 22, 2016

I could really use this for 1.12 as well. If this is an area I could jump in and issue a PR, I'm happy to get started on it.

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Jul 22, 2016

We need to decide first; services are not "containers", so not all options can be / should be copied to service create

@gerred

This comment has been minimized.

gerred commented Jul 22, 2016

@thaJeztah Another consideration - I have different needs between a replicated service and a global one. If a jobs service type is introduced, which has been discussed, those needs might be different too.

The global one I may expect to have more flags/options around, just given the nature of "other things" I might be doing with them (monitoring, networking, running containers, etc.). I suppose I could have a global service that mounts the docker socket that then runs a privileged container on each node, but that seems messy (now my the tasks in my global service are managing the lifecycle of a container on each engine separately).

Hopefully that helps with some of that discussion.

@frellus

This comment has been minimized.

frellus commented Jul 22, 2016

If services != containers, why do you pass an image name to the create command? Seems like you would rather pass something like a manifest (maybe exactly like the docker-compose.yml file ?).

For this issue, if it is a PR, I'll phrase it in user story format: As a user of swarm, I want to create services and containers which run under privileged mode. How do I do this?

I'm happy to help any way that I can!

@gerred

This comment has been minimized.

gerred commented Jul 22, 2016

@frellus You're right - stacks/DABs are really what I need but they're also really early and don't have a service type option associated with them yet. There's also some other little nits there I need to write a more specific issue around in compose. Ultimately it's all still a bit of a chicken and egg problem - in one I get privileged, in the other I get service types. :) It'll all shake out, for now just reporting my uses to help provide as much data as I can! 😄

@padyx

This comment has been minimized.

Contributor

padyx commented Jul 29, 2016

I am very interested in this, because as far as I know the Oracle DB cannot run in a container without either the --privileged option or specifying the --shm flag when running a container.

Given that they are both not supported yet in API 1.24 for services AFAICS, it would be impossible to replace a Docker Swarm (standalone) by Docker 1.12 Swarm to run such services.

Edit: Oracle just published Docker files https://github.com/oracle/docker-images/tree/master/OracleDatabase, so this big hurdle is resolved for us.

@bboreham

This comment has been minimized.

Contributor

bboreham commented Sep 7, 2016

Drive-by observation: you should do --cap-add before --privileged, to encourage people to be more granular in what they need. #25885 relates.

@b0ch3nski

This comment has been minimized.

b0ch3nski commented Sep 8, 2016

I'd really like to see it implemented soon - there are more solutions that require --privileged flag to function properly, e.g. cAdvisor which I'm using for containers performance monitoring.

@acaranta

This comment has been minimized.

acaranta commented Sep 14, 2016

If I may add ... --privileged and/or --device* are quite critical for the case you need to run containers using GPU/CUDA calculations ....
Placement rules to allow these kind of containers to run on specific hosts can be used ... but not being able to actually use the GPU ..... kind of renders the swarmmode useless for us ... :(

@AkihiroSuda

This comment has been minimized.

Member

AkihiroSuda commented Sep 14, 2016

Just FYI, linking the PR for supporting "device" to this issue:
docker/swarmkit#1244
docker/swarmkit#1355

Even though there would be --device, I think --privileged is still attractive (e.g. for DinD)

@frimdo

This comment has been minimized.

frimdo commented Sep 14, 2016

+1

--cap-add and --cap-drop Is a must. --privileged would be nice. Are there any plans implementing it?

@fjammes

This comment has been minimized.

fjammes commented Oct 15, 2016

Missing --cap-add to use swarm mode in production, my management push me to move towards kubernetes if this option is not added soon. Do you have some plan and agenda for adding this feature please?

@dalefwillis

This comment has been minimized.

dalefwillis commented Oct 20, 2016

+1

--cap-add would be a huge help!

@seiferteric

This comment has been minimized.

seiferteric commented Nov 29, 2016

Working on kind of a workaround. It will run your privileged app in a secondary container by mounting /var/run/docker.sock in your service and proxying tcp connections back to the service container with socat and unix sockets. Still needs some work though.

AkihiroSuda referenced this issue in AkihiroSuda/docker Dec 1, 2016

new make target: test-integration-cli-parallel
Usage: $ make test-integration-cli-parallel

parallel(1) needs to be installed.

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
@calh

This comment has been minimized.

calh commented Dec 8, 2016

I'd also like to see the --cap-add on docker services. I've written a workaround, similar to @seiferteric's if anyone would like to try it out: https://github.com/calh/docker_priv_proxy

I'm using signal traps, socat, and the docker socket to pair swarm mode service containers with a local privileged mode container. It seems to work well so far!

@seiferteric

This comment has been minimized.

seiferteric commented Mar 8, 2017

Just another use case, I want to run keepalived with vrrp on a swarm and it needs net=host and --cap-add=NET_ADMIN, so cap-add would be great.

@ulm0

This comment has been minimized.

ulm0 commented Feb 5, 2018

I'm facing this issue as well when trying to deploy GitLab in Docker Swarm

The package needs to set values but it can't and the service just keep restarting

/opt/gitlab/embedded/bin/runsvdir-start: line 24: ulimit: pending signals: cannot modify limit: Operation not permitted
/opt/gitlab/embedded/bin/runsvdir-start: line 37: /proc/sys/fs/file-max: Read-only file system
@mwaeckerlin

This comment has been minimized.

mwaeckerlin commented Feb 8, 2018

@jonasddev, I updated my jenkins image to fix the permissions within the container, I added to my entry-point-script:

# add user to group that has access to /var/run/docker.sock
addgroup --gid $(stat -c '%g' /var/run/docker.sock) extdock || true
usermod -a -G $(stat -c '%g' /var/run/docker.sock) jenkins || true

Now the next problem:

  • docker in docker, access through shared /var/run/docker.sock
  • mounting volumes does not work

This is the problem in a mwaeckerlin/jenkins container:

ubuntu$ docker run -d --restart unless-stopped -v /var/run/docker.sock:/var/run/docker.sock --name jenkins -p 8080:8080/tcp -p 50000:50000/tcp --volumes-from jenkins-volumes mwaeckerlin/jenkins
ubuntu$ docker exec -it -u jenkins jenkins bash

jenkins$ docker create -v /var/lib/jenkins/workspace/mrw-c++.rpm/distro/fedora-27:/workdir -v /var/lib/jenkins/.gnupg:/var/lib/jenkins/.gnupg -e LANG=en_US.UTF-8 -e HOME=/var/lib/jenkins -e TERM=xterm -e DEBIAN_FRONTEND=noninteractive -e DEBCONF_NONINTERACTIVE_SEEN=true -e BUILD_NUMBER=64 -w /workdir fedora:27 sleep infinity
cd93d11c2634b4e4094cc2541996e30a4c08c60923b8896e0bd7e439c7d9c673
jenkins$ docker start cd93d11c2634b4e4094cc2541996e30a4c08c60923b8896e0bd7e439c7d9c673
cd93d11c2634b4e4094cc2541996e30a4c08c60923b8896e0bd7e439c7d9c673
jenkins$ ls /var/lib/jenkins/workspace/mrw-c++.rpm/distro/fedora-27
AUTHORS                      COPYING                   mrw-c++.spec.in
autogen.sh                   debian                    NEWS
ax_check_qt.m4               demangle.h                README
ax_cxx_compile_stdcxx_11.m4  dependency-graph.sh       resolve-debbuilddeps.sh
ax_init_standard_project.m4  doc                       resolve-rpmbuilddeps.sh
bootstrap.sh                 examples                  rpmsign.exp
build-in-docker.conf         INSTALL                   sql-to-dot.sed
build-in-docker.sh           mac-create-app-bundle.sh  src
build-resource-file.sh       makefile.am               suppressions.valgrind
ChangeLog                    makefile_test.inc.am      template.sh
checkinstall.sh              mrw-c++.desktop.in        test
configure.ac                 mrw-c++-minimal.spec.in   valcheck.sh
jenkins$ docker exec -u 107 -it 033088f9008601d5f9f9034744b579910ee1f88bdf3a61d2f0354b8454ba94ed bash

docker$ ls /workdir
docker$ mount | grep /var/lib/jenkins/workspace/mrw-c++.rpm/distro/fedora-27
/dev/mapper/big-root_crypt on /workdir type btrfs (rw,relatime,space_cache,subvolid=257,subvol=/@/var/lib/jenkins/workspace/mrw-c++.rpm/distro/fedora-27)
docker$

So in a jenkins-container (here named jenkins, a container (here named docker) is started. A directory from the jenkins comtainer is mounted int the container in the container, but there, it is empty. The mount is visible in the mount command.

Any idea?

@mwaeckerlin

This comment has been minimized.

mwaeckerlin commented Feb 8, 2018

BTW: #21109

@mwaeckerlin

This comment has been minimized.

mwaeckerlin commented Feb 8, 2018

The problem is describe here in a coment: http://container-solutions.com/running-docker-in-jenkins-in-docker/

When using docker run inside the jenkins container with volumes, you are actually sharing a folder of the host, not a folder within the jenkins container. To make that folder “visible” to jenkins (otherwise it is out of your control), that location should have a parent location that matches the volume that was used to run the jenkins image itself.

Of course, I want to mount from docker container to the container in the container and not from the host.

It is even a huge security hole: The docker container must not have access to the filesystem of the host where it is runnning!

Any simple solution for this?

@mwaeckerlin

This comment has been minimized.

mwaeckerlin commented Feb 8, 2018

Security Problem

If using -v /var/run/docker.sock:/var/run/docker.sock for docker in docker, you get full access to the host and you are no more jailed within the container!

Here a demonstration of the security issue mounting the socket:

marc@jupiter:~/docker/dockindock$ echo 'this is the host' > /tmp/test
marc@jupiter:~/docker/dockindock$ docker run -d --name dockindock -v /var/run/docker.sock:/var/run/docker.sock -v $(which docker):/usr/bin/docker mwaeckerlin/dockindock sleep infinity
d44fbd58e44a180e388621d39aff64652bc4118973c2cbc86a1738ffb481ebaf
marc@jupiter:~/docker/dockindock$ docker exec -it dockindock bash
root@docker[d44fbd58e44a]:/# cat /tmp/test
cat: /tmp/test: No such file or directory
root@docker[d44fbd58e44a]:/# echo 'this is the outer container' > /tmp/test
root@docker[d44fbd58e44a]:/# docker ps
CONTAINER ID        IMAGE                    COMMAND             CREATED             STATUS              PORTS               NAMES
d44fbd58e44a        mwaeckerlin/dockindock   "sleep infinity"    16 minutes ago      Up 16 minutes                           dockindock
root@docker[d44fbd58e44a]:/# docker run -it --rm -v /tmp:/tmp mwaeckerlin/ubuntu-base bash
root@docker[f197131c54c7]:/# cat /tmp/test
this is the host

So, the solution of using -v /var/run/docker.sock:/var/run/docker.sock is a security issue and an absolute no-go! @jonasddev

So either we get --privileged for swarm, or there is need for any other, better solution, that solves this issue!

Wanted Behaviour

Here a demonstration of how it should work, see the restricted access:

marc@jupiter:~/docker/dockindock$ echo 'this is the host' > /tmp/test
marc@jupiter:~/docker/dockindock$ docker run -d --name dockindock --privileged mwaeckerlin/dockindock
655c67b2a6d9f06da8bf630889710ee596f006331a036dc009c86a9e04ea0201
marc@jupiter:~/docker/dockindock$ docker exec -it dockindock bash
root@docker[655c67b2a6d9]:/# cat /tmp/test
cat: /tmp/test: No such file or directory
root@docker[655c67b2a6d9]:/# echo 'this is the outer container' > /tmp/test
root@docker[655c67b2a6d9]:/# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
root@docker[655c67b2a6d9]:/# docker run -it --rm -v /tmp:/tmp mwaeckerlin/ubuntu-base bash
Unable to find image 'mwaeckerlin/ubuntu-base:latest' locally
latest: Pulling from mwaeckerlin/ubuntu-base
1be7f2b886e8: Pull complete 
6fbc4a21b806: Pull complete 
c71a6f8e1378: Pull complete 
4be3072e5a37: Pull complete 
06c6d2f59700: Pull complete 
04fca7013ee9: Pull complete 
7a66494bf7fe: Pull complete 
be1530d02718: Pull complete 
57cb4fb92cd1: Pull complete 
4170a785b84a: Pull complete 
36570a7926c8: Pull complete 
34218f1ce9d6: Pull complete 
Digest: sha256:e9207a59d15739dec5d1b55412f15c0661383fad23f8d2914b7b688d193c0871
Status: Downloaded newer image for mwaeckerlin/ubuntu-base:latest
root@docker[bcfc1e9bc756]:/# cat /tmp/test
this is the outer container

As you see, there is no access to the images from outside of the outer container, docker ps does not show the host's containers and the directory cannot be mounted from outside of the container, but view is limited to the outer container. This is real encapsulation, that's how it must be.

Conclusion

There is no solution for docker in docker in a docker swarm, unless option --privileged is supported in docker swarm!

The option --privileged is a must to have docker containers in docker containers, encapsulated without access to the whole swarm!

@bitsofinfo

This comment has been minimized.

bitsofinfo commented Feb 9, 2018

Please add, needed for fuse!

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Feb 9, 2018

So, the solution of using -v /var/run/docker.sock:/var/run/docker.sock is a security issue and an absolute no-go!

Correct; if you’re blind-mounting the socket, you’re not running docker-in-docker; you’re controlling the hosts daemon from inside the container; in many cases this may actually be preferable, see http://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/

The option --privileged is a must to have docker containers in docker containers, encapsulated without access to the whole swarm!

First of all, only manager nodes have access to the whole swarm; worker nodes can only control the worker node itself

But be aware that —privileged is equivalent to having full root access on the host; there is no protection whatsoever, processes inside the container can escape the container, and have full access to the host (for example, have a look at /dev inside a privileged container, and you see it has access to all devices from the host)

@mwaeckerlin

This comment has been minimized.

mwaeckerlin commented Feb 9, 2018

@thaJeztah writes:

But be aware that —privileged is equivalent to having full root access on the host; there is no protection whatsoever, processes inside the container can escape the container, and have full access to the host (for example, have a look at /dev inside a privileged container, and you see it has access to all devices from the host)

Then let's adapt the requirement. What is a good solution to have:

  • Real independent Docker in Docker
  • Full security, no access to the host, no escape
  • Can run in a swarm

For me, not the --privileged flag is important, but the real and secure docker in docker.

Possible use cases:

  1. Jenkins with untrusted users that can setup their own build jobs
  2. Sell docker playground to any unkown audience, let them start any docker services from within docker

Currently I am running a jenkins server in a local docker container using --privileged, and I would like to migrate this to a swarm. The jenkins instanciates a dedicated docker container for every build, i.e. for cross-build to windows and to any linux distribution, e.g. to build dep-packaged for ubuntu xenial, it runs the build in a container from mwaeckerlin/ubuntu:xenial-amd64.

@AkihiroSuda

This comment has been minimized.

Member

AkihiroSuda commented Feb 9, 2018

@mwaeckerlin

We need to get nested runc to work without root privileges first.
You may want to follow this issue: opencontainers/runc#1658

@redhog

This comment has been minimized.

redhog commented Mar 5, 2018

I'd like to add that Kubernetes does support adding capabilities. For an example, check out https://caveofcode.com/2017/06/how-to-setup-a-vpn-connection-from-inside-a-pod-in-kubernetes/ (this is the exact same use case I have, and that I'd like to run on docker swarm).

@thiagolsfortunato

This comment has been minimized.

thiagolsfortunato commented Jun 13, 2018

Any news ?

@bitsofinfo

This comment has been minimized.

bitsofinfo commented Jun 14, 2018

the news is all k8!

@denis-isaev

This comment has been minimized.

denis-isaev commented Jun 15, 2018

+1

@killcity

This comment has been minimized.

killcity commented Jun 15, 2018

@kurtrwall

This comment has been minimized.

kurtrwall commented Jun 25, 2018

Just first off, thank you @thaJeztah and co-devs for being attentive on this. I'm sure this is a point of contention among yourselves.

Really just came here to say that I've been successful with bind mounting the docker socket and using docker binaries in the container with Jenkins in a swarm (not bind mounting the binaries). In that specific case, I've constrained Jenkins to only run on manager nodes. Anyone should make sure the permissions are set correctly, especially when you're working with GlusterFS, as all of my Jenkins nodes run on top of GlusterFS volumes with no issues so far (been about 6 months).

The --privileged flag was something I used when prototyping but when it wasn't available I found that just setting the correct permissions did the trick for my purposes; something everyone should be aware of and practicing when bind mounting anyway. My case is different than many others, so your mileage will vary.

@TAGC

This comment has been minimized.

TAGC commented Jul 16, 2018

Damn, forgot to come back and congratulate this issue on its second birthday.

@denis-isaev

This comment has been minimized.

denis-isaev commented Jul 16, 2018

zabbix-agent need privileged mode to monitor host resources.

@mister2d

This comment has been minimized.

mister2d commented Aug 8, 2018

Need privileged mode to run systemd enabled containers! Like dogtag-ca for instance.

@valleedelisle

This comment has been minimized.

valleedelisle commented Sep 5, 2018

Trying to deploy dell openmanage exporter and it doesn't work. I understand that --priv / --cap-add can be a security issue, but if I want to shoot myself in the foot, I should be able to do so.

@manvalls

This comment has been minimized.

manvalls commented Oct 7, 2018

Hey guys, I've been maintaining my own poorly-patched fork of docker for a while with support for --cap-add and --privileged in docker stack deploy, maybe we should create a proper fork?

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Oct 8, 2018

@manvalls I think --cap-add (or --capabilities, i.e., not merging defaults, but require the full set to be specified) is something that would be accepted; have you considered contributing, and opening a pull request to discuss that option?

@manvalls

This comment has been minimized.

manvalls commented Oct 9, 2018

Hi @thaJeztah, this has already been done:
#26849
docker/swarmkit#1565

Like many people here I've been closely monitoring these issues for a long time until it came clear that there were only two options: switching to kubernetes or forking docker.

I'm pretty sure I'm not the only one who opted to gather all those PR together and maintain their own docker fork, and I'm really glad I did so because I, like many others, love what you guys did with docker swarm.

If adding these features to upstream docker in a convenient way is against its principles, which really seems to be the case, it feels only natural for all those forks which lots of people are likely using already to unite together. Maybe even you guys could help maintain it, or even own it?

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Oct 9, 2018

@manvalls the PR you linked to was closed because that implementation did --cap-add / --cap-drop. The proposal was to have a --capabilities flag (which would override the defaults)
#26849 (comment) :

I think it is better long term if we switch to a --capabilities interface which says which you actually need, even if you might get more by default for compatibility.

If someone wants to work on that, that's a change that will likely get accepted.

@pwFoo

This comment has been minimized.

pwFoo commented Oct 31, 2018

Also need device, privileged, cap_add / cap_drop. No progress here?

@AkihiroSuda

This comment has been minimized.

Member

AkihiroSuda commented Oct 31, 2018

@pwFoo A workaround is to bind-mount /var/run/docker.sock and call docker run within service containers.

Maintainers: Although I agree we should support "entitlements" in the long-term plan, as we already have been supporting bind-mounting arbitrary path including docker.sock, I don't see any security degradation in implementing docker service create --privileged.

@olljanat

This comment has been minimized.

Contributor

olljanat commented Oct 31, 2018

@pwFoo there is that #26849 PR which you can take as template and just implement asked changes to get it merged (look comments above). That beauty of open source that you can do that yourself 😉

It is still possible to get that feature out on 19.03 if someone just take to step and start implement it.

EDIT: Looks that #26849 have been reopened today :)

@pwFoo

This comment has been minimized.

pwFoo commented Oct 31, 2018

I'm new with Go. So I don't think I should do it...
But maybe someone with more experience could do it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment