New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support --privileged when create service #1030

Open
jimmyxian opened this Issue Jun 20, 2016 · 50 comments

Comments

Projects
None yet
@jimmyxian
Copy link
Contributor

jimmyxian commented Jun 20, 2016

When create some container, we need privileged=True. But swarmkit does not support this.
I want to implement this by the following steps:

  • Add privileged field in ContainerSpec.
  • Pass privileged to HostConfig when create container.

If it's the right direction, I will submit a PR. :)

Also, there so many container options in docker-engine. If all the options are supported in that way, It's a huge work.Maybe we need have a good way to support all the options?

@dmcgowan

This comment has been minimized.

Copy link
Member

dmcgowan commented Jun 30, 2016

Privileged containers are needed for any use case which needs to spin up Docker-in-Docker, a common pattern for testing not only docker itself but complex build setups such as allowed by https://github.com/docker/golem. I am looking forward to porting golem to swarm when this is enabled.

@stevvooe

This comment has been minimized.

Copy link
Contributor

stevvooe commented Jul 6, 2016

Also, there so many container options in docker-engine. If all the options are supported in that way, It's a huge work.Maybe we need have a good way to support all the options?

We don't actually want to do this. There are number of things that are hard or impossible to support in a clustered environment. There are other things that are downright insecure, such as privileged, that need to be rethought.

What aspect of privileged containers are you using? For example, what part enables DIND to work?

@dmcgowan

This comment has been minimized.

Copy link
Member

dmcgowan commented Jul 6, 2016

We don't actually want to do this.

Many people will and it is a common pattern. However I do agree that time should be spent to figure out how to do it right since adding this will mean it can never be removed along with any security headaches it brings.

As for DIND in particular. I am not sure of all the settings that would need to be added to enable this, getting this working I think would cover many use cases for CI. Try running the make test run command without the privileged flag will yield...

mount: permission denied
Could not mount /sys/kernel/security.
AppArmor detection and --privileged mode might break.
mount: permission denied

The reason is obvious from https://github.com/docker/docker/blob/master/hack/dind.

@stevvooe

This comment has been minimized.

Copy link
Contributor

stevvooe commented Jul 6, 2016

@dmcgowan Supporting all of the container and host config options levied a massive amount of complexity on the swarm project for little gain. Each feature we add in services needs to be well-considered and designed for operation in a clustered environment. This is especially true for anything that requires privileged execution. (interesting aspect: ContainerSpec ends up looking a lot like HostConfig container.Config, in an image).

Right now, swarm services aren't really setup well for CI or build use cases. We don't really have the concept of batch jobs and collecting logs is problematic. This will come in the future.

Do we have any online use cases for actual services?

@dmcgowan

This comment has been minimized.

Copy link
Member

dmcgowan commented Jul 7, 2016

Looking at the container setup code within docker the following is done by setting the privileged flag

@stevvooe

This comment has been minimized.

Copy link
Contributor

stevvooe commented Jul 7, 2016

@dmcgowan Wow, thanks for the overview!

cc @aluzzardi

@chanwit

This comment has been minimized.

Copy link

chanwit commented Jul 9, 2016

Global service fits really well for deploying monitoring agents on every node. And most of those agents require privileged access to some /proc. In my case, I want to monitor conntrack.

Please add support for this privileged flag, cc @aluzzardi

@stevvooe

This comment has been minimized.

Copy link
Contributor

stevvooe commented Jul 11, 2016

@chanwit: when not running with --privileged, what error do you get when trying to monitor conntrack?

@aluzzardi

This comment has been minimized.

Copy link
Contributor

aluzzardi commented Jul 11, 2016

@stevvooe @chanwit @jimmyxian I was initially against adding --privileged. Seeing how many requests we got for this, I'm in favor for adding it now

@jimmyxian

This comment has been minimized.

Copy link
Contributor

jimmyxian commented Jul 12, 2016

@aluzzardi Thanks, I will rebase the PR. :)

@chanwit

This comment has been minimized.

Copy link

chanwit commented Jul 12, 2016

@stevvooe my bad. it's another issue :-(
not relating to privileged.

Anyway I gave up running monitoring agents inside container for now.

@alex-glv

This comment has been minimized.

Copy link

alex-glv commented Jul 13, 2016

Trying to run weave-scope using the new swarm, also hitting the roadblock with the privileged mode (or the lack thereof). Would be great to have it supported.

@lukemarsden

This comment has been minimized.

Copy link

lukemarsden commented Sep 7, 2016

Any update on this @aluzzardi? Would be great to get weave scope working with swarmkit. And we really do need to access /proc. cc @errordeveloper

@errordeveloper

This comment has been minimized.

Copy link

errordeveloper commented Oct 12, 2016

@alex-glv we have hack that makes it work until this gets resolved: https://github.com/weaveworks/scope-global-swarm-service.

Also, there is a conversation around explicit --capabilities flag: moby/moby#26849 (comment).

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Oct 15, 2016

Also, related issue in docker/docker moby/moby#24862

@alexellis

This comment has been minimized.

Copy link

alexellis commented Nov 7, 2016

Where are we with this issue now? @aluzzardi - still feeling positive?

@marcellodesales

This comment has been minimized.

Copy link

marcellodesales commented Nov 8, 2016

:( I need to run Global services please!

@stevvooe

This comment has been minimized.

Copy link
Contributor

stevvooe commented Nov 8, 2016

Note that we are attempting to address this with security profiles, proposed in #1722. The goal is to provide cluster operators much finer grain of access control for security containment without making the model more complex that --privileged.

@shankarkc

This comment has been minimized.

Copy link

shankarkc commented Nov 23, 2016

Is there an ETA for this? I am hitting small /dev/shm on docker containers. I have to increase the size. I have used service to create containers. Now as service doesnt allow me to run in privilaged mode, I cannot unmount/remount /dev/shm

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Nov 23, 2016

@shankarkc would mounting a tmpfs work? Haven't tried, but;

--mount type=tmpfs,target=/dev/shm,.....

Possibly requires docker 1.13 (for tmpfs)

@shankarkc

This comment has been minimized.

Copy link

shankarkc commented Nov 25, 2016

@ginjo

This comment has been minimized.

Copy link

ginjo commented Jan 25, 2017

Curious where we are on the issue of allowing deeper privileges in swarm containers. My use case involves vpn software that requires addition of an ip interface and a couple of routing rules.

Is the focus of this issue now on security profiles #1722 , and if so, will they allow deeper privileges in swarm containers?

@mikeytag

This comment has been minimized.

Copy link

mikeytag commented Jan 28, 2017

I second @ginjo. I'm trying to get an OpenVPN container up in swarm but can't without privileged, cap_add=NET_ADMIN and device=/dev/net/tun

If anyone has a workaround. I'm all ears.

@marcellodesales

This comment has been minimized.

Copy link

marcellodesales commented Feb 1, 2017

I have a couple of use cases for the need of running mode=global containers in my cluster:

  • We are still using older Monitoring images that require privileges...
  • I have a raspberry pi cluster that accesses GPIO that require privileges...

Any status?

@sono-bfio

This comment has been minimized.

Copy link

sono-bfio commented Feb 2, 2017

👍 This or the at least the ability to pass in devices in my cases (Working with nvidia GPUs in my case).

@ventz

This comment has been minimized.

Copy link

ventz commented Feb 3, 2017

Same use case as @mikeytag -- need to add OpenVPN as a service.

@getvivekv

This comment has been minimized.

Copy link

getvivekv commented Feb 10, 2017

@AkihiroSuda Would you please elaborate how this can be done ?

@AkihiroSuda

This comment has been minimized.

Copy link
Member

AkihiroSuda commented Feb 10, 2017

The command I mentioned above is composed of two parts:

  1. docker service create --mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock foo
  2. Execute docker run --privileged in the service container created in the step 1

The step 1 creates a service with the access to Docker API socket.
So you need to install the docker binary (only client is enough) to the image foo.

In the step 2, you can create privileged containers using the socket.

Note that the privileged programs still does not work on the service.
You can just let the service create new containers for the privileged programs.
So you might need some additional work about the lifecycle of the containers created.

@alexellis

This comment has been minimized.

Copy link

alexellis commented Feb 11, 2017

@getvivekv

At best this is a temporary work-around ... at worst it's confusing and as @AkihiroSuda mentions it opens a can of worms around managing the privileged containers created. You may be better off with creating your privileged containers manually and then using an attachable overlay network so that you can join them to swarm services.

I.e.

  • Create an attachable network
  • Deploy services onto the swarm
  • Create ad-hoc containers with --network flag
$ docker network create mixed --driver overlay --attachable
c9xwtgk8259j8rnq3xifvt9kv
$ docker service create redis --network=mixed redis
$ docker run --network=mixed -d some_privileged_image_that_uses_redis

+1 x 1000 for --privileged services on Swarm, for Raspberry Pi etc.

Cross-referencing:

  • (Missing from Swarmmode --cap-add #25885)

moby/moby#25885

  • api: add support for security profiles #1722

#1722

@stevvooe

This comment has been minimized.

Copy link
Contributor

stevvooe commented Feb 13, 2017

@alexellis @marcellodesales @getvivekv #1722 is definitely the way forward here. I am not sure what the major concerns are and I have not yet seen a better proposal that provides effective node-level ACL controls of privileged execution.

I also am not sure that the implications of using privileged are broadly understood. For example, the workaround is something you'd not want to run on a manager node, as it may get access to key material. This is why we are generally asking for use case (ie monitoring, gpio, etc.), so thanks for bearing with us. The more narrow we make this, the more secure your containers can be by default.

@gerwim

This comment has been minimized.

Copy link

gerwim commented Mar 8, 2017

@stevvooe Thanks for the heads up. For me, the use case would consist of two topics: NFS and OpenVPN. They either require --privileged or the appropriate --cap-add abilities.

@RRAlex

This comment has been minimized.

Copy link

RRAlex commented Mar 8, 2017

Adding --privileged would be a step backward, we really need to re-integrate control over capabilities (and seccomp profiles I'm guessing) on services/swarm.

In some case (OpenVPN and other) it's actually blocking swarm use, in other case, it's leaving too much capabilities in a container that doesn't require all of them and thus making it more exploitable.

My 2¢, but this goes against all the security that docker had achieved and also ruins everyone's effort put into securing their containers and finding what to add or remove. Platform agnosticism should never compromise the ability to use the local security measure of an OS. Worst case, some platform should simply ignore some options.

@mikeytag

This comment has been minimized.

Copy link

mikeytag commented Mar 14, 2017

I thought I would add a little more context into our use case with OpenVPN and Swarm to give people much smarter than myself more info into what would be the best way to implement this. I agree with many of the devs on this thread that adding --privileged is the wrong answer and we need something with much more granular control.

We are using an overlay network that our containers all deploy with stacks. This central network allows us to use aliases so that the right containers can connect to others over certain ports. What I need to do occasionally is VPN into our "cluster" and then be able to SSH/connect to ports on other containers via the Swarm network. These are ports that are not exposed to the outside world but just used internally. The awesome auto DNS feature is what I'm hoping to tap into most.

For now, I have a container called vpn which exposes SSH over port 2222 and I connect to it from my laptop/workstation with SSHuttle so I can use it as a jump host of sorts to connect to other containers when needed. This works reasonably well as a poor man's VPN but using OpenVPN would be a lot cleaner.

It helps to not have to give every dev docker-machine exports and certs and such. Instead we have one container that gives them access to whatever other container they need to diagnose and solve problems within specific containers. Running docker attach everywhere is just a lot messier, not to mention we'd be sharing the private keys of the instances created with docker-machine all over the place which I'm not super keen to do.

Anyway, I hope that gives some clarity as to what we are trying to do with OpenVPN and Swarm. I'm sure others have different setups and scenarios. The key here is to keep Docker Swarm secure and scalable with whatever solution is developed.

EDIT: I should add that I would totally do one privileged container run manually on one machine for OpenVPN but I can't run containers like that outside of Swarm and connect them to the overlay network that Swarm is using.

@RRAlex

This comment has been minimized.

Copy link

RRAlex commented Mar 15, 2017

@mikeytag if you declare your overlay networks as --attachable, you should be able to run things manually on them from other machines.
It's not idea at all, but until capabilities are re-implemented && if your needs are static 😥

@mikeytag

This comment has been minimized.

Copy link

mikeytag commented Mar 17, 2017

Interesting. I wish I would have done that before I had everything running in production! Looks like I might need to schedule some down time late at night in the future to try to make that work.

@justintime4tea

This comment has been minimized.

Copy link

justintime4tea commented Mar 18, 2017

I know this is the wrong place but I have no idea where to find the answer and I'm sure many of you could point me in the right direction. I'll delete comment afterwords to keep the issue clean. This is related to --cap-add however. How do I regenerate the protobufs in a vendored (swarmkit) "paclage" (wrong term?) ? I am no newb developer by any means but I must admit all this go stuff is making me goooo crazy :) Thanks in advanced.

@waltherg

This comment has been minimized.

Copy link

waltherg commented May 12, 2017

@mikeytag @RRAlex I hit the same limitation when attempting to implement the exact same scenario that @mikeytag described (OpenVPN server in overlay network to access ports I do not want to open up to the world).

I attempted testing the --attachable network variant but got an error I can't make sense of:

$ docker network create --driver overlay --attachable test_network --opt encrypted
$ docker service create --name test_webserver --hostname test_webserver --network test_network --restart-condition any nginx
$ docker run -v MY_OPENVPN_VOLUME:/etc/openvpn -d -p 1195:1194/udp --cap-add=NET_ADMIN --network=test_network kylemanna/openvpn

For the docker run invocation I get the following error message:

docker: Error response from daemon: Could not attach to network test_network: context deadline exceeded.

The OpenVPN server is configured as described here (https://github.com/kylemanna/docker-openvpn) and works as expected in the scenario where my Docker network is a single-host bridge network (i.e. non-swarm mode).

Any suggestions on how to proceed here would be greatly appreciated!
The Docker server version we run is 1.13.

Update: I stand corrected this actually worked now on my second attempt. No clue what caused the aforementioned error. I still see odd behavior though: The nginx service in my above example has, say IP (virtual IP?) 10.0.0.2 which I can't reach from my work laptop when connected to the OpenVPN server (neither ping nor curl reach that IP). However I can reach the actual container's IP (10.0.0.4) from my laptop. I suppose Docker swarm mode redirects from the service's name / IP to the respective containers' names / IPs for load-balancing and being unable to reach the service's IP from outside may be by design?

@waltherg

This comment has been minimized.

Copy link

waltherg commented May 12, 2017

@mikeytag does the sshuttle approach you describe work without --cap-add NET_ADMIN? Do you start the sshuttle container through docker service and host constraint or regularly with docker run and --network NAME_OF_YOUR_SWARM_OVERLAY_NETWORK?

@mikeytag

This comment has been minimized.

Copy link

mikeytag commented May 15, 2017

@waltherg when I was using sshuttle I was deploying the "vpn" container via stacks but making sure that it was deployed to a specific machine and exposing the port 2222 so the container SSH could respond to the outside world.

I got the OpenVPN setup working too using --attachable. Here's the run command that I use. Maybe there's something in here that's valuable to you or others.

docker run -itd \
  -h vpn-1 \
  --restart=always \
  --name vpn-1 \
  --network=NETWORKNAMEHERE \
  --cap-add=NET_ADMIN \
  --privileged=true \
  -p 1194:1194 \
  --ulimit nofile=65536:65536 \
  YOURFAVORITEOPENVPNIMAGEHERE:latest
@dcarastan

This comment has been minimized.

Copy link

dcarastan commented Aug 22, 2017

I ran into it while building a Concourse CI stack. The privileged options is needed for apps that orchestrate containers. Keep this option around and make it a swarm init parameter. Privileged containers should not be accepted by default.

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Sep 18, 2017

Linking the entitlements proposal here, which is relevant moby/moby#32801

@suda

This comment has been minimized.

Copy link

suda commented Apr 15, 2018

Having the same issues with VPN containers as @mikeytag and @waltherg I ended up running them as systemd services. At least this way they get to be supervised. This, combined with Alex's work-around to allow them to talk on the same network seems to do the trick.

I agree that the privileged mode really might be a huge security issue but at least the ability to manage capabilities should give some level of granularity.

@dbiswas1

This comment has been minimized.

Copy link

dbiswas1 commented Apr 26, 2018

I am having the same issue on launching the Chrome container. chrome container requires the container to be in privileged mode. is anybody here setup chrome in SWARM? i am all ears to hear a solution

@shankarkc

This comment has been minimized.

Copy link

shankarkc commented Apr 27, 2018

I have selenium grid running in swarm mode. What is the issue @dbiswas1 ? Can you give more details?

@alexellis

This comment has been minimized.

Copy link

alexellis commented Apr 27, 2018

With your grid were you also able to mount /dev/shm and have it work? This is generally needed for Chrome.

@shankarkc

This comment has been minimized.

Copy link

shankarkc commented Apr 27, 2018

@mathiasbrito

This comment has been minimized.

Copy link

mathiasbrito commented Jul 31, 2018

I’ll say just one word on why It should be implemented, IoT, more and more people are considering the use of docker container for IoT scenarios, and we need privileged containers to access attached devices sometimes. Using swarm to manage a set of IoT devices is really interesting, but definitely without privileged devices, it loses a lot of its magic in case of IoT.

@olljanat

This comment has been minimized.

Copy link
Contributor

olljanat commented Oct 9, 2018

Swarmkit team have made proposal of device support to #2682

Please comment to there your thought about if that fits to your use cases.

EDIT: There now there looks to be suggested solution on this message:
moby/moby#24862 (comment)

@wanyvic

This comment has been minimized.

Copy link

wanyvic commented Oct 25, 2018

I found a solution to solve the problem.and I also can use cap_net_admin in swarm mode.
you should modify the runtime source code to add the capabilities which you need.(it will be a local default setting).
for example I add the CAP_NET_ADMIN to my runtime(used nvidia-container-runtime)
wanyvic/nvidia-container-runtime.
After that rubuild it. started a container(use swarm mode), input:
capsh --print
CAP_NET_ADMIN can be found:
root@25303a54ebb3:/# capsh --print Current:=cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_admin,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap+eip Bounding set =cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_admin,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap Securebits: 00/0x0/1'b0 secure-noroot: no (unlocked) secure-no-suid-fixup: no (unlocked) secure-keep-caps: no (unlocked) uid=0(root) gid=0(root) groups= root@25303a54ebb3:/#
this method is not good.It also can't to set the cap_add or cap_drop in the docker-compose.yml.
but I can't find the other way to solve it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment