Add support --privileged when create service

[jimmyxian]

When create some container, we need privileged=True. But swarmkit does not support this.
I want to implement this by the following steps:

• Add privileged field in ContainerSpec.
• Pass privileged to HostConfig when create container.

If it's the right direction, I will submit a PR. :)

Also, there so many container options in docker-engine. If all the options are supported in that way, It's a huge work.Maybe we need have a good way to support all the options?

[dmcgowan]

Privileged containers are needed for any use case which needs to spin up Docker-in-Docker, a common pattern for testing not only docker itself but complex build setups such as allowed by https://github.com/docker/golem. I am looking forward to porting golem to swarm when this is enabled.

[stevvooe]

Also, there so many container options in docker-engine. If all the options are supported in that way, It's a huge work.Maybe we need have a good way to support all the options?

We don't actually want to do this. There are number of things that are hard or impossible to support in a clustered environment. There are other things that are downright insecure, such as privileged, that need to be rethought.

What aspect of privileged containers are you using? For example, what part enables DIND to work?

[dmcgowan]

We don't actually want to do this.

Many people will and it is a common pattern. However I do agree that time should be spent to figure out how to do it right since adding this will mean it can never be removed along with any security headaches it brings.

As for DIND in particular. I am not sure of all the settings that would need to be added to enable this, getting this working I think would cover many use cases for CI. Try running the make test run command without the privileged flag will yield...

mount: permission denied
Could not mount /sys/kernel/security.
AppArmor detection and --privileged mode might break.
mount: permission denied

The reason is obvious from https://github.com/docker/docker/blob/master/hack/dind.

[stevvooe]

@dmcgowan Supporting all of the container and host config options levied a massive amount of complexity on the swarm project for little gain. Each feature we add in services needs to be well-considered and designed for operation in a clustered environment. This is especially true for anything that requires privileged execution. (interesting aspect: ContainerSpec ends up looking a lot like HostConfig container.Config, in an image).

Right now, swarm services aren't really setup well for CI or build use cases. We don't really have the concept of batch jobs and collecting logs is problematic. This will come in the future.

Do we have any online use cases for actual services?

[dmcgowan]

Looking at the container setup code within docker the following is done by setting the privileged flag

• Set empty process label
• Warn of incompatibility with user namespaces
• Add all host devices from /dev
• Add device cgroup access rwm allow, see https://github.com/docker/docker/blob/master/daemon/oci_linux.go#L92
• Add all capabilities
• Clear read only flag for /sys mount
• Set read only paths to nil (*specs.Spec).Linux.ReadonlyPaths = nil, see https://github.com/docker/docker/blob/master/daemon/oci_linux.go#L561
• Set masked paths to nil (*specs.Spec).Linux.MaskedPaths = nil, see https://github.com/docker/docker/blob/master/daemon/oci_linux.go#L562
• Clear read only flag for cgroup mount
• Set app armor profile "unconfined"

[stevvooe]

@dmcgowan Wow, thanks for the overview!

cc @aluzzardi

[chanwit]

Global service fits really well for deploying monitoring agents on every node. And most of those agents require privileged access to some /proc. In my case, I want to monitor conntrack.

Please add support for this privileged flag, cc @aluzzardi

[stevvooe]

@chanwit: when not running with --privileged, what error do you get when trying to monitor conntrack?

[aluzzardi]

@stevvooe @chanwit @jimmyxian I was initially against adding --privileged. Seeing how many requests we got for this, I'm in favor for adding it now

[jimmyxian]

@aluzzardi Thanks, I will rebase the PR. :)

[chanwit]

@stevvooe my bad. it's another issue :-(
not relating to privileged.

Anyway I gave up running monitoring agents inside container for now.

[sashkachan]

Trying to run weave-scope using the new swarm, also hitting the roadblock with the privileged mode (or the lack thereof). Would be great to have it supported.

[lukemarsden]

Any update on this @aluzzardi? Would be great to get weave scope working with swarmkit. And we really do need to access /proc. cc @errordeveloper

[errordeveloper]

@alex-glv we have hack that makes it work until this gets resolved: https://github.com/weaveworks/scope-global-swarm-service.

Also, there is a conversation around explicit --capabilities flag: moby/moby#26849 (comment).

[thaJeztah]

Also, related issue in docker/docker moby/moby#24862

[alexellis]

Where are we with this issue now? @aluzzardi - still feeling positive?

[marcellodesales]

:( I need to run Global services please!

[stevvooe]

Note that we are attempting to address this with security profiles, proposed in #1722. The goal is to provide cluster operators much finer grain of access control for security containment without making the model more complex that --privileged.

[shankarkc]

Is there an ETA for this? I am hitting small /dev/shm on docker containers. I have to increase the size. I have used service to create containers. Now as service doesnt allow me to run in privilaged mode, I cannot unmount/remount /dev/shm

[thaJeztah]

@shankarkc would mounting a tmpfs work? Haven't tried, but;

--mount type=tmpfs,target=/dev/shm,.....

Possibly requires docker 1.13 (for tmpfs)

[shankarkc]

Thanks for input. Installed 1.3 It works.

…

On Nov 23, 2016 9:17 PM, "Sebastiaan van Stijn" ***@***.***> wrote: @shankarkc <https://github.com/shankarkc> would mounting a tmpfs work? Haven't tried, but; --mount type=tmpfs,target=/dev/shm,..... Possibly requires docker 1.13 (for tmpfs) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1030 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEb7VwTJd6gbf-x7o2Cf5vV8NEGTbMGyks5rBF_zgaJpZM4I5UR9> .

[ginjo]

Curious where we are on the issue of allowing deeper privileges in swarm containers. My use case involves vpn software that requires addition of an ip interface and a couple of routing rules.

Is the focus of this issue now on security profiles #1722 , and if so, will they allow deeper privileges in swarm containers?

[mikeytag]

I second @ginjo. I'm trying to get an OpenVPN container up in swarm but can't without privileged, cap_add=NET_ADMIN and device=/dev/net/tun

If anyone has a workaround. I'm all ears.

[marcellodesales]

I have a couple of use cases for the need of running mode=global containers in my cluster:

• We are still using older Monitoring images that require privileges...
• I have a raspberry pi cluster that accesses GPIO that require privileges...

Any status?

[sono-bfio]

👍 This or the at least the ability to pass in devices in my cases (Working with nvidia GPUs in my case).

[mathiasbrito]

I’ll say just one word on why It should be implemented, IoT, more and more people are considering the use of docker container for IoT scenarios, and we need privileged containers to access attached devices sometimes. Using swarm to manage a set of IoT devices is really interesting, but definitely without privileged devices, it loses a lot of its magic in case of IoT.

[olljanat]

Swarmkit team have made proposal of device support to #2682

Please comment to there your thought about if that fits to your use cases.

EDIT: There now there looks to be suggested solution on this message:
moby/moby#24862 (comment)

[wanyvic]

I found a solution to solve the problem.and I also can use cap_net_admin in swarm mode.
you should modify the runtime source code to add the capabilities which you need.(it will be a local default setting).
for example I add the CAP_NET_ADMIN to my runtime(used nvidia-container-runtime)
wanyvic/nvidia-container-runtime.
After that rubuild it. started a container(use swarm mode), input:
capsh --print
CAP_NET_ADMIN can be found:
root@25303a54ebb3:/# capsh --print Current:=cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_admin,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap+eip Bounding set =cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_admin,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap Securebits: 00/0x0/1'b0 secure-noroot: no (unlocked) secure-no-suid-fixup: no (unlocked) secure-keep-caps: no (unlocked) uid=0(root) gid=0(root) groups= root@25303a54ebb3:/#
this method is not good.It also can't to set the cap_add or cap_drop in the docker-compose.yml.
but I can't find the other way to solve it.

[FZZFh]

When create some container, we need privileged=True. But swarmkit does not support this.
I want to implement this by the following steps:

• Add privileged field in ContainerSpec.
• Pass privileged to HostConfig when create container.

If it's the right direction, I will submit a PR. :)

Also, there so many container options in docker-engine. If all the options are supported in that way, It's a huge work.Maybe we need have a good way to support all the options?

When create some container, we need privileged=True. But swarmkit does not support this.
I want to implement this by the following steps:

• Add privileged field in ContainerSpec.
• Pass privileged to HostConfig when create container.

If it's the right direction, I will submit a PR. :)

Also, there so many container options in docker-engine. If all the options are supported in that way, It's a huge work.Maybe we need have a good way to support all the options?

hi,i meet this problem again , i don't know how to Add 'privileged' field in ContainerSpec. and Pass 'privileged' to HostConfig when create container ,in fact ,i don't know it's mean,
can you just give me a guide to achieve the two things .
sorry for my poor english.
老哥，这两句是要配置啥啊，我没看懂英文，可以说明一下吗，谢谢：）

[olljanat]

@FZZFh this issue is about feature request. It does not exist yet.

I'm working on it and you can see status on moby/moby#25885 (comment) target is get it released as part of Docker 19.06.

[olljanat]

moby/moby#39173 was merged so this feature will ship as part of Docker 19.06 / 19.09 (not sure actual version yet).

Please, look my question about CLI side implementation on moby/moby#24862 (comment)

[prologic]

Very nice! I've been anxiously waiting or this feaure for years!

[couillonnade]

I am wondering if there is a safe way to use it now since neither 19.06 nor 19.09 seem to be on the the horizon. Could not find any information about the next stable release and what happened to the release cycle we use to have. Anyone has information on that?

[olljanat]

FYI moby/moby#25885 (comment)

[brian-finisher]

I think there is still very much a strong case for the equivalent of --privileged in Docker Swarm, particularly for the embedded, robotics, IoT and hobbyist spaces. It seems this may be the best use of Docker Swarm these days, anyways, as the rest of the world is migrating to Kubernetes and apparently has the compute resources to do so.

There was a simple solution

From what I can gather, arguments against privileged support are effectively that it is too open and finer grained allowances should be given.
#1129 added this feature, but was shot down
Then there were security profiles. Shot down
Then there were Entitlements. Doesn't seem to be any progress on it.

I'm very empathetic to striving for a perfect solution, particularly for cloud users (though... are there many left?). But, this request is from 2016. Here we are, 6 years later, and do not have any working solutions - but several open and unimplemented proposals. The world and Swarm's userbase seems to have changed in that time.

Inconsistent with Docker CLI & docker-compose

It is both surprising and confusing that features available in "docker run" and docker-compose yaml are not supported in swarm mode. Why, exactly, am I allowed to use --privileged in these but not in swarm? I should very much like there to be consistent behavior between these modes. As an end-user, I basically just want Swarm to be a cluster-equivalent layer over docker-compose, which is a config-file layer above 'docker run'. While I know this isn't the actual architecture, it does make sense conceptually.

Problems with forcing explicit devices in the device proposal

While I appreciate the thinking behind the device proposal for swarm, it does create a bit of a nightmare to have to explicitly list out allowed devices - particularly when those devices may be somewhat arbitrary or random, or dynamically created/destroyed.

Take, for instance, USB devices. I may have a USB device that ends up on /dev/bus/usb/001/003 or perhaps /dev/bus/usb/002/001 and whatnot. It may just depend on how somebody was feeling like plugging things in that day.

Ideally, I would like any and all USB devices available to my containers on a given node. I want the logic for enumerating those devices, finding the correct serial number, etc, to live within a container. I don't want to have to edit .yml files every time a hardware config changes.

With --privileged, it is possible to bind-mount an entire directory of devices. /dev/bus/usb. Now my containers can access any USB devices, enumerate USB devices, etc. If a USB hub is connected, disconnected, etc, it will be available in the container dynamically. This is not the case with the --devices proposal equivalent.

There are no other reasonable cluster orchestration alternatives for embedded systems

I have researched running Kubernetes on our embedded computers, and, it is a resource hog. Even microk8s has kubelite at about 540mb of RAM usage, 30-40% CPU just sitting there - not even running any containers. Taking up 1/4th the resources of a 2GB Raspberry Pi or Jetson Nano is unacceptable. Docker Swarm's unique advantage is its efficiency.

Balena doesn't support swarms/clusters.

Without privileged containers, my only current reasonable option is to give up orchestration with a single .yml config, and instead need to install and run docker-compose on every one of my nodes. I'll have to ssh into each one manually to start/stop services.

[djmaze]

@brian-finisher The workaround given by AkihiroSuda works pretty fine for me. Can also be easily specified in a stack yaml.

(I admit a clean solution would be nicer though.)

[bluepuma77]

We just started detailed monitoring of Docker Swarm with prometheus. We need to watch drives on dedicated servers, so smartmon is the tool of choice. After testing with a container in --privileged mode, I wanted to create a service to run it on all nodes - just to find that --privileged is not supported in Docker Swarm. What???

I agree with @brian-finisher that we would love to see a pragmatic solution, that is available in short term. Discussions on over-engineered solution with granular security settings that don't get implemented do net help, they just drive users away. The --privileged functionality is already in Docker, so it should be not hard to enable it in Swarm.

Please give Docker Swarm users the opportunity to use --privileged, we are aware of the security implications.

[Hobbit44]

As we roll into 2023 and coming up on 7 years that this issue has been around, any sign of this being done?

[olljanat]

Up to date pull request switch implements this exist on #3072 but because maintainers didn't wanted to pickup it to version 23.0.0 that can happen earliest on version which comes after that (API changes are only allowed in main versions).

Afaiu maintainers are now busy to finalize that version and after that discussion can start (once again) that what would be solution which they would be ready to accept.