Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing from Swarmmode --cap-add #25885

Open
alexellis opened this issue Aug 19, 2016 · 82 comments · May be fixed by docker/cli#2199

Comments

@alexellis
Copy link
Contributor

@alexellis alexellis commented Aug 19, 2016

Some form of --cap-add or optional elevated privilege system would be required for accessing GPIO pins on ARM devices. Since ARM is becoming better supported by the Docker engine I would like to raise this to attention.

We tend to need to write to /dev/mem and there is currently a capability for that in "regular flavoured swarm".

I would like to build out some IoT PoCs with Docker and swarmmode and support for this would really help. CC/ @DieterReuter @StefanScherer

@justincormack

This comment has been minimized.

Copy link
Contributor

@justincormack justincormack commented Aug 19, 2016

Ugh, /dev/mem gives you full access to the whole memory of the machine, ie even more than root. Isn't there some more sane API for gpio?

@alexellis

This comment has been minimized.

Copy link
Contributor Author

@alexellis alexellis commented Aug 19, 2016

I know that it's opening a can of worms. There is also /dev/i2c which would be very useful too. There are some workarounds but they generally involve root at some point.

Writing to /sys/class/gpio/ may also be an option, but may involve re-writing Pi libraries.

This might need more looking into, here are a couple of related links.

https://dissectionbydavid.wordpress.com/2013/10/21/raspberry-pi-using-gpio-wiringpi-without-root-sudo-access/

http://elinux.org/RPi_GPIO_Code_Samples

@justincormack

This comment has been minimized.

Copy link
Contributor

@justincormack justincormack commented Aug 19, 2016

/sys/class/gpio seems a sane interface, and docker service create --mount src=/sys/class/gpio,dst=/sys/class/gpio,type=bind ... ought to work right now - can you test?

@alexellis

This comment has been minimized.

Copy link
Contributor Author

@alexellis alexellis commented Sep 13, 2016

It would be worth trying. I checked with the guys at Pimoroni and they advised against using these interfaces claiming high latency.

alex-berger added a commit to alex-berger/docker that referenced this issue Sep 23, 2016
Fixes moby#25885

Having this in place we can now use capabilities in the following CLI calls respectively in the corresponding API calls.
- docker service create
- docker service update

Signed-off-by: Alexander Berger <alex-berger@gmx.ch>
@jmMeessen

This comment has been minimized.

Copy link

@jmMeessen jmMeessen commented Oct 19, 2016

@alexellis
Could you please clarify your phrase "[Pimoroni] advised against using these interfaces claiming high latency"

I am currently (for a docker demo/training) developing containers using Pimoroni Piglow and Display-o-tron Hat on RPi's. I faced the issue of the --privileged flag. I didn't find it very "well behaved" and preferred to show/teach the principles of limiting access.
I finally got it to work by specifying the --device /dev/i2c-1 flag.

Is this the recommended way to do it or did I miss something ?

By the way, this is where I store my experiments https://github.com/jmMeessen/rpi-docker-images
Next step is to containerize the Display-o-tron and integrate the exercises in a swarm. (and write/publish some notes)

PS: I am using the latest Hypriot distribution, V1.1.0

@alexellis

This comment has been minimized.

Copy link
Contributor Author

@alexellis alexellis commented Oct 19, 2016

This is a bit off-topic but hopefully @justincormack et al will tolerate it.

@jmMeessen the /sys/class/gpio interface allows interaction with GPIO (not I2C) and has a high latency. Pimoroni have advised against it and the majority of their devices assume full unprivileged access to memory to use memory mapping etc with the GPIO pins.

I2c is a different scenario - a single device is often enough for interaction i.e. /dev/i2c-1. This is how scroll-phat works for instance.

In general you should not need to port or re-write any of their code to use it in Docker, but you will often need to run as a privileged container. Privileged containers are not possible with Swarm Mode but classic swarm will allow them to run.

So in summary: i2c may be different, but GPIO-based libraries probably need full access to memory. Sometimes the Pimoroni code just wraps existing libraries - so take each one case-by-case. Maybe even install the Pimoroni libraries with pip? I think they are working to port all libraries to apt-get.

Classic swarm: https://github.com/alexellis/datacenter-sensor

ARM Docker images (including Pimoroni):
https://github.com/alexellis/docker-arm/tree/master/images/armv6
https://github.com/alexellis/docker-arm/tree/master/images/armhf

Hypriot or Raspbian should not make a difference - they are both Debian derivatives which run the same Debian (.deb) packages from get.docker.com.

@justincormack

This comment has been minimized.

Copy link
Contributor

@justincormack justincormack commented Nov 3, 2016

See docker/swarmkit#1722 for initial discussion of capabilities and privileged framework for swarmkit.

@RRAlex

This comment has been minimized.

Copy link

@RRAlex RRAlex commented Mar 13, 2017

While waiting for docker/swarmkit#1722, is there a way (beside recompiling the daemon) to change the default capabilities of spawned containers (even daemon wide) to be able to run those that require something like NET_ADMIN?

@albers

This comment has been minimized.

Copy link
Member

@albers albers commented Mar 16, 2017

I would like to use keepalived in a swarm, which requires NET_ADMIN capability.

@sirlatrom

This comment has been minimized.

Copy link
Contributor

@sirlatrom sirlatrom commented Mar 27, 2017

FWIW, elasticsearch requires the IPC_LOCK capability, making it impossible to deploy a Swarm Mode stack with ElasticSearch until this is resolved...

@a-jung

This comment has been minimized.

Copy link

@a-jung a-jung commented Apr 5, 2017

Hi all! Like @sirlatrom, I tried to deploy a Swarm Mode stack with Elastic (image: "docker.elastic.co/elasticsearch/elasticsearch:5.3.0") It fails due to the IPC_LOCK capability.

@albers

This comment has been minimized.

Copy link
Member

@albers albers commented Apr 5, 2017

@a-jung the link is broken.

@a-jung

This comment has been minimized.

Copy link

@a-jung a-jung commented Apr 5, 2017

@albers Sorry for that. I just pasted the URI which I use in the compose file to pull the image from elastic.co. Installation docs They use cap_add: and - IPC_LOCK in their example file.

@magg

This comment has been minimized.

Copy link

@magg magg commented Apr 13, 2017

any update on this? is there any plans to include the feature in swarm mode.

I'm currently using swarm standalone to deploy my containers with cap_add in a cloud but I'm encountering many issues... swarm mode would ease the pain

please give us an ETA on this

thanks

@azzeddinefaik

This comment has been minimized.

Copy link

@azzeddinefaik azzeddinefaik commented Apr 25, 2017

+1

@morph027

This comment has been minimized.

Copy link

@morph027 morph027 commented Apr 28, 2017

Coming here for Elasticsearch too ... right now i added the ES manually to the attachable swarm network...but this does not scale well ;)

@olljanat

This comment has been minimized.

Copy link
Contributor

@olljanat olljanat commented Jun 11, 2019

Why not —cap-add looks it is in containers ?

@prologic because then switches on service update would be --cap-add-add and --cap-add-rm which is ugly. It is mentioned on old commets/PRs and was biggest reason why original implementation was not approved couple of years ago.

EDIT: link to original comment #26849 (comment)

@BretFisher

This comment has been minimized.

Copy link

@BretFisher BretFisher commented Jun 11, 2019

One of the principles of keeping Docker Swarm simple is that everything is avail in service create, service update and stack yaml. You can expect that a feature is implemented in all three. Teams have reasons for going with services-only or stacks-only, so I'd prefer not to see this diverge from the original goals. Any real-world service command is "ugly" in that it's hundreds of characters and not easily typed, but not every use case can/will use stacks.

My vote is it's in all the commands, or it'll be less useful.

@thaJeztah

This comment has been minimized.

Copy link
Member

@thaJeztah thaJeztah commented Jun 12, 2019

Agreed, it should be in all three.

One thing to look at is;

  • do we want the short-hand options? --capabilities=all (convenient, but easy to shoot-oneself-in-the-foot)
  • how do we handle service update --cap-add=foo / --cap-rm=foo on a service that was started without --capabilities=<custom list>?
    • services without --capabilities set won't have a list of capabilities in the service-spec
    • "diffing" won't be possible, unless the CLI / client requests the active set of capabilities from the service/daemon somehow

Basically; we need to prevent this situation;

User creates a service without setting capabilities; containers will have the default set of capabilities

docker service create --name myservice busybox

User attempts to add a capability

docker service update --cap-add NET_ADMIN myservice

However, the service now ends up having a single capability (NET_ADMIN, nothing else).

Alternatively (very verbose, and less convenient); implement docker service update --capabilities instead, and require the user to always provide the exact list of capabilities that needs to be set, instead of providing "diffing" flags (x-add / x-rm).

  • pro: result will be clear
  • con: very verbose
  • meh: probably ok for services defined in a stack (docker-compose.yml); easy to add/remove a capability by editing the compose file; much less convenient when using the CLI (docker service update); easy to make mistakes there.
@sirlatrom

This comment has been minimized.

Copy link
Contributor

@sirlatrom sirlatrom commented Jun 12, 2019

* pro: result will be clear

* con: very verbose

* meh: probably ok for services defined in a stack (`docker-compose.yml`); easy to add/remove a capability by editing the compose file; much _less_ convenient when using the CLI (`docker service update`); easy to make mistakes there.

Also, how would one distinguish between removing any added capabilities (with --cap-rm) vs. 'resetting' to having not specified any, which arguably should lead to the default capabilities? The same goes for --capabilities, would --capabilities "" mean no capabilities or the default set?

@thaJeztah

This comment has been minimized.

Copy link
Member

@thaJeztah thaJeztah commented Jun 12, 2019

Also, how would one distinguish between removing any added capabilities (with --cap-rm) vs. 'resetting' to having not specified any

Possibly for the CLI we'd need (ugh) magic values (--capabilities=default / --capabilities=none).

One thing to look into is "what are the defaults" (also look at #39297); if defaults can be configured per daemon, the result in a Swarm situation will be unpredictable; a task deployed on one node might get different capabilities than a task deployed on another node.

My ideal would be that the Service spec fully describes the service's capabilities, which means that when creating a service, its spec contains all the capabilities that it has; also in the "default" situation. This would take care of that situation (and take care of situations where the defaults are changed). In addition, if we want configurable defaults; those defaults should be specified at the SwarmKit / manager level, not per daemon (at least when deploying a service).

That would likely be a breaking change though (as in; existing services won't have those values set)

@sirlatrom

This comment has been minimized.

Copy link
Contributor

@sirlatrom sirlatrom commented Jun 12, 2019

In addition, if we want configurable defaults; those defaults should be specified at the SwarmKit / manager level, not per daemon (at least when deploying a service).

That sounds like the best option, and could be implemented by the manager explicitly sending the set of capabilities along with any task, even when the default set is requested (by whatever means that is expressed).

@thaJeztah

This comment has been minimized.

Copy link
Member

@thaJeztah thaJeztah commented Jun 12, 2019

That sounds like the best option, and could be implemented by the manager explicitly sending the set of capabilities along with any task, even when the default set is requested (by whatever means that is expressed).

That's a bit of a grey area; IIRC, there have been some discussions in the past about "altering" the create/update requests server-side. Those boiled down to; an API call to create a service, followed by an API call to inspect that service should produce the same information (baring current 'state' etc.).

I commented similar things on a couple of other PR's; what would (likely) be needed is a way for the client to get the defaults from the manager/daemon, so sequence of events would be something like;

Create a service:

  • fetch defaults
  • apply config set by user to the defaults
  • send create request to the daemon/manager

Update a service

  • fetch current service-spec
  • apply changes set by user
  • send update request to the daemon/manager
@thaJeztah

This comment has been minimized.

Copy link
Member

@thaJeztah thaJeztah commented Jun 12, 2019

/cc @dperny

@sirlatrom

This comment has been minimized.

Copy link
Contributor

@sirlatrom sirlatrom commented Jun 12, 2019

an API call to create a service, followed by an API call to inspect that service should produce the same information (baring current 'state' etc.).

That definitely makes sense. It could still be achieved for the docker service commands by having them build the correct API call, since they also set e.g. replicated mode etc. even if not specified on the CLI. But obviously, existing API clients would be hit by this change, since their requests would not include a capabilities list.

@thaJeztah

This comment has been minimized.

Copy link
Member

@thaJeztah thaJeztah commented Jun 12, 2019

So, the API would be able to distinguish them (it would be either nil (not set), or [] (set, but explicitly empty)), so likely it would be able to fallback to the old behaviour in case of nil.

@sirlatrom

This comment has been minimized.

Copy link
Contributor

@sirlatrom sirlatrom commented Jun 12, 2019

So, the API would be able to distinguish them (it would be either nil (not set), or [] (set, but explicitly empty)), so likely it would be able to fallback to the old behaviour in case of nil.

That's great news, and, I would argue, supports the case for docker service create/update to add the default capabilities (however they are determined) to services in order to ensure the same set of capabilities across the swarm, whereas the API clients that do not specify a set of capabilities would get the current behaviour of using the engines' own default sets.

@olljanat

This comment has been minimized.

Copy link
Contributor

@olljanat olljanat commented Jun 14, 2019

That's a bit of a grey area; IIRC, there have been some discussions in the past about "altering" the create/update requests server-side. Those boiled down to; an API call to create a service, followed by an API call to inspect that service should produce the same information (baring current 'state' etc.).

I commented similar things on a couple of other PR's; what would (likely) be needed is a way for the client to get the defaults from the manager/daemon, so sequence of events would be something like;

Create a service:

  • fetch defaults
  • apply config set by user to the defaults
  • send create request to the daemon/manager

Update a service

  • fetch current service-spec
  • apply changes set by user
  • send update request to the daemon/manager

There is at least my proposal to support changing defaults in swarm level on docker/swarmkit#2794

That sounds like the best option, and could be implemented by the manager explicitly sending the set of capabilities along with any task, even when the default set is requested (by whatever means that is expressed).

That can be done but we need to verify that that it does not break Windows containers.

@burnMyDread

This comment has been minimized.

Copy link

@burnMyDread burnMyDread commented Jun 17, 2019

@thaJeztah @sirlatrom I wanted to follow up with you guys on what the plan is for the capabilities. It sounds like the plan is for client to ask the daemon for the default capabilities then specify the complete list of capabilities? Is that correct?

@thaJeztah

This comment has been minimized.

Copy link
Member

@thaJeztah thaJeztah commented Jun 17, 2019

To be discussed here; currently there's no endpoint in place to pass the defaults to the client, so that would have to be implemented

@tconrado

This comment has been minimized.

Copy link

@tconrado tconrado commented Jul 18, 2019

@tjmehta hello, is possible to have an ETA?

@thaJeztah thaJeztah added this to backlog in maintainers-session Jul 18, 2019
@olljanat

This comment has been minimized.

Copy link
Contributor

@olljanat olljanat commented Jul 18, 2019

@tconrado I have no idea who is @tjmehta or why you did ping him/her here but from my side ETA is still next version. 19.03 looks to be delayed so I assume that there will not be 19.06 so most probably it is 19.09 (which code freeze is on September) and hopefully that is released still during this year

@olljanat

This comment has been minimized.

Copy link
Contributor

@olljanat olljanat commented Sep 10, 2019

Status update: This have been a bit hold during summer but I got now cap_add/cap_drop/privileged settings working with stack using docker/cli#1940 PTAL and provide comments to that. I will create separate PR to provide docker service command flags.

duncan-brown added a commit to duncan-brown/ce-it-infrastructure that referenced this issue Oct 27, 2019
duncan-brown added a commit to cosmic-explorer/ce-it-infrastructure that referenced this issue Oct 27, 2019
@cjdcordeiro

This comment has been minimized.

Copy link

@cjdcordeiro cjdcordeiro commented Oct 30, 2019

@olljanat was the PR created?

@olljanat

This comment has been minimized.

Copy link
Contributor

@olljanat olljanat commented Oct 30, 2019

@cjdcordeiro for command line? Not yet as someone need review docker/cli#1940 first.

@mhemrg

This comment has been minimized.

Copy link

@mhemrg mhemrg commented Oct 31, 2019

@olljanat What about the rest api?

@olljanat

This comment has been minimized.

Copy link
Contributor

@olljanat olljanat commented Oct 31, 2019

@mhemrg merged already. plz look #25885 (comment)

@gpulido

This comment has been minimized.

Copy link

@gpulido gpulido commented Oct 31, 2019

Hello,
In which version of docker it will be available to be used as parameters of the docker compose?
Thanks!

@Vaults

This comment has been minimized.

Copy link

@Vaults Vaults commented Nov 11, 2019

@olljanat Thank you so much for implementing this! I hope they're gonna review the docker/cli#1940 PR soon. I'd be really happy if the command flags were to be implemented :)

@olljanat

This comment has been minimized.

Copy link
Contributor

@olljanat olljanat commented Nov 23, 2019

FYI. This is very unofficial information but I try tell something what I know because people are very eagerly asking about this feature.

Because of Mirantis to acquired Docker Enterprise and some of Docker Inc employees was moved there it is currently very unclear when they are able to get release process working again which why at least I don't know that what will be the next Docker version or when it will be released.

However, whole feature is implemented and works as far I can see so who ever want to test it can do it by downloading latest nightly build of Docker engine (dockerd) from https://master.dockerproject.org and my custom build version of Docker CLI from https://github.com/olljanat/cli/releases/tag/beta1
You can also find usage examples for CLI from docker/cli#2199 and for Stack from docker/cli#1940 If you find bugs from those please leave comment to correct PR. Also notice that syntax might still change during review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
You can’t perform that action at this time.