New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[epic] add more options to `service create` / `service update` #25303

Open
thaJeztah opened this Issue Aug 1, 2016 · 100 comments

Comments

Projects
None yet
@thaJeztah
Member

thaJeztah commented Aug 1, 2016

The service create and service update commands do not support all options that docker run / docker create supports. Some options are not implemented yet, whereas other options may either not be implemented (because they don't make sense in the context of a service, or are not portable / cross platform).

We should add more options for services, however instead of blindly copying every option, we should make sure the options are implemented properly, which may require using different names for the options and/or different kind of values.

I tried to create an overview of all options on docker run, and to match them with the docker service create options we currently have; I may have missed some, or made the wrong "translation", so input is welcome here

Status Issue docker run docker service Notes
#27902 --add-host
#27552 (for exec) -a, --attach n/a does not apply to services, as there are multiple containers backing it There may be usecases for this, but design/implementation needs discussion
--blkio-weight
--blkio-weight-device
#25885 --cap-add
--cap-drop
--cgroup-parent
--cidfile does not apply to services, as there are multiple containers backing it
--cpu-percent
--cpu-period --limit-cpu --limit-cpu sets a combination of "cpu period" and "cpu quota" see #27958 for the docker run implementation
--cpu-quota --limit-cpu --limit-cpu sets a combination of "cpu period" and "cpu quota" see #27958 for the docker run implementation
--cpu-rt-period
--cpu-rt-runtime
-c, --cpu-shares
--cpus --limit-cpu --limit-cpu sets a combination of "cpu period" and "cpu quota" see #27958 for the docker run implementation
#30477 --cpuset-cpus
--cpuset-mems
-d, --detach -d is the default
--detach-keys No interactive services, so not needed
#24865 / docker/swarmkit#1244 --device
--device-cgroup-rule devices are host specific, so may not make sense
#32602 --device-read-bps devices are host specific, so may not make sense
#32602 --device-read-iops devices are host specific, so may not make sense
#32602 --device-write-bps devices are host specific, so may not make sense
#32602 --device-write-iops devices are host specific, so may not make sense
--disable-content-trust
#24391 --dns PR #27567
#24391 --dns-option --dns-option, --dns-option-add, --dns-option-rm PR #27567
#24391 --dns-search --dns-search, --dns-search-add, --dns-search-rm PR #27567
#29171 --entrypoint
-e, --env -e, --env
#24712 #31595 --env-file PR #24844
--expose
#25317 --group-add --group
#27369 --health-cmd
#27369 --health-interval duration
#27369 --health-retries
--health-start-period
#27369 --health-timeout duration
#24877 -h, --hostname
#34529, docker/cli#51 #34639 --init --init PR docker/swarmkit#2350, docker/swarmkit#2652, #36895, #37183, docker/cli#1116, docker/cli#479, docker/cli#1129
#32300 -i, --interactive does not apply to services, as there are multiple containers backing it There may be usecases for this, but design/implementation needs discussion
#24170 / #29816 --ip does not apply to services, as there are multiple containers backing it. Update: possibly useful to set the VIP
#24170 / #29816 --ip6 does not apply to services, as there are multiple containers backing it. Update: possibly useful to set the VIP
--ipc
#31616, docker/cli#414 --isolation PR #34424, docker/cli#426, docker/swarmkit#2342
--kernel-memory
-l, --label --container-label
--label-file
--link will be resolved through --network-alias
--link-local-ip does not apply to services, as there are multiple containers backing it
--log-driver --log-driver
--log-opt --log-opt
#31092 --mac-address does not apply to services, as there are multiple containers backing it
-m, --memory --limit-memory
--memory-reservation --reserve-memory
#34654 --memory-swap PR: #37872
#34654 --memory-swappiness PR: #37872
--mount --mount, --mount-add, --mount-rm
🔳 --name NOTE: --name sets the service name, not the container's name
#28573 --network --network host networking (see #25873) added through #32981.
#28247 -- --network-add/--network-rm are added in docker 17.05 docker/swarmkit#1029
#24787 --network-alias
--no-healthcheck
--oom-kill-disable
🔳 #34703 --oom-score-adj swarmkit PR: docker/swarmkit#2371
docker/swarmkit#1605 --pid
#28618 --pids-limit
--platform
#24862 / docker/swarmkit#1030 --privileged docker/swarmkit#1722
-p, --publish -p, --publish NOTE: does not support <ip-address> (#26696, #32299)
-P, --publish-all when defining a service; explicitly define ports to publish
#30162 --read-only #29972
--restart --restart-condition, --restart-delay, --restart-max-attempts, --restart-window
--rm SwarmKit keeps old tasks (containers) around, but removes them, based on --task-history-limit
--runtime
#25209 --security-opt --credential-spec (#32339) is equivalent for --security opt credentialspec=... SELinux can be set through API (#32339)
#26714 --shm-size Possible through --mount type=tmpfs,target=/dev/shm
--sig-proxy
#25696 --stop-signal PR #30754
--stop-timeout --stop-grace-period New in 1.13 (see #22566)
#28619 --storage-opt
🚧 #25209, #31961, moby/libentitlement#35 --sysctl PR #37701, docker/swarmkit#2729
--tmpfs --mount type=tmpfs
#25644 -t, --tty Implemented in SwarmKit docker/swarmkit#1370. Docker PR is #28076
#25209 --ulimit
#25304 -u, --user -u, --user Does not support group / gid (see #25304 (comment))
#37560 --userns
--uts
-v, --volume --mount UX improvement needed (add -v flag?)
--volume-driver --mount UX improvement needed (add -v flag?)
--volumes-from does not apply to services, as there are multiple containers backing it
-w, --workdir -w, --workdir
@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Aug 1, 2016

@yongtang

This comment has been minimized.

Member

yongtang commented Aug 1, 2016

@thaJeztah I saw on #25276 and the above list that --net=host is not supported. That could be a very useful option in some deployment scenarios. I am wondering if there are any technical challenges that prevent the usage of --net=host?

@yongtang

This comment has been minimized.

Member

yongtang commented Aug 1, 2016

@thaJeztah For --mount there is an open issue related: #25253

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Aug 1, 2016

@yongtang I know that for some options there's additional discussion needed; e.g. --network=host can have some side effects (how does it work in combination with load-balancing? also --network=host can be considered insecure)

@yongtang

This comment has been minimized.

Member

yongtang commented Aug 1, 2016

Thanks @thaJeztah for the explanation.

@justincormack

This comment has been minimized.

Contributor

justincormack commented Aug 3, 2016

@thaJeztah --ip and --ip6 don't make sense either as there are multiple containers.

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Aug 3, 2016

Good call, I'll edit

@stevvooe

This comment has been minimized.

Contributor

stevvooe commented Aug 3, 2016

  • -i is not necessary.
  • --groud-add will be --group.
  • --link and --link-local-ip are never going to be implemented for services.
  • --publish-all doesn't make sense.
  • --cidfile doesn't make sense
  • --volumes-from is never going to happen
@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Aug 3, 2016

@stevvooe agreed on most; in doubt on --publish-all, although I can see it possibly being difficult to implement (ports aren't known until the image was pulled / inspected, perhaps that's more difficult in Swarm mode networking)

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Aug 3, 2016

Thinking about --link - providing that may allow for container/service scoped aliasing; but we should analyze use-cases and see if it makes sense

@stevvooe

This comment has been minimized.

Contributor

stevvooe commented Aug 3, 2016

@stevvooe agreed on most; in doubt on --publish-all, although I can see it possibly being difficult to implement (ports aren't known until the image was pulled / inspected, perhaps that's more difficult in Swarm mode networking)

This just doesn't make sense. For the most part, ports are already exported on the container's IP. In swarm services, this would bind all ports across the cluster, which isn't useful. The other issue is that the manager has no idea what these ports actually are, without inspecting the image. Exported cluster ports should be considered individually and carefully.

Thinking about --link - providing that may allow for container/service scoped aliasing; but we should analyze use-cases and see if it makes sense

This is provided by overlay networking and network aliasing. In general, --link should be considered deprecated. Interdependent containers don't yield reliable architectures. Dependencies should be resolved before creation (based on a rendevouz value or service name/alias).

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Aug 3, 2016

Yes, implementing --network-alias will probably solve most use cases; we need to think of a UX for that, perhaps that may require a syntax similar to --mount to provide per-network options (see #24787 (comment))

@hvpareja

This comment has been minimized.

hvpareja commented Aug 4, 2016

I've also seen the options --env-add and --env-rm in docker service update. They replace --env flag. They seems to be missing in your table.

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Aug 4, 2016

@hvpareja hm, yes, service update has different flags to add/remove options for flags that can be provided multiple times, not sure if we need to include that here, but thanks for the suggestion

@kyhavlov

This comment has been minimized.

kyhavlov commented Aug 5, 2016

+1 for being able to set net=host for services, that would be very useful for high performance things that don't want the docker network overhead

@tzz

This comment has been minimized.

tzz commented Aug 8, 2016

In our environment we'd like to use --add-host, is that a possibility? Thank you!

@stevvooe

This comment has been minimized.

Contributor

stevvooe commented Aug 8, 2016

@tzz How do you use --add-host in your environment? Do you use it to route to external services or internal services?

The reason I ask is we are curious if this belongs on the network, as part of a service or requires a separate abstraction to manager to external service discovery. @mrjana

@tzz

This comment has been minimized.

tzz commented Aug 9, 2016

@stevvooe internal services with external visibility. Our problem is that service.wanted resolves in DNS to an external IP address, which is appropriate for some connections, but other connections need the internal IP address for the same service to avoid NAT and routing. Other possibilities for us are dnsmasq and keeping both addresses in DNS, but either way that's a lot more work than --add-host and less dynamic at service creation time. I hope that's helpful.

@stevvooe

This comment has been minimized.

Contributor

stevvooe commented Aug 10, 2016

@tzz Thanks for the explanation!

Would it be correct to say that you want this at the cluster, network or service-level? For example, will have different values for service.wanted depending if you are in different services?

@tzz

This comment has been minimized.

tzz commented Aug 10, 2016

@stevvooe for stable environments (production) the lookup can be stable at the cluster or network level. That's our most important use case. For unstable (dev and QA) environments we may need to change the lookup at the service level more dynamically, but that's a less important use case for us.

@drajen

This comment has been minimized.

drajen commented Aug 12, 2016

I'm pulling my hair that docker service won't allow -t. Certain apps can't live without a TTY.

@stevvooe

This comment has been minimized.

Contributor

stevvooe commented Aug 12, 2016

@drajen Could you file a bug on SwarmKit?

Out of curiosity, what applications are you running as services that require a TTY?

@drajen

This comment has been minimized.

drajen commented Aug 12, 2016

@stevvooe I was running Samba in foreground mode which apparently requires a tty. Check the samba-publicshare image on docker hub. You may easily write a wrapper around it but there might be more breakage out there.

@stevvooe

This comment has been minimized.

Contributor

stevvooe commented Aug 12, 2016

@drajen That is a bummer. I'm really sorry we missed that. I wish someone spoke up in the RC period.

I've submitted docker/swarmkit#1370 and docker/engine-api#364 for service support. We'll have to follow that up with the analogous flags in docker service.

@drajen

This comment has been minimized.

drajen commented Aug 13, 2016

@stevvooe thanks for giving this attention!

@seongkki

This comment has been minimized.

seongkki commented Jul 5, 2017

Any news on #31595?

@pascalandy

This comment has been minimized.

pascalandy commented Aug 30, 2017

I can’t stop thinking about the Zero-downtime deployments with rolling upgrades topic. For me Moby would be basically perfect with this feature working (+ a stable stateful solution. I count on the Infinit team for this.)

What’s the game plan to empower users to do this?

EDIT: the flag exist! see #30321 (comment)

Cheers!
Pascal

@daixiang0

This comment has been minimized.

daixiang0 commented Nov 13, 2017

Could add --device-cgroup-rule option for service?

@carlfischer1

This comment has been minimized.

carlfischer1 commented Dec 11, 2017

@thaJeztah housekeeping: --isolation is implemented

@mbentley

This comment has been minimized.

Contributor

mbentley commented Mar 28, 2018

I haven't see much action on this issue but another thing that I have not seen solved yet from classic swarm is the ability to schedule containers with affinities. Not sure if that is out of scope of this epic or not but that is a bit of a gap in being able to have one process per container in Swarm mode.

@malkir

This comment has been minimized.

malkir commented Mar 29, 2018

Please consider prioritizing adding a --sysctl flag to Docker Service. It being missing nullifies the use of Swarm in high performance environments when having to, for example, wait 60s for a TIME_WAIT connection to clean out. Tens of thousands of TIME_WAIT connections can exist within an individual container and it severely affects performance. A modification of net.ipv4.tcp_fin_timeout or net.ipv4.tcp_tw_reuse on the host network namespace is not adequate when it makes no affect to the containers themselves. Falling back to docker run is not ideal, as we lose all the benefits of service and swarm.

@sirlatrom

This comment has been minimized.

sirlatrom commented May 14, 2018

I'm looking forward to the day I can no longer create a distributed fork bomb using docker service create bash -c ':(){ :|:& };:; wait'.

@jefflill

This comment has been minimized.

jefflill commented Jul 30, 2018

--userns for docker service create and update would be useful.

I'm trying to use userns-remap to secure my cluster containers and services transparently by default but I have a few services that need to mount files from the Docker host and I was hoping to use docker service create --userns=host ... for these few services but there doesn't appear to be a way to do this using the normal command line. I suppose I could deploy a stack but I'd rather not because it mangles the service name to be stack-service instead of just service.

I was hoping to use --user=ID to explicitly run the service with a specific host user but this didn't work (I don't think this overrides userns-remap.

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Jul 30, 2018

@jefflill could you open a separate issue for that, then I'll link it from the table at the top

@jefflill

This comment has been minimized.

jefflill commented Jul 30, 2018

@thaJeztah: Done. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment