Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[epic] add more options to service create / service update #25303

Open
thaJeztah opened this issue Aug 1, 2016 · 104 comments
Open

[epic] add more options to service create / service update #25303

thaJeztah opened this issue Aug 1, 2016 · 104 comments

Comments

@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Aug 1, 2016

The service create and service update commands do not support all options that docker run / docker create supports. Some options are not implemented yet, whereas other options may either not be implemented (because they don't make sense in the context of a service, or are not portable / cross platform).

We should add more options for services, however instead of blindly copying every option, we should make sure the options are implemented properly, which may require using different names for the options and/or different kind of values.

I tried to create an overview of all options on docker run, and to match them with the docker service create options we currently have; I may have missed some, or made the wrong "translation", so input is welcome here

Status Issue docker run docker service Notes
#27902 --add-host
#27552 (for exec) -a, --attach n/a does not apply to services, as there are multiple containers backing it There may be usecases for this, but design/implementation needs discussion
--blkio-weight
--blkio-weight-device
#25885 --cap-add docker/cli#2663 docker/cli#2687 docker/cli#2709
#25885 --cap-drop docker/cli#2663 docker/cli#2687 docker/cli#2709
--cgroup-parent
--cidfile does not apply to services, as there are multiple containers backing it
--cpu-percent
--cpu-period --limit-cpu --limit-cpu sets a combination of "cpu period" and "cpu quota" see #27958 for the docker run implementation
--cpu-quota --limit-cpu --limit-cpu sets a combination of "cpu period" and "cpu quota" see #27958 for the docker run implementation
--cpu-rt-period
--cpu-rt-runtime
-c, --cpu-shares
--cpus --limit-cpu --limit-cpu sets a combination of "cpu period" and "cpu quota" see #27958 for the docker run implementation
#30477 --cpuset-cpus
--cpuset-mems
-d, --detach -d is the default
--detach-keys No interactive services, so not needed
#24865 / docker/swarmkit#1244 --device
--device-cgroup-rule devices are host specific, so may not make sense
#32602 --device-read-bps devices are host specific, so may not make sense
#32602 --device-read-iops devices are host specific, so may not make sense
#32602 --device-write-bps devices are host specific, so may not make sense
#32602 --device-write-iops devices are host specific, so may not make sense
--disable-content-trust
#24391 --dns PR #27567
#24391 --dns-option --dns-option, --dns-option-add, --dns-option-rm PR #27567
#24391 --dns-search --dns-search, --dns-search-add, --dns-search-rm PR #27567
#29171 --entrypoint
-e, --env -e, --env
#24712 #31595 --env-file PR #24844
--expose
--gpus
#25317 --group-add --group
#27369 --health-cmd
#27369 --health-interval duration
#27369 --health-retries
--health-start-period
#27369 --health-timeout duration
#24877 -h, --hostname
#34529, docker/cli#51 #34639 --init --init PR docker/swarmkit#2350, docker/swarmkit#2652, #36895, #37183, docker/cli#1116, docker/cli#479, docker/cli#1129
#32300 -i, --interactive does not apply to services, as there are multiple containers backing it There may be usecases for this, but design/implementation needs discussion
#24170 / #29816 --ip does not apply to services, as there are multiple containers backing it. Update: possibly useful to set the VIP
#24170 / #29816 --ip6 does not apply to services, as there are multiple containers backing it. Update: possibly useful to set the VIP
--ipc
#31616, docker/cli#414 --isolation PR #34424, docker/cli#426, docker/swarmkit#2342
--kernel-memory Feature is deprecated in the kernel; see #41254, #41252
-l, --label --container-label
--label-file
--link will be resolved through --network-alias
--link-local-ip does not apply to services, as there are multiple containers backing it
--log-driver --log-driver
--log-opt --log-opt
#31092 --mac-address does not apply to services, as there are multiple containers backing it
-m, --memory --limit-memory
--memory-reservation --reserve-memory
🚧 #34654 --memory-swap PR: #37872
🚧 #34654 --memory-swappiness PR: #37872
--mount --mount, --mount-add, --mount-rm
🔳 --name NOTE: --name sets the service name, not the container's name
#28573 --network --network host networking (see #25873) added through #32981.
#28247 -- --network-add/--network-rm are added in docker 17.05 docker/swarmkit#1029
#24787 --network-alias
--no-healthcheck
--oom-kill-disable
🔳 #34703 --oom-score-adj swarmkit PR: docker/swarmkit#2371
docker/swarmkit#1605 --pid
#28618 --pids-limit PR: #39882 swarmkit PR: docker/swarmkit#2415 (vendored: #35326)
--platform
#24862 / docker/swarmkit#1030 --privileged docker/swarmkit#1722
-p, --publish -p, --publish NOTE: does not support <ip-address> (#26696, #32299)
-P, --publish-all when defining a service; explicitly define ports to publish
#30162 --read-only #29972
--restart --restart-condition, --restart-delay, --restart-max-attempts, --restart-window
--rm SwarmKit keeps old tasks (containers) around, but removes them, based on --task-history-limit
--runtime
#25209 -> #41371 --security-opt --credential-spec (#32339) is equivalent for --security opt credentialspec=... SELinux can be set through API (#32339)
#26714 --shm-size Possible through --mount type=tmpfs,target=/dev/shm
--sig-proxy
#25696 --stop-signal PR #30754
--stop-timeout --stop-grace-period New in 1.13 (see #22566)
#28619 --storage-opt
#25209, #31961, moby/libentitlement#35 --sysctl PR #37701, docker/swarmkit#2729, docker/cli#1754
--tmpfs --mount type=tmpfs
#25644 -t, --tty Implemented in SwarmKit docker/swarmkit#1370. Docker PR is #28076
#25209 --ulimit PRs: docker/swarmkit#2967, #41284, docker/cli#2660 docker/cli#2712
#25304 -u, --user -u, --user Does not support group / gid (see #25304 (comment))
#37560 --userns
--uts
-v, --volume --mount UX improvement needed (add -v flag?)
--volume-driver --mount UX improvement needed (add -v flag?)
--volumes-from does not apply to services, as there are multiple containers backing it
-w, --workdir -w, --workdir
@thaJeztah
Copy link
Member Author

@thaJeztah thaJeztah commented Aug 1, 2016

@yongtang
Copy link
Member

@yongtang yongtang commented Aug 1, 2016

@thaJeztah I saw on #25276 and the above list that --net=host is not supported. That could be a very useful option in some deployment scenarios. I am wondering if there are any technical challenges that prevent the usage of --net=host?

@yongtang
Copy link
Member

@yongtang yongtang commented Aug 1, 2016

@thaJeztah For --mount there is an open issue related: #25253

@thaJeztah
Copy link
Member Author

@thaJeztah thaJeztah commented Aug 1, 2016

@yongtang I know that for some options there's additional discussion needed; e.g. --network=host can have some side effects (how does it work in combination with load-balancing? also --network=host can be considered insecure)

@yongtang
Copy link
Member

@yongtang yongtang commented Aug 1, 2016

Thanks @thaJeztah for the explanation.

@justincormack
Copy link
Contributor

@justincormack justincormack commented Aug 3, 2016

@thaJeztah --ip and --ip6 don't make sense either as there are multiple containers.

@thaJeztah
Copy link
Member Author

@thaJeztah thaJeztah commented Aug 3, 2016

Good call, I'll edit

@stevvooe
Copy link
Contributor

@stevvooe stevvooe commented Aug 3, 2016

  • -i is not necessary.
  • --groud-add will be --group.
  • --link and --link-local-ip are never going to be implemented for services.
  • --publish-all doesn't make sense.
  • --cidfile doesn't make sense
  • --volumes-from is never going to happen
@thaJeztah
Copy link
Member Author

@thaJeztah thaJeztah commented Aug 3, 2016

@stevvooe agreed on most; in doubt on --publish-all, although I can see it possibly being difficult to implement (ports aren't known until the image was pulled / inspected, perhaps that's more difficult in Swarm mode networking)

@thaJeztah
Copy link
Member Author

@thaJeztah thaJeztah commented Aug 3, 2016

Thinking about --link - providing that may allow for container/service scoped aliasing; but we should analyze use-cases and see if it makes sense

@stevvooe
Copy link
Contributor

@stevvooe stevvooe commented Aug 3, 2016

@stevvooe agreed on most; in doubt on --publish-all, although I can see it possibly being difficult to implement (ports aren't known until the image was pulled / inspected, perhaps that's more difficult in Swarm mode networking)

This just doesn't make sense. For the most part, ports are already exported on the container's IP. In swarm services, this would bind all ports across the cluster, which isn't useful. The other issue is that the manager has no idea what these ports actually are, without inspecting the image. Exported cluster ports should be considered individually and carefully.

Thinking about --link - providing that may allow for container/service scoped aliasing; but we should analyze use-cases and see if it makes sense

This is provided by overlay networking and network aliasing. In general, --link should be considered deprecated. Interdependent containers don't yield reliable architectures. Dependencies should be resolved before creation (based on a rendevouz value or service name/alias).

@thaJeztah
Copy link
Member Author

@thaJeztah thaJeztah commented Aug 3, 2016

Yes, implementing --network-alias will probably solve most use cases; we need to think of a UX for that, perhaps that may require a syntax similar to --mount to provide per-network options (see #24787 (comment))

@hvpareja
Copy link

@hvpareja hvpareja commented Aug 4, 2016

I've also seen the options --env-add and --env-rm in docker service update. They replace --env flag. They seems to be missing in your table.

@thaJeztah
Copy link
Member Author

@thaJeztah thaJeztah commented Aug 4, 2016

@hvpareja hm, yes, service update has different flags to add/remove options for flags that can be provided multiple times, not sure if we need to include that here, but thanks for the suggestion

@kyhavlov
Copy link

@kyhavlov kyhavlov commented Aug 5, 2016

+1 for being able to set net=host for services, that would be very useful for high performance things that don't want the docker network overhead

@tzz
Copy link

@tzz tzz commented Aug 8, 2016

In our environment we'd like to use --add-host, is that a possibility? Thank you!

@stevvooe
Copy link
Contributor

@stevvooe stevvooe commented Aug 8, 2016

@tzz How do you use --add-host in your environment? Do you use it to route to external services or internal services?

The reason I ask is we are curious if this belongs on the network, as part of a service or requires a separate abstraction to manager to external service discovery. @mrjana

@tzz
Copy link

@tzz tzz commented Aug 9, 2016

@stevvooe internal services with external visibility. Our problem is that service.wanted resolves in DNS to an external IP address, which is appropriate for some connections, but other connections need the internal IP address for the same service to avoid NAT and routing. Other possibilities for us are dnsmasq and keeping both addresses in DNS, but either way that's a lot more work than --add-host and less dynamic at service creation time. I hope that's helpful.

@stevvooe
Copy link
Contributor

@stevvooe stevvooe commented Aug 10, 2016

@tzz Thanks for the explanation!

Would it be correct to say that you want this at the cluster, network or service-level? For example, will have different values for service.wanted depending if you are in different services?

@tzz
Copy link

@tzz tzz commented Aug 10, 2016

@stevvooe for stable environments (production) the lookup can be stable at the cluster or network level. That's our most important use case. For unstable (dev and QA) environments we may need to change the lookup at the service level more dynamically, but that's a less important use case for us.

@datamattsson
Copy link

@datamattsson datamattsson commented Aug 12, 2016

I'm pulling my hair that docker service won't allow -t. Certain apps can't live without a TTY.

@stevvooe
Copy link
Contributor

@stevvooe stevvooe commented Aug 12, 2016

@drajen Could you file a bug on SwarmKit?

Out of curiosity, what applications are you running as services that require a TTY?

@datamattsson
Copy link

@datamattsson datamattsson commented Aug 12, 2016

@stevvooe I was running Samba in foreground mode which apparently requires a tty. Check the samba-publicshare image on docker hub. You may easily write a wrapper around it but there might be more breakage out there.

@stevvooe
Copy link
Contributor

@stevvooe stevvooe commented Aug 12, 2016

@drajen That is a bummer. I'm really sorry we missed that. I wish someone spoke up in the RC period.

I've submitted docker/swarmkit#1370 and docker/engine-api#364 for service support. We'll have to follow that up with the analogous flags in docker service.

@datamattsson
Copy link

@datamattsson datamattsson commented Aug 13, 2016

@stevvooe thanks for giving this attention!

@mbentley
Copy link
Contributor

@mbentley mbentley commented Mar 28, 2018

I haven't see much action on this issue but another thing that I have not seen solved yet from classic swarm is the ability to schedule containers with affinities. Not sure if that is out of scope of this epic or not but that is a bit of a gap in being able to have one process per container in Swarm mode.

@malkir
Copy link

@malkir malkir commented Mar 29, 2018

Please consider prioritizing adding a --sysctl flag to Docker Service. It being missing nullifies the use of Swarm in high performance environments when having to, for example, wait 60s for a TIME_WAIT connection to clean out. Tens of thousands of TIME_WAIT connections can exist within an individual container and it severely affects performance. A modification of net.ipv4.tcp_fin_timeout or net.ipv4.tcp_tw_reuse on the host network namespace is not adequate when it makes no affect to the containers themselves. Falling back to docker run is not ideal, as we lose all the benefits of service and swarm.

@sirlatrom
Copy link
Contributor

@sirlatrom sirlatrom commented May 14, 2018

I'm looking forward to the day I can no longer create a distributed fork bomb using docker service create bash -c ':(){ :|:& };:; wait'.

@jefflill
Copy link

@jefflill jefflill commented Jul 30, 2018

--userns for docker service create and update would be useful.

I'm trying to use userns-remap to secure my cluster containers and services transparently by default but I have a few services that need to mount files from the Docker host and I was hoping to use docker service create --userns=host ... for these few services but there doesn't appear to be a way to do this using the normal command line. I suppose I could deploy a stack but I'd rather not because it mangles the service name to be stack-service instead of just service.

I was hoping to use --user=ID to explicitly run the service with a specific host user but this didn't work (I don't think this overrides userns-remap.

@thaJeztah
Copy link
Member Author

@thaJeztah thaJeztah commented Jul 30, 2018

@jefflill could you open a separate issue for that, then I'll link it from the table at the top

@jefflill
Copy link

@jefflill jefflill commented Jul 30, 2018

@thaJeztah: Done. Thanks.

@hholst80
Copy link

@hholst80 hholst80 commented Mar 4, 2019

What extra information is needed for the implementation of --interactive? Why cannot the implementation used by docker-compose be used here (stdin_keep: true)?

We need this to work efficiently with discrete resources like GPUs.

@thaJeztah
Copy link
Member Author

@thaJeztah thaJeztah commented Mar 4, 2019

What extra information is needed for the implementation of --interactive? Why cannot the implementation used by docker-compose be used here (stdin_keep: true)?

When running a swarm service, the container itself can run on any node in the swarm cluster; the API won't be able to connect to a container that's on any other host than the manager that is connected to.

@ahmadfarisfs
Copy link

@ahmadfarisfs ahmadfarisfs commented Jun 11, 2020

I think adding --env-file options to docker service update should be fine. Right now it's only on docker service create

@thaJeztah
Copy link
Member Author

@thaJeztah thaJeztah commented Jun 11, 2020

@ahmadfarisfs better to discuss on #31595, which also lists some of the reasons it's more problematic for docker service update.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
21.x Planning
  
To do
Linked pull requests

Successfully merging a pull request may close this issue.

None yet