service update can be blocked by long download

[sblackstone]

If we update a service's image, e.g.

docker service update --image <someurl>

it will first stop the service, then attempt to download the image.

However, our registry is in the US and our docker services are in manila, so depending on the day's routes in Asia, sometimes this can unpredictably become extremely slow..

It would be better if the update would first download the image and then start updating services to minimize the downtime...

[cpuguy83]

Can you paste the output of docker info and docker version?

Thought we fixed something around this.

Thanks!

[sblackstone]

Containers: 107
Running: 0
Paused: 0
Stopped: 107
Images: 18
Server Version: 1.12.2
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 333
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: host bridge overlay null
Swarm: active
NodeID: 5kek26sq72jq2fb63rm3mlh8q
Is Manager: true
ClusterID: f1vzzpday73gjnbjx93r7cady
Managers: 3
Nodes: 3
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Heartbeat Tick: 1
Election Tick: 3
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Node Address: 192.168.119.99
Runtimes: runc
Default Runtime: runc
Security Options:
Kernel Version: 3.16.0-4-amd64
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 494.4 MiB
Name: rtg-mnl-enterprise-manager-001
ID: CA22:CFNP:A2KW:GGMB:XULM:XLRY:YAXZ:ZUZU:FLL6:Q42R:H7M5:NA5L
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: No oom kill disable support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
Labels:
provider=generic
Insecure Registries:
127.0.0.0/8

Client:
Version: 1.12.2
API version: 1.24
Go version: go1.6.3
Git commit: bb80604
Built: Tue Oct 11 17:43:41 2016
OS/Arch: linux/amd64

Server:
Version: 1.12.2
API version: 1.24
Go version: go1.6.3
Git commit: bb80604
Built: Tue Oct 11 17:43:41 2016
OS/Arch: linux/amd64

[aaronlehmann]

This might be a good application of the "rollout mode" concept proposed in moby/swarmkit#1383. We could have a rollout mode that pulls the new image before stopping the old task.

[aaronlehmann]

Or possibly we could just change the behavior. I'm not sure there's a good reason anymore why we shut down the old task before pulling the image. There used to be orchestrator limitations that prevented that, but I think it's possible after moby/swarmkit#1363.