swarm: needless update with --resolve-image changed

[zarnovican]

Description

Stack deploy with --resolve-image changed will update the service even if the image/tag has not changed.

The first docker stack deploy.. after tag change will update the service as expected. The next deploy will update the service again, even if the tag has not changed. Any subsequent deploy won't update the service (as expected).

Steps to reproduce the issue:

1. create stack config test.yaml

version: "3.3"

services:

    redis:
        image: redis:3.2.8

2. deploy stack (first deploy)

docker stack deploy -c test.yaml --resolve-image changed test

3. deploy same config again (second deploy)

docker stack deploy -c test.yaml --resolve-image changed test

Describe the results you received:

In step 3, service redis is re-deployed.

$ docker service ps test_redis
ID                  NAME                IMAGE               NODE                DESIRED STATE       CURRENT STATE             ERROR               PORTS
s6f5clqoq9jv        test_redis.1        redis:3.2.8         ip-10-72-178-242    Running             Running 42 seconds ago
kg95b9zje4x2         \_ test_redis.1    redis:3.2.8         ip-10-72-178-242    Shutdown            Shutdown 45 seconds ago

Describe the results you expected:

Service should not re-deploy, since the tag has not changed.

Additional information you deem important (e.g. issue happens only occasionally):

Presumably, it is because on first deploy (and any tag change), Docker will make a lookup to registry and store image id with sha256 hash. After the second deploy, you can observe that the current image is without the sha256, while the previous spec has it..

         "Spec": {
            "Name": "test_redis",
            "TaskTemplate": {
                "ContainerSpec": {
                    "Image": "redis:3.2.8",
...
        "PreviousSpec": {
            "Name": "test_redis",
            "TaskTemplate": {
                "ContainerSpec": {
                    "Image": "redis:3.2.8@sha256:8eb0cd22644dc1218680db876411f45baac493ac9a0bfd391d6efdef0767ea6b",

Subsequent deploy will not make that lookup and it will asume the image is redis:3.2.8. Which will not trigger any update.

Output of docker version:

Client:
 Version:      17.06.0-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:23:31 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.06.0-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:19:04 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 7
 Running: 2
 Paused: 0
 Stopped: 5
Images: 3
Server Version: 17.06.0-ce
Storage Driver: aufs
 Root Dir: /mnt/docker/aufs
 Backing Filesystem: extfs
 Dirs: 47
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: p13y70uqdo83gzma54cha3aua
 Is Manager: true
 ClusterID: km78r0i9lti4ihw7y4clj6pn9
 Managers: 3
 Nodes: 3
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Root Rotation In Progress: false
 Node Address: 10.73.190.96
 Manager Addresses:
  10.72.178.242:2377
  10.73.190.96:2377
  10.76.153.210:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfb82a876ecc11b5ca0977d1733adbe58599088a
runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-1020-aws
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.611GiB
Name: ip-10-73-190-96
ID: PZGW:KBDE:X3UR:U43Z:R74J:LCGE:JYQZ:SEJ2:IYYU:U2GF:O5LW:PYNQ
Docker Root Dir: /mnt/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):
AWS

[thaJeztah]

Confirmed, I can reproduce this on;

Client:
 Version:      17.06.1-ce-rc1
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   77b4dce
 Built:        Fri Jul 14 07:38:15 2017
 OS/Arch:      darwin/amd64

Server:
 Version:      17.06.1-ce-rc1
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   77b4dce
 Built:        Fri Jul 14 07:33:35 2017
 OS/Arch:      linux/amd64
 Experimental: true

$ docker stack deploy -c docker-compose.yml repro-34242
Creating network repro-34242_default
Creating service repro-34242_redis

$ docker stack ps repro-34242
ID                  NAME                  IMAGE               NODE                DESIRED STATE       CURRENT STATE           ERROR               PORTS
zf82fdczud3f        repro-34242_redis.1   redis:3.2.8         moby                Running             Running 6 seconds ago

Re-deploy the stack

$ docker stack deploy -c docker-compose.yml --resolve-image changed repro-34242
Updating service repro-34242_redis (id: mh3im046fem2l4u4x9vrwxfvb)

$ docker stack ps repro-34242

ID                  NAME                      IMAGE               NODE                DESIRED STATE       CURRENT STATE             ERROR               PORTS
s0y54fj3zpyo        repro-34242_redis.1       redis:3.2.8         moby                Running             Running 8 seconds ago
zf82fdczud3f         \_ repro-34242_redis.1   redis:3.2.8         moby                Shutdown            Shutdown 11 seconds ago

Compare Spec <--> PreviousSpec

$ docker service inspect repro-34242_redis -f '{{.Spec.TaskTemplate.ContainerSpec.Image}} <--> {{.PreviousSpec.TaskTemplate.ContainerSpec.Image}}'
redis:3.2.8 <--> redis:3.2.8@sha256:8eb0cd22644dc1218680db876411f45baac493ac9a0bfd391d6efdef0767ea6b

Let me open a pull request to fix this

[zarnovican]

Not resolved. 👎

Hi, @thaJeztah. I have re-tested this issue on 17.09 and 17.12 and both are still affected.

Diffing the task template Spec vs PreviousSpec, I can see difference in..

                "Placement": {
                    "Platforms": [
                        {
                            "Architecture": "amd64",
                            "OS": "linux"
                        }
                    ]
                },

.. which is mentioned in docker/cli#386 as fixed. Even if that PR was merged to docker/cli master in Aug 2017, it is still not in latest docker-ce ubuntu package.. I'm confused 😕

$ docker version
Client:
 Version:      17.09.1-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   19e2cf6
 Built:        Thu Dec  7 22:24:23 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.09.1-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   19e2cf6
 Built:        Thu Dec  7 22:23:00 2017
 OS/Arch:      linux/amd64
 Experimental: false

$ docker version
Client:
 Version:	17.12.0-ce
 API version:	1.35
 Go version:	go1.9.2
 Git commit:	c97c6d6
 Built:	Wed Dec 27 20:11:19 2017
 OS/Arch:	linux/amd64

Server:
 Engine:
  Version:	17.12.0-ce
  API version:	1.35 (minimum version 1.12)
  Go version:	go1.9.2
  Git commit:	c97c6d6
  Built:	Wed Dec 27 20:09:53 2017
  OS/Arch:	linux/amd64
  Experimental:	false

[thaJeztah]

@zarnovican the platform part was removed from that patch, only the image (redis:3.2.8 <--> redis:3.2.8@sha256:8eb0cd22644dc1218680db876411f45baac493ac9a0bfd391d6efdef0767ea6b) change; see docker/cli#386 (comment), and #34268 which was opened for that

[zarnovican]

I'm not able to reproduce this issue on 18.03 (even if #34268 is still open). I'll consider it resolved. Thanks.