docker stack deploy does not report missing images

[eyz]

Description

Steps to reproduce the issue:

1. Deploy a stack from images which no longer (or never) existed on the designated registry

Describe the results you received:
The Docker CLI command stack deploy gives no indication that the images do not exist, and leaves service tasks at 0/X started with no indication of underlying problem. The service logs do not explain the underlying issue either.

Describe the results you expected:
docker stack deploy -c should receive a message from the underlying Docker Engines that the images cannot be pulled, and thus the operation is blocked until the image exists

Additional information you deem important (e.g. issue happens only occasionally):
Here is output that shows the docker service logs have no pertinent output -

$ export DOCKER_HOST=tcp:IP:2375
$ docker stack deploy --with-registry-auth -c FILE.yml
$ docker service ls -q
zht30tzql7xo
d0rtfw5gcv57
ogu9irz557d6
uuqexaegfibt
$ docker service logs zh
$ docker service logs d0
$ docker service logs og
$ docker service logs uu
$

ID                  NAME                     MODE                REPLICAS            IMAGE                                                   PORTS
zht30tzql7xo        redacted_test_service_a  replicated          0/3                 registry.MYDOMAIN:5000/redacted_test_service_a:347d034  *:8501->8500/tcp
d0rtfw5gcv57        redacted_test_service_b  replicated          0/3                 registry.MYDOMAIN:5000/redacted_test_service_b:347d034  *:8198->8998/tcp,*:8199->8999/tcp
ogu9irz557d6        redacted_test_service_c  replicated          0/3                 registry.MYDOMAIN:5000/redacted_test_service_c:347d034  *:8298->8998/tcp,*:8299->8999/tcp
uuqexaegfibt        redacted_test_service_d  replicated          0/0                 registry.MYDOMAIN:5000/redacted_test_service_d:347d034
 

Output of docker version:

Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:42:45 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   afdb6d44a8
 Built:        Wed Sep 27 13:05:49 2017
 OS/Arch:      linux/amd64
 Experimental: true

Output of docker info:

Containers: 2
 Running: 1
 Paused: 0
 Stopped: 1
Images: 4
Server Version: 17.09.0-ce
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: vlpem14ayl9gxmx5fmxq5bw0h
 Is Manager: true
 ClusterID: mtdknk21b5rd9msmw40xzeloa
 Managers: 3
 Nodes: 3
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Autolock Managers: false
 Root Rotation In Progress: false
 Node Address: 10.210.102.211
 Manager Addresses:
  10.210.102.211:2377
  10.210.102.212:2377
  10.210.102.213:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.51-linuxkit
Operating System: Alpine Linux v3.6 (containerized)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.787GiB
Name: linuxkittest1.MYDOMAIN.com
ID: D7YE:ZJPI:ICBY:KEIC:4OGX:Q4NZ:DPRN:SSK2:7YP3:X4E7:SJ6P:HXLT
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 55
 Goroutines: 212
 System Time: 2017-09-27T21:44:18.133551864Z
 EventsListeners: 1
Username: eyz77
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
Registry Mirrors:
 http://registry.MYDOMAIN.com:5000/
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):
I am connected to a remote docker-ce 17.09.0 dind on LinuxKit via export DOCKER_HOST=tcp://IP:2375. I suspect the implementation of Docker Engine is not the cause, however.

[sylvainmouquet]

A solution is to set replica to 0 in the yml, to run the docker stack deploy and to change the replica value with docker service update with option --detach=false to get the status

docker service update --replicas=5 <service_name> --detach=false

[thaJeztah]

Yes, the general behavior is by design; i.e., a service can be deployed, and docker will start a reconciliation loop to converge the service state with the desired state. An image that is not present may become present at some point (e.g., the image is being pushed, etc). The same can apply for (e.g.) resources not being available yet (port, memory), and the service will automatically be deployed once that condition is met (a node is added, or the port becomes available).

Having said the above; the feedback can definitely be improved, which was the reason for adding the interactive (--detach=false) option for docker service commands; this will be the default mode in a future release. There is no such option yet for docker stack deploy, but there's an issue tracking this effort; docker/cli#373 (continued from #32367).

Meanwhile, you can use the docker service ps command to verify the status of tasks that are being deployed; see the example below:

cat > docker-compose.yml <<EOF
version: '3.3'
services:
  something:
    image: nosuchimage:nosuchtag
EOF

$ docker stack deploy -c docker-compose.yml repro-35003

$ docker service ps repro-35003_something

ID                  NAME                          IMAGE                   NODE                DESIRED STATE       CURRENT STATE             ERROR                              PORTS
hgzm9iefqx2s        repro-35003_something.1       nosuchimage:nosuchtag   moby                Ready               Rejected 3 seconds ago    "No such image: nosuchimage:no…"
s2jimp8k511s         \_ repro-35003_something.1   nosuchimage:nosuchtag   moby                Shutdown            Rejected 7 seconds ago    "No such image: nosuchimage:no…"
rlb8xhnaajqn         \_ repro-35003_something.1   nosuchimage:nosuchtag   moby                Shutdown            Rejected 13 seconds ago   "No such image: nosuchimage:no…"
ub22ox3lunp1         \_ repro-35003_something.1   nosuchimage:nosuchtag   moby                Shutdown            Rejected 18 seconds ago   "No such image: nosuchimage:no…"
ugvyhviervyb         \_ repro-35003_something.1   nosuchimage:nosuchtag   moby                Shutdown            Rejected 19 seconds ago   "No such image: nosuchimage:no…"

I'm closing this issue, because we're tracking this effort through docker/cli#373, but I invite you to participate in that discussion 👍

[eyz]

Thanks. I didn't check service ps

[kasir-barati]

thanks @thaJeztah. your comment help me a lot. But i do not see any --detach flag for docker stack deploy! can you help me. I have the same problem with docker stack deploy.

[thaJeztah]

Correct, it's not there for docker stack deploy yet; see my previous comment;

There is no such option yet for docker stack deploy, but there's an issue tracking this effort; docker/cli#373 (continued from #32367).