Starting swarm services results in images with <none> tag

[thaJeztah]

Possibly just a "nit"; but now that services "pin" an image by digest, the pull also pulls by digest, resulting in local images to not have a "tag" visible;

before creating the service; no images are available;

$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE

create a service using the nginx:alpine image

$ docker service create --name web nginx:alpine
x72opmu78pv8cklr7fu4j2w18

After the service is created, the nginx image is present locally, but tag is empty (<none>);

$ docker images

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
nginx               <none>              d964ab5d0abe        5 hours ago         54.9 MB

Not sure this can be solved easily, but I can see this being confusing.

/cc @nishanttotla @aaronlehmann

[aaronlehmann]

Yeah, pull by digest won't create a tag.

If we were to change docker pull image:tag@digest to create a tag as well, it would automatically fix this services use case. However there are some complexities here - Docker's interpretation of references currently throws away the tag if there's a digest present, and fixing this has some implications throughout the code base and wasn't planned for 1.13. Also IIRC the way pull is implemented in Docker's HTTP API has a nonstandard way of specifying the tag or digest, and we might have to make an API change to make it possible to pass both at the same time.

A possible workaround would be to make the swarm executor create the tag after pulling by digest if one does not exist, but this seems like a hack and I feel like it's not a good design to have the executor mucking around with tags.

[nishanttotla]

I agree that we shouldn't add this hack into the Swarm executor.

We could consider resolving this issue for 1.14, but I suspect this will also go hand in hand with making docker/docker/reference converge with docker/distribution/reference.

[aluzzardi]

Tags may not map directly to digests.

You could run two busybox:latest services with the two of them mapping to a different digest, so I don't know if displaying the tag is correct

[thaJeztah]

@aluzzardi yes, was discussing with @aaronlehmann, we definitely can't fix this for 1.13 without major refactoring, but since swarm resolves the digest from the image:tag provided (and docker service ps shows image:tag@sha256:digest, I think the expected result should be to have both image:tag and digest in the image cache.

I'm adding this to the 1.14 milestone so that we can discuss what options are possible

[aaronlehmann]

@aluzzardi makes a good point that there may be multiple services that reference the same tag but were resolved to different digests. In this case, it would be bad for the tag to switch between the two images depending which was pulled most recently. This seems like a good argument for services not having ownership of the tag.

[patran]

Observed the same image tag set to when using docker stack deploy on 17.03.0-ce. In this case, the tag should have been set to a specific version such as "1.255"

docker info
Containers: 2
Running: 2
Paused: 0
Stopped: 0
Images: 15
Server Version: 17.03.0-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: active
NodeID: tx0s9bitg2858pprm672u4sjk
Is Manager: true
ClusterID: 9s5f2yjdf2hgzihiqxxaj7qqc
Managers: 3
Nodes: 4
Orchestration:
Task History Retention Limit: 5

[kumlali]

This behavior causes some tools such as Artifactory to fail as they expect a not null tag:

• http://stackoverflow.com/questions/42480732/docker-pull-by-digest-does-not-work-with-docker-1-13-1-artifactory-5-0-1
• https://www.jfrog.com/jira/browse/RTFACT-10543

The tag is not empty if we call

$ docker run ...

But as @thaJeztah mentioned, in the case of

$ docker service create ...

the tag is empty. Therefore, while

$ docker run --name web artifactory.mycompany.com/nginx:alpine

works

$docker service create --name web artifactory.mycompany.com/nginx:alpine

fails as it causes Artifactory to throw NullPointerException.

[kumlali]

https://www.jfrog.com/jira/browse/RTFACT-10543 fixed with Artifactory 5.2.1 release. I confirmed Artifactory happily handles null tag now.

[robertofabrizi]

This is really bad imo, the tag column was created because it's really needed rather than a luxury, otherwise what am I supposed to infer from this output?

[ec2-user@ip-x-x-x-x ~]$ docker images | grep ms-stat
usr.eu-west-1.amazonaws.com/ms/ms-stat <none> f0ac3b6ab7f5 2 weeks ago 804 MB
usr.eu-west-1.amazonaws.com/ms/ms-stat <none> f465bee8575c 4 weeks ago 803 MB
usr.eu-west-1.amazonaws.com/ms/ms-stat <none> bfb9c9722a07 4 weeks ago 803 MB
usr.eu-west-1.amazonaws.com/ms/ms-stat <none> 5e99dbebf6ad 5 weeks ago 801 MB
usr.eu-west-1.amazonaws.com/ms/ms-stat <none> 2eb899fca0a3 2 months ago 801 MB
usr.eu-west-1.amazonaws.com/ms/ms-stat <none> d2d54ce887b7 2 months ago 801 MB
usr.eu-west-1.amazonaws.com/ms/ms-stat <none> dcda47dd5bdd 2 months ago 801 MB

[mikegcoleman]

Is the behavior not consistent with Windows? It appears on Windows you do get an image tag whereas Linux you do not

[gaui]

Milestone 17.04 and still not fixed in 17.09 🙃

[SvenDowideit]

@thaJeztah weird thing I'm seeming with v18.3:

dow184@OA-24-MEL:~/gitlab-sync$ docker ps | grep proxy
29ae03847340        imtsc-cont-reg.it.csiro.au/onaci/thredds-proxy:master                            "/bin/sh -c 'python …"   3 minutes ago       Up 3 minutes                                         ereefs_thredds-proxy.1.9yv69ejw110fjktfxn8ays7p5
650bb2918d67        imtsc-cont-reg.it.csiro.au/onaci/thredds-proxy:master                            "/bin/sh -c 'python …"   2 days ago          Up 2 days                                            habs_thredds-proxy.1.wcudhka5pdnxzf3l2g03rj72u
dow184@OA-24-MEL:~/gitlab-sync$ docker image ls | grep proxy
imtsc-cont-reg.it.csiro.au/onaci/thredds-proxy                   master                                     0f5eecad9645        7 minutes ago       661MB
imtsc-cont-reg.it.csiro.au/onaci/thredds-proxy                   <none>                                     65d821519e71        8 days ago          661MB
dow184@OA-24-MEL:~/gitlab-sync$ docker rmi 65d821519e71
Error response from daemon: conflict: unable to delete 65d821519e71 (cannot be forced) - image is being used by running container 29ae03847340

not only was the service started with "none", but docker ps is then telling us that the running container is using a different image..

[FirassKoubaa]

any solutions for this ?

[droplet-js]

I have same question.

registry                                  <none>              sha256:2a5b47a613fd7e9d28120fa77016554c3dffa8913b6a314ede518447ddc68e2f   9c1f09fe9a86        4 hours ago         33.3MB
joxit/docker-registry-ui                  <none>              sha256:52384cd84e8e58a446ff990c2108d35863ea8a8d64bcc410deb5caa5550bfd8c   f59f8d09b3d1        23 hours ago        18.5MB
drone/agent                               <none>              sha256:3bda60d4e6f3cb296ea821bda22388da40fcd4b420a2096d66ba2d8a9a5c3bc4   120e85b52894        2 days ago          16.7MB
drone/drone                               <none>              sha256:fc7f36550adfd65ae4b995fd38eb562960266781521ac576a7499526339813c5   5728ad548bc8        2 days ago          63.5MB
traefik                                   <none>              sha256:a9512e9e1d2c0eda3a971926f50e97467fbeecfe07eeeedc4224af4ee5329a4e   b0a0d7271403        6 days ago          69.3MB
portainer/portainer                       <none>              sha256:ef5dedff75eeae437d09280b6a09d429b232b8fd08c96464deb8759f9bfdb25c   a01958db7424        9 days ago          72.2MB
gogs/gogs                                 <none>              sha256:b6716a863349d22fbd139da69274f44d4109936a4a47fa2582b5d082a2c3b7cd   0688786a8635        9 days ago          91.9MB
nginx                                     <none>              sha256:5d32f60db294b5deb55d078cd4feb410ad88e6fe77500c87d3970eca97f54dba   568c4670fa80        3 weeks ago         109MB
hyperapp/frp                              <none>              sha256:f8af6876896d09f87896dd91c8ca8bc6264cdaabc126b162f366ab3855cfc82a   cc7a591c9c3a        2 months ago        20.7MB

[root@centos dev-ops]# docker -v
Docker version 18.09.0, build 4d60db4

[djbrown]

@thaJeztah any plans on this?
i understand the "dilemma" but its a major pain for me :/

[djbrown]

I think the resolution of the image id could or should be up to the user to decide. it's nice, that docker swarm does this behind the scenes, but I think users that care about this (same id on every node) would be just fine to do it by themselves. users that don't care about it, would be fine if swarm didn't prevent minor discrepancies.

Is this argument good enough to change the current behaviour?🤔

[thaJeztah]

I understand the "dilemma" but its a major pain for me :/

What is the exact problem you're running into because of this?

but I think users that care about this (same id on every node) would be just fine to do it by themselves

Manually pulling images to keep them up-to-date on each node means you'd need (ssh) access to each node, and to authenticate with registries on each node, which is a risk as those credentials would/could persist on worker nodes, and compromising a worker node would also compromise those credentials.
When images are pulled through docker swarm workers, credentials are shared through the encrypted swarm store, and won't linger around on the node.

users that don't care about it, would be fine if swarm didn't prevent minor discrepancies.

Minor discrepancies could include the difference between an image with a critical vulnerability on one node, and a patched image on another node. It also avoids "spoofing" images (e.g., putting a fake ubuntu:latest image in a node's local image cache). Swarm services are designed to be declarative; the Spec of the service describes the service, and swarm makes sure all instances of the service are identical (through the reconciliation loop)

Is this argument good enough to change the current behaviour?🤔

The default will definitely not change.

The default is to resolve the digest;

docker service create --name myservice nginx:1.20.0-alpine

docker service inspect --format '{{ .Spec.TaskTemplate.ContainerSpec.Image }}' myservice
nginx:1.20.0-alpine@sha256:e015192ec74937149dce3aa1feb8af016b7cce3a2896246b623cfd55c14939a6
------------------- -----------------------------------------------------------------------
     |                                  |
     |                                  +-------- digest that the image was resolved to when last updating the service
     |
     +--------------------------------------------image reference that was used to create the service

But if you want images to be pulled by their reference, you can pass the --no-resolve-image option; doing so would still pull the image on each node (to make sure the image exist, and to keep the registry as source of truth);

docker image rm -f nginx:1.20.0-alpine
docker service create --no-resolve-image --name myservice nginx:1.20.0-alpine

The service will now reference the image by name/tag:

docker service inspect --format '{{ .Spec.TaskTemplate.ContainerSpec.Image }}' myservice
nginx:1.20.0-alpine

And use name:tag to pull it:

docker image ls --filter 'reference=*nginx'
REPOSITORY   TAG             IMAGE ID       CREATED        SIZE
nginx        1.20.0-alpine   3b715e351972   13 days ago    22.6MB

[djbrown]

Thanks for the explanation.
My pain points are

• resolution speed (and thus deployment speed) is slow (might not be related to swarm)
• non trivial traceability (deeper inspection needed)
  Probably docker stack deploy --resolve-image=changed ... is what I was looking for and makes this Issue a "nit" for me too.
  While searching I found docker stack deploy creates no tags #31574. That seems to be a dupe... (docker stack deploy creates no tags #31574 (comment))