docker stack deploy creates no tags

[Goram]

Hi, I want to change my docker-compose files to version 3, so I tried it with an example:

version: '3'

services:

  bitbucket:
    image: atlassian/bitbucket-server:4.14
    hostname: bitbucket
    ports:
      - "7990:7990"

Steps to reproduce the issue:

1. docker stack deploy --compose-file .\docker-stack.yml name
2. docker images

Describe the results you received:

REPOSITORY                   TAG                 IMAGE ID            CREATED             SIZE
atlassian/bitbucket-server   <none>              38116e66106c        5 days ago          659 MB

Describe the results you expected:

REPOSITORY                   TAG                 IMAGE ID            CREATED             SIZE
atlassian/bitbucket-server     4.14            38116e66106c        5 days ago          659 MB

Output of docker version:

Client:
 Version:      17.03.0-ce
 API version:  1.26
 Go version:   go1.7.5
 Git commit:   60ccb22
 Built:        Thu Feb 23 10:40:59 2017
 OS/Arch:      windows/amd64

Server:
 Version:      17.03.0-ce
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   3a232c8
 Built:        Tue Feb 28 07:52:04 2017
 OS/Arch:      linux/amd64
 Experimental: true

Output of docker info:

Containers: 1                                                       
 Running: 1                                                         
 Paused: 0                                                          
 Stopped: 0                                                         
Images: 1                                                           
Server Version: 17.03.0-ce                                          
Storage Driver: aufs                                                
 Root Dir: /var/lib/docker/aufs                                     
 Backing Filesystem: extfs                                          
 Dirs: 13                                                           
 Dirperm1 Supported: true                                           
Logging Driver: json-file                                           
Cgroup Driver: cgroupfs                                             
Plugins:                                                            
 Volume: local                                                      
 Network: bridge host ipvlan macvlan null overlay                   
Swarm: active                                                       
 NodeID: 8mltynfgboqwe40jt77glbm73                                  
 Is Manager: true                                                   
 ClusterID: 30qn69c470mipl2w85q09xpuj                               
 Managers: 1                                                        
 Nodes: 1                                                           
 Orchestration:                                                     
  Task History Retention Limit: 5                                   
 Raft:                                                              
  Snapshot Interval: 10000                                          
  Number of Old Snapshots to Retain: 0                              
  Heartbeat Tick: 1                                                 
  Election Tick: 3                                                  
 Dispatcher:                                                        
  Heartbeat Period: 5 seconds                                       
 CA Configuration:                                                  
  Expiry Duration: 3 months                                         
 Node Address: 192.168.65.2                                         
 Manager Addresses:                                                 
  192.168.65.2:2377                                                 
Runtimes: runc                                                      
Default Runtime: runc                                               
Init Binary: docker-init                                            
containerd version: 977c511eda0925a723debdc94d09459af49d082a        
runc version: a01dafd48bc1c7cc12bdb01206f9fea7dd6feb70              
init version: 949e6fa                                               
Security Options:                                                   
 seccomp                                                            
  Profile: default                                                  
Kernel Version: 4.9.12-moby                                         
Operating System: Alpine Linux v3.5                                 
OSType: linux                                                       
Architecture: x86_64                                                
CPUs: 2                                                             
Total Memory: 1.934 GiB                                             
Name: moby                                                          
ID: F37C:NYPX:LXGB:EEKS:TFHO:Y5BG:SACC:XP76:C4PB:CEGN:BGMU:3D5R     
Docker Root Dir: /var/lib/docker                                    
Debug Mode (client): false                                          
Debug Mode (server): true                                           
 File Descriptors: 48                                               
 Goroutines: 151                                                    
 System Time: 2017-03-06T13:15:56.2977373Z                          
 EventsListeners: 1                                                 
Registry: https://index.docker.io/v1/                               
Experimental: true                                                  
Insecure Registries:                                                
 127.0.0.0/8                                                        
Live Restore Enabled: false                                         

[thaJeztah]

Perhaps the documentation needs to be enhanced on this part. What's happening when deploying a service (or stack), is that docker resolves the digest (immutable reference) of the image you specified. Doing so, guarantees that every instance of the service runs exactly the same image. "tags" are mutable, so for example, the foo:123 image can be overwritten/updated with a newer version, which can result in a service on "node A" running a different version of foo:123, then the same service running on "node B". Ideally the docker-compose file itself should use immutable tags, so that when deploying, you can test the specified version, and know that deploying the stack results in exactly the same image as you used during testing. Many people will use image:tag to specify the image, and for that reason docker will do the "look up" of the image digest for you.

When pulling an image by digest (foo@sha256:aabbff11), no tag is created, because the same digest can be shared by different tags.

It was discussed at some occasions to automatically tag the image after it was pulled by digest, if the original image was specified by image:tag. There were concerns about that possibly overwriting a local tag.

/cc @dmcgowan @aaronlehmann ^^

[Goram]

Ah ok, that clarifies a lot, but there is another hassle with that:

PS D:\development\docker\bitbucket> docker images
REPOSITORY                   TAG                 IMAGE ID            CREATED             SIZE
atlassian/bitbucket-server   <none>              38116e66106c        5 days ago          659 MB

PS D:\development\docker\bitbucket> docker ps
CONTAINER ID        IMAGE                                                                                                COMMAND                CREATED             STATUS              PORTS                NAMES
86059b45bba9        atlassian/bitbucket-server@sha256:5eb22593d07c378f0ebebd6341f46383cbb959196ea544225ee43897b7c26801   "/entrypoint.sh -fg"   9 minutes ago       Up 9 minutes        7990/tcp, 7999/tcp   thisissparta_bitbucket.1.w29wgli9pzitz1kknzgbiyukh

PS D:\development\docker\bitbucket> docker service ls
ID            NAME                    MODE        REPLICAS  IMAGE
ec0llilmcps4  thisissparta_bitbucket  replicated  1/1       atlassian/bitbucket-server:4.14

To see that the container is the one started off of the image is really difficult. Where the service ls shows the image with tag.

[stevvooe]

@thaJeztah This might be as simple as preserving the tag in the digest-qualified reference. For example, something like this:

foo:latest@sha256:aabbff11

@nishanttotla Is this an easy fix?

[aaronlehmann]

@stevvooe: We do preserve the tag in the reference, but we do not overwrite the tag in the local node's reference store, because this might be an unexpected side effect of running a service. For example, let's say I set up a service foo:latest which gets pinned to the digest foo:latest@sha256:aabbff11. Then the next day I push a new version of foo:latest with digest sha256:ccddee22. I pull this latest version on node1, so foo:latest points to the sha256:ccddee22 version. But then if the service happened to run on node1 later on, overwriting the tag on that node to point to the older version would be a bad side effect. Thus running a service does not create or update any tags right now.

Perhaps as a compromise we could create the tag if it does not exist locally, but avoid overwriting any existing tags?

[stevvooe]

@aaronlehmann Understood.

Could we populate this field in docker ps with the value from the actual running container or whatever preserves the reference?

[aaronlehmann]

I believe we do.

[aluzzardi]

Somehow no, the running tag doesn't get shown on ps

[aaronlehmann]

Ah, you're right. I think this is a known issue in currently released versions because the forked version of the reference package couldn't handle both a tag and a digest.

I just tried on master, and while we remove the digest by default for readability (#30848), docker ps --no-trunc shows the full reference, including tag and digest.

[nishanttotla]

@aaronlehmann @aluzzardi I can see the tag on docker ps when I use the master branch. For example:

$ docker service create alpine:3.5 sleep 1000
7vy6p7kp2j0di9gqsktsbi2mi

$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
alpine              <none>              4a415e366388        6 days ago          3.99MB

$ docker service ls
ID                  NAME                MODE                REPLICAS            IMAGE
7vy6p7kp2j0d        kind_stonebraker    replicated          1/1                 alpine:3.5

$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED              STATUS              PORTS               NAMES
180bdd2f2605        alpine:3.5          "sleep 1000"        About a minute ago   Up About a minute                       kind_stonebraker.1.yd0yvsu199jq6xs8lk8dbby47

$ docker ps --no-trunc
CONTAINER ID                                                       IMAGE                                                                                COMMAND             CREATED              STATUS              PORTS               NAMES
180bdd2f2605e07eea4e7b9bd6c7b10ff43057a8e451332c1dcdb70b1d7e9477   alpine:3.5@sha256:58e1a1bb75db1b5a24a462dd5e2915277ea06438c3f105138f97eb53149673c4   "sleep 1000"        About a minute ago   Up About a minute                       kind_stonebraker.1.yd0yvsu199jq6xs8lk8dbby47

[aluzzardi]

@aaronlehmann @nishanttotla Oh, I believe we're both right. It shows the tag unless you use latest (which is the one I used in my test):

~ ❯❯❯ docker service create --name foo nginx:latest
1d0gjiz4o5tyspwf2wol3rc3u

~ ❯❯❯ docker ps
CONTAINER ID        IMAGE                                                                           COMMAND                  CREATED             STATUS              PORTS               NAMES
b908bbd859e7        nginx@sha256:52a189e49c0c797cfc5cbfe578c68c225d160fb13a42954144b29af3fe4fe335   "nginx -g 'daemon ..."   15 seconds ago      Up 12 seconds       80/tcp, 443/tcp     foo.1.gfxnjii2d9pehpfb0nmtako2k

[nishanttotla]

@aluzzardi that doesn't look right, because you can see the digest without --no-trunc.

I believe you might have tested on a slightly older commit. I get exactly the same behavior with latest as well.

[aluzzardi]

That was from the latest release, not master. Has this been fixed on master?

[nishanttotla]

@aluzzardi yes, it has.

[aluzzardi]

@nishanttotla @aaronlehmann Can this one as well as #31656 be closed?

Do we need to backport to 17.03?

[aaronlehmann]

I suggest not backporting. It's not a high priority issue, and I think the fix in master depended on the new reference package (which was an enormous PR).

But I don't think we should close this issue, since it's about whether services should create tags.

[aaronlehmann]

Maybe this is a duplicate of #28908 though?

[styfle]

I recently noticed the issue of the missing tag but I am not using latest tag, I have a v1.0.0 convention for tags. I am using stable Docker CE version. See this question for more info.

[thaJeztah]

@styfle this does not only apply to the :latest tag. Any :tag that is specified is resolved to the immutable identified (digest) of the image, so that the exact version of the image is used on every node. Tags are mutable so, :v1.0.0 can be overwritten by a "newer" :v1.0.0, for that reason the "digest" is used.

[styfle]

@thaJeztah Thanks for the clarification. It sounds like @nishanttotla said this is fixed in master. I'm currently using Docker CE 17.03.1-ce-win5 (10743)...how can I install the latest Windows version to confirm this is fixed? Or if there is no nightly build, when is the next docker release expected?

[hierozbosch]

From above it looks like the services in a compose file should reference image by digest, not tag. So I'm trying like this:

version: "3.2"
services:
   test_site:
      image:  445523203977.dkr.ecr.us-west-2.amazonaws.com/test_site@sha256:4f43add65249....

I'm authenticated with ECS, but I get 'Rejected... Image Not Found' errors on deploy.
AWS ECS seems to reference images by tag, so it looks like a catch-22: either you can pull the image from ECS by tag but can't deploy it since the tag is set to <none>; or you can reference by digest, but then ECS can't find it. Is there a way around this?

[thaJeztah]

@hierozbosch if ECS does not support pulling by digest, their registry does not comply with the specs; in that case I suggest opening a ticket with them.

In docker 17.06 you can disable resolving the image digest with the --resolve-image=never option on docker stack deploy, possibly that helps as a workaround

[hierozbosch]

Thanks @thaJeztah. ECS does indeed support the pull-by-digest according to spec. The problem was me using the wrong digest: I got it from the image I pushed up to ECS (using docker image ls --no-trunc). Now I see there's a different digest when it gets listed on ECS. I'm using that now--it's all good. Thanks.

[thiagolsfortunato]

Any news ?

[kaysond]

This creates a minor annoyance with docker ps --filter ancestor=.

Consider this example where I've deployed traefik via a stack:

bash-5.0# docker image ls
REPOSITORY   TAG         IMAGE ID       CREATED         SIZE
traefik      <none>      ddb0268e5b5f   3 days ago      92.4MB
bash-5.0# docker inspect --format "{{ .Config.Image }}" 2006
traefik:v2.3.7@sha256:0181e35c5af98f7f30fb391f91a6dbd281a90d7cf971e9909e26afd4ea923251

This fails as expected because without a tag, ps defaults to :latest

bash-5.0# docker ps --filter ancestor=traefik
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

This works, obviously:

bash-5.0# docker ps --filter ancestor=traefik:v2.3.7@sha256:0181e35c5af98f7f30fb391f91a6dbd281a90d7cf971e9909e26afd4ea923251
CONTAINER ID   IMAGE            COMMAND                  CREATED          STATUS          PORTS     NAMES
2006ff2635f9   traefik:v2.3.7   "/entrypoint.sh trae…"   11 minutes ago   Up 11 minutes   80/tcp    test_reverseproxy.nr8qswjgf2aftdbiz9wra7ti3.uumpxd5bj2x1a7yy6tfdtboe6

I was hoping this wouldn't fail, but it does:

bash-5.0# docker ps --filter ancestor=traefik:v2.3.7
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

@thaJeztah any thoughts? It looks like the code for the ancestor filter turns the argument into an image id, so since the tag doesn't exist it fails. What about doing a semantic comparison against .Config.Image instead of a sha256 compare against .Image?

[Kotnstantin]

I understand what after docker stack deploy create untaged images, use digest (immutable link) of the specified image, for each service, but how do I remove part of the images that are currently unused?

Example in my case:
`docker images

REPOSITORY TAG IMAGE ID CREATED SIZE
registry.org.com/web-dev/nginx.proxyswarm/nginxwww 76aa8885d7aa 18 hours ago 23.6MB
registry.org.com/web-dev/nginx.proxyswarm/nginxwww df4f6548e721 18 hours ago 23.6MB
registry.org.com/web-dev/control-room/webuicontrol 30eb086260e2 20 hours ago 303MB
registry.org.com/web-dev/control-room/webuicontrol d8185fef4c33 20 hours ago 303MB
registry.org.com/web-dev/control-room/sharecontrol a118f8bb397b 20 hours ago 265MB
registry.org.com/web-dev/control-room/sqlservicecontrol 6d6f1b92db01 20 hours ago 242MB
registry.org.com/web-dev/control-room/webuicontrol 014a07d46b79 22 hours ago 303MB
registry.org.com/web-dev/control-room/sharecontrol ee1ef4d909f4 22 hours ago 265MB
registry.org.com/web-dev/control-room/sqlservicecontrol 4d1eb86a9bd3 22 hours ago 242MB
...`
49 images

`docker images -a

REPOSITORY TAG IMAGE ID CREATED SIZE
registry.org.com/web-dev/nginx.proxyswarm/nginxwww 76aa8885d7aa 18 hours ago 23.6MB
registry.org.com/web-dev/nginx.proxyswarm/nginxwww df4f6548e721 18 hours ago 23.6MB
registry.org.com/web-dev/control-room/webuicontrol 30eb086260e2 20 hours ago 303MB
registry.org.com/web-dev/control-room/webuicontrol d8185fef4c33 20 hours ago 303MB
registry.org.com/web-dev/control-room/sharecontrol a118f8bb397b 20 hours ago 265MB
registry.org.com/web-dev/control-room/sqlservicecontrol 6d6f1b92db01 20 hours ago 242MB
registry.org.com/web-dev/control-room/webuicontrol 014a07d46b79 22 hours ago 303MB
registry.org.com/web-dev/control-room/sharecontrol ee1ef4d909f4 22 hours ago 265MB
registry.org.com/web-dev/control-room/sqlservicecontrol 4d1eb86a9bd3 22 hours ago 242MB
...

49 images

`docker images -f "dangling=true"

REPOSITORY TAG IMAGE ID CREATED SIZE
registry.org.com/web-dev/nginx.proxyswarm/nginxwww 76aa8885d7aa 18 hours ago 23.6MB
registry.org.com/web-dev/nginx.proxyswarm/nginxwww df4f6548e721 18 hours ago 23.6MB
registry.org.com/web-dev/control-room/webuicontrol 30eb086260e2 20 hours ago 303MB
registry.org.com/web-dev/control-room/webuicontrol d8185fef4c33 20 hours ago 303MB
registry.org.com/web-dev/control-room/sharecontrol a118f8bb397b 20 hours ago 265MB
registry.org.com/web-dev/control-room/sqlservicecontrol 6d6f1b92db01 20 hours ago 242MB
registry.org.com/web-dev/control-room/webuicontrol 014a07d46b79 22 hours ago 303MB
registry.org.com/web-dev/control-room/sharecontrol ee1ef4d909f4 22 hours ago 265MB
registry.org.com/web-dev/control-room/sqlservicecontrol 4d1eb86a9bd3 22 hours ago 242MB
...`

43 images

I have 5 stacks (3 stacks of 3 services each, 1 stack of 4 services, 1 stack of 1 service) of just 14 services without scale setting (14 containers))

What images unused?
And what if I use the docker rmi $(docker images -f "dangling=true" -q) -force ?
I assume that everything will fall.

[thaJeztah]

@Kotnstantin would docker image prune (or docker system prune) work?