1.10: keeps pushing same image

[ibuildthecloud]

When running 1.10 I keep pushing the same images over and over again. If I do docker push multiple times it doesn't seem to say "Layer already exists" for what I just pushed.

[aaronlehmann]

Are you pushing to a v1 registry or v2 registry?

I'm not seeing these repeated pushes with either protocol version:

v2:

root@e61481334bc9:/go/src/github.com/docker/docker# docker push registry:5000/alpine
The push refers to a repository [registry:5000/alpine]
6b00b82071ac: Pushed 
latest: digest: sha256:6371e0134ff330629c625f28d34b6357c99cc4c1c6ea1061b9375f7ff2f43706 size: 1352
root@e61481334bc9:/go/src/github.com/docker/docker# docker push registry:5000/alpine
The push refers to a repository [registry:5000/alpine]
6b00b82071ac: Layer already exists 
latest: digest: sha256:6371e0134ff330629c625f28d34b6357c99cc4c1c6ea1061b9375f7ff2f43706 size: 1352

v1:

root@e61481334bc9:/go/src/github.com/docker/docker# docker push registry:5000/alpine
The push refers to a repository [registry:5000/alpine]
Sending image list
Pushing repository registry:5000/alpine
2b6968525989: Image successfully pushed 
Pushing tag for rev [2b6968525989] on {http://registry:5000/v1/repositories/alpine/tags/latest}
root@e61481334bc9:/go/src/github.com/docker/docker# docker push registry:5000/alpine
The push refers to a repository [registry:5000/alpine]
Sending image list
Pushing repository registry:5000/alpine
Image 2b6968525989 already pushed, skipping
Pushing tag for rev [2b6968525989] on {http://registry:5000/v1/repositories/alpine/tags/latest}

[ibuildthecloud]

This is interesting... So if I push the same images I was pushing before, it now says layer already exists. What I was doing was pushing a series of images that were all based on the same common image. As you can see below the 191273970f44: layer is in most of the images but each push would push it again.

v0.4.3-dev: digest: sha256:9988e24a06a194e7cee9beec070c01729b614de0f73a80d7ffbfec08bc1f62f3 size: 1372
The push refers to a repository [docker.io/rancher/os-base]
5f70bf18a086: Layer already exists 
52a9cf3f11a8: Pushed 
4d7890e8b3bf: Pushed 
4e0a630d2bbc: Pushed 
f42a904f1fd3: Pushed 
191273970f44: Pushed 
v0.4.3-dev: digest: sha256:65c892a5a43fa65a610266d5db0fd1d61adb6c74793a1e7bacc48fee3cd125ba size: 5151
The push refers to a repository [docker.io/rancher/os-acpid]
5f70bf18a086: Layer already exists 
52a9cf3f11a8: Pushed 
4d7890e8b3bf: Pushed 
4e0a630d2bbc: Pushed 
f42a904f1fd3: Pushed 
191273970f44: Pushed 
v0.4.3-dev: digest: sha256:86d3c446b9cd4f2a12d17b3cf18b42d665ffc9716759ba302f4882bea229b44d size: 5654
The push refers to a repository [docker.io/rancher/os-autoformat]
5f70bf18a086: Layer already exists 
b5a18c111367: Pushed 
6d0a6b94807d: Pushed 
52a9cf3f11a8: Pushed 
4d7890e8b3bf: Pushed 
4e0a630d2bbc: Pushed 
f42a904f1fd3: Pushed 
191273970f44: Pushed 
v0.4.3-dev: digest: sha256:c83f1d1be9c50c380ef261405f2120899fb9a2549e9f003ef32bc74e98611e47 size: 6666
The push refers to a repository [docker.io/rancher/os-cloudinit]
5f70bf18a086: Layer already exists 
d143fff25776: Pushed 
52a9cf3f11a8: Pushed 
4d7890e8b3bf: Pushed 
4e0a630d2bbc: Pushed 
f42a904f1fd3: Pushed 
191273970f44: Pushed 
v0.4.3-dev: digest: sha256:c2f456a131cf8df6a30b4731d9a09d276261df58919115aad0261828b394e204 size: 6136
The push refers to a repository [docker.io/rancher/os-console]
5f70bf18a086: Layer already exists 
6617e06170e4: Pushed 
576a7382fe08: Pushed 
7ad6b6b5a1a0: Pushed 
52a9cf3f11a8: Pushed 
4d7890e8b3bf: Pushed 
4e0a630d2bbc: Pushed 
f42a904f1fd3: Pushed 
191273970f44: Pushed 
v0.4.3-dev: digest: sha256:af28be0a926eeec2e9aa4edb051555d25064aef4c2eeba3c388d24ef45b4009c size: 7697
The push refers to a repository [docker.io/rancher/os-docker]
5f70bf18a086: Layer already exists 
52a9cf3f11a8: Pushed 
4d7890e8b3bf: Pushed 
4e0a630d2bbc: Pushed 
f42a904f1fd3: Pushed 
191273970f44: Pushed 
v0.4.3-dev: digest: sha256:bd9442e705d53dce71339ef2094fe5db06cad9587f23cce386384c50dea14da9 size: 5647
The push refers to a repository [docker.io/rancher/os-headers]
5f70bf18a086: Layer already exists 
41a628dae1f0: Pushed 
52a9cf3f11a8: Pushed 
4d7890e8b3bf: Pushed 
4e0a630d2bbc: Pushed 
f42a904f1fd3: Pushed 
191273970f44: Pushed 
v0.4.3-dev: digest: sha256:26977edb654d8abdf865b29703d6c153e7de94dd535b35ff11b344c4d23c0098 size: 6127
The push refers to a repository [docker.io/rancher/os-network]
5f70bf18a086: Layer already exists 
52a9cf3f11a8: Pushed 
4d7890e8b3bf: Pushed 
4e0a630d2bbc: Pushed 
f42a904f1fd3: Pushed 
191273970f44: Pushed 

[aaronlehmann]

This looks like the classic cross-repository push issue. Currently, the registry doesn't know which other repositories a client has permission to access, so it only allows layers to be shared within a specific repository (such as rancher/os-base). The first time those layers are pushed to a different repository, they need to be re-pushed to prove to the registry that the client has access to them. (Note that the layers end up duplicated on disk even though they are pushed multiple times).

The good news is that there's active work underway to seamlessly share layers between different repositories. The first step is this registry-side PR: distribution/distribution#1269. @BrianBland also has some code for the engine that will make use of this to push a layer from a second repository where it already exists.

[dmcgowan]

This item is already covered by #14018 in the roadmap. Closing this as a duplicate.

[joshk0]

(Note that the layers end up duplicated on disk even though they are pushed multiple times).

"Don't" end up duplicated on disk perhaps?

[aaronlehmann]

"Don't" end up duplicated on disk perhaps?

Yes, thanks for the correction.

[joshk0]

Cool. Yeah, we hit this bug regularly, so just want to make sure we're not blowing out our cloud storage bucket with a zillion copies of our debian base image.

[harmeetsingh-work]

Same here, I pushed my image and then pulled the same image and then when I pushed the same image it started uploading, instead of saying Already exists, please fix this issue.

Docker version 1.10.2, build c3959b1

[thaJeztah]

@BlackAdmin what version of the registry are you using? This feature was implemented in docker 1.10, but requires registry 2.3.0 (also see distribution/distribution#1501)

[harmeetsingh-work]

@thaJeztah Sorry, but how to check registry version ? :)

[thaJeztah]

@BlackAdmin are you running your own private registry, or Docker Hub? Docker Hub does not yet run registry 2.3 (You can try sending an email to support@docker.com to check if there's a date known for 2.3 to be rolled out)

[harmeetsingh-work]

@thaJeztah Me on Docker Hub, ok got it.
Thank you very much :)