Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Different image ids when pulling on different servers #17749

Closed
teodor-pripoae opened this issue Nov 6, 2015 · 11 comments
Closed

Different image ids when pulling on different servers #17749

teodor-pripoae opened this issue Nov 6, 2015 · 11 comments

Comments

@teodor-pripoae
Copy link

Hi

I upgraded from lxc-docker-1.7.1 to lxc-docker-1.9.0 on ubuntu 14.04. I'm pulling docker images on every chef provision (check for updated version). I can see docker has the same image id when pulling, but docker images says different image ids.

Let me know if I can help with any logs or debug information.

# server 1
$ docker pull kuende/slugrunner:2.0
2.0: Pulling from kuende/slugrunner
Digest: sha256:c5785763b6fbe936c8ee9c4ecd3df27c84fb6904b21ae0506e846ad83975f225
Status: Image is up to date for kuende/slugrunner:2.0

# server 2
$ docker pull kuende/slugrunner:2.0
2.0: Pulling from kuende/slugrunner
Digest: sha256:c5785763b6fbe936c8ee9c4ecd3df27c84fb6904b21ae0506e846ad83975f225
Status: Image is up to date for kuende/slugrunner:2.0

# server 1
$ docker images | grep slugrunner
kuende/slugrunner   2.0                 4622c1512a6e        11 days ago         915.7 MB
# server 2
$ docker images | grep slugrunner
kuende/slugrunner   2.0                 c9ac17a6ed1b        11 days ago         915.7 MB
@GordonTheTurtle
Copy link

Hi!

Please read this important information about creating issues.

If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.

If you suspect your issue is a bug, please edit your issue description to include the BUG REPORT INFORMATION shown below. If you fail to provide this information within 7 days, we cannot debug your issue and will close it. We will, however, reopen it if you later provide the information.

This is an automated, informational response.

Thank you.

For more information about reporting issues, see https://github.com/docker/docker/blob/master/CONTRIBUTING.md#reporting-other-issues


BUG REPORT INFORMATION

Use the commands below to provide key information from your environment:

docker version:
docker info:
uname -a:

Provide additional environment details (AWS, VirtualBox, physical, etc.):

List the steps to reproduce the issue:
1.
2.
3.

Describe the results you received:

Describe the results you expected:

Provide additional info you think is important:

----------END REPORT ---------

#ENEEDMOREINFO

@teodor-pripoae
Copy link
Author

System information:

$ # same on both servers
$ docker version
Client:
 Version:      1.9.0
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   76d6bc9
 Built:        Tue Nov  3 19:20:09 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.0
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   76d6bc9
 Built:        Tue Nov  3 19:20:09 UTC 2015
 OS/Arch:      linux/amd64

$ # First server
$ docker info
Containers: 0
Images: 30
Server Version: 1.9.0
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 30
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.19.0-32-generic
Operating System: Ubuntu 14.04.3 LTS
CPUs: 1
Total Memory: 1.657 GiB
Name: c01.<redacted>
ID: <redacted>
WARNING: No swap limit support

$ # second server
$ docker info 
Containers: 7
Images: 29
Server Version: 1.9.0
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 43
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.19.0-32-generic
Operating System: Ubuntu 14.04.3 LTS
CPUs: 1
Total Memory: 1.657 GiB
Name: c02.<redacted>
ID: <redacted>
WARNING: No swap limit support

$ # first server 
$ uname -a
Linux c01.<redacted> 3.19.0-32-generic #37~14.04.1-Ubuntu SMP Thu Oct 22 09:41:40 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

$ # second server
$ uname -a
Linux c02.<redacted> 3.19.0-32-generic #37~14.04.1-Ubuntu SMP Thu Oct 22 09:41:40 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Both servers are on google compute engine. Steps for reproducing are in the first post. This problem happened only for first server (c01). First server(c01) seems to have the correct image id, all others have the wrong image id. After removing docker, wiping /var/lib/docker and installing again the image id is the same as on first server. The server used to build the image still have the old version. Removing the image and pulling again on build server still shows the old image id.

@amancevice
Copy link

I'm seeing similar behavior, but only for users using Docker 1.9 vs. 1.8. Most of us are on Docker 1.8, while a few of us are testing Docker 1.9. We store our images on a private registry running distribution/2.1.1

Users running Docker 1.8 all see the same image ID when pulling, while users running 1.9 see a different-same image ID. For example, all users running 1.8 may pull <registry>/<image>:latest and get ID 123456789abc, while users running 1.9 may pull the same tag and get ID fedcba987654

Example user running 1.8:

$ docker version
Client:
 Version:      1.8.1
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   d12ea79
 Built:        Thu Aug 13 02:49:29 UTC 2015
 OS/Arch:      darwin/amd64

Server:
 Version:      1.8.1
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   d12ea79
 Built:        Thu Aug 13 02:49:29 UTC 2015
 OS/Arch:      linux/amd64

$ docker info
Containers: 22
Images: 338
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 382
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.0.9-boot2docker
Operating System: Boot2Docker 1.8.1 (TCL 6.3); master : 7f12e95 - Thu Aug 13 03:24:56 UTC 2015
CPUs: 1
Total Memory: 996.2 MiB
Name: default
ID: XWDR:NDLK:7CBQ:N7PL:OEAP:KIUU:IMXV:2SNN:ITJC:VZIX:MZ7O:SVJP
Debug mode (server): true
File Descriptors: 11
Goroutines: 36
System Time: 2015-11-23T16:23:43.918742631Z
EventsListeners: 0
Init SHA1: 
Init Path: /usr/local/bin/docker
Docker Root Dir: /mnt/sda1/var/lib/docker
Labels:
 provider=virtualbox

$ uname -a
Darwin ip-10-100-0-28.us-west-2.compute.internal 14.5.0 Darwin Kernel Version 14.5.0: Wed Jul 29 02:26:53 PDT 2015; root:xnu-2782.40.9~1/RELEASE_X86_64 x86_64

Example user running 1.9

$ docker version
Client:
 Version:      1.9.0
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   76d6bc9
 Built:        Tue Nov  3 19:20:09 UTC 2015
 OS/Arch:      darwin/amd64

Server:
 Version:      1.9.0
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   76d6bc9
 Built:        Tue Nov  3 19:20:09 UTC 2015
 OS/Arch:      linux/amd64

$ docker info
Containers: 0
Images: 60
Server Version: 1.9.0
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 60
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.1.12-boot2docker
Operating System: Boot2Docker 1.9.0 (TCL 6.4); master : 16e4a2a - Tue Nov  3 19:49:22 UTC 2015
CPUs: 1
Total Memory: 3.859 GiB
Name: default
ID: UEW4:WNAU:FRED:J3JT:6YTY:BZJ4:PJA6:6RBX:444D:B7JT:4ILX:SWBY
Debug mode (server): true
 File Descriptors: 13
 Goroutines: 20
 System Time: 2015-11-23T16:34:48.61314161Z
 EventsListeners: 0
 Init SHA1: 
 Init Path: /usr/local/bin/docker
 Docker Root Dir: /mnt/sda1/var/lib/docker
Labels:
 provider=virtualbox

$ uname -a
Darwin ip-10-100-0-105.us-west-2.compute.internal 15.0.0 Darwin Kernel Version 15.0.0: Sat Sep 19 15:53:46 PDT 2015; root:xnu-3247.10.11~1/RELEASE_X86_64 x86_64

@tonistiigi
Copy link
Member

Images pulled from v2 registries after v1.8.3 are stored by a checksum based ID, but if we can validate that previously pulled image was the same, we will not redownload. I would presume that server1 already had kuende/slugrunner:2.0 pulled before upgrading and server2 didn't.

Having different ID should be harmless and have no effect on pushing/building for example. If you wan't to get the same ID remove image from server1 and redownload.

This issue will go away after #17924

@teodor-pripoae
Copy link
Author

Ok, thanks ! I was afraid that images got corrupted after upgrade, but I verified content and it looks the same.

Edit: I was was using the default public registry.

@amancevice
Copy link

Thanks @tonistiigi. I suspect what happened in my case is that a user using 1.9.0 pushed an image to a v2 registry and a different user pulled this image, re-tagged as "latest", and pushed to the same registry. The result is that users >1.8.2 see the two images as different and download different layers for each.

@thaJeztah
Copy link
Member

#17924 Was merged, so I'm closing this issue

@rajeshchowdary143
Copy link

Hi
Im using docker version 1.11.2, here i was building an image in server1 and pushing it to nexus and pulling it back in server2 and re-tagging with some prefix from there and re-pushing it back and finally pulling back newly tagged image on server2. both of the them are same docker versions.
When i'm building an image in server1 a random sha hash value is getting allocated and i'm pushing it to nexus, when i'm pulling back that from server2 it still shows same sha value, later when i'm re-tagging that image with 'docker tag imagename imagename-release', its still shows same hash value. but when i tried pushing it and pulling it back onto same 2nd server where i retagged it, its showing different hash value now.
Server 1 and 2 details and output of hash given below.
Server1:
Client:
Version: 1.11.2
API version: 1.23
Go version: go1.5.4
Git commit: b9f10c9
Built: Wed Jun 1 21:23:11 2016
OS/Arch: linux/amd64

Server:
Version: 1.11.2
API version: 1.23
Go version: go1.5.4
Git commit: b9f10c9
Built: Wed Jun 1 21:23:11 2016
OS/Arch: linux/amd64

Containers: 10
Running: 1
Paused: 0
Stopped: 9
Images: 168
Server Version: 1.11.2
Storage Driver: devicemapper
Pool Name: docker-253:6-1310724-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 9.136 GB
Data Space Total: 107.4 GB
Data Space Available: 55.6 GB
Metadata Space Used: 13.91 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.134 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /app/docker-data/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Either use --storage-opt dm.thinpooldev or use --storage-opt dm.no_warn_on_loop_devices=true to suppress this warning.
Metadata loop file: /app/docker-data/devicemapper/devicemapper/metadata
Library Version: 1.02.107-RHEL7 (2015-12-01)
Logging Driver: journald
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge null host
Kernel Version: 3.10.0-327.13.1.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.2 (Maipo)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.64 GiB
Name: d1xvoaxaxxxx.xxxxxx.com
ID: PQNS:4EOI:7F5Y:LXKV:XLCI:7C6L:7Z6J:RATA:SBDC:NXJR:JD4V:UDLX
Docker Root Dir: /app/docker-data
Debug mode (client): false
Debug mode (server): false
Http Proxy: http://prod-proxy-out.xxxxxx.com:8080
Https Proxy: http://prod-proxy-out.xxxxxx.com:8080
No Proxy: .xxx.com,*.xxxxx.com,.xxxxxx.com,120.0.0.1,localhost:8081,147.22.0.0/16,10.55.0.0/16
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

uname -a
Linux d1xvoaxaxxxx.xxxx.com 3.10.0-327.13.1.el7.x86_64 #1 SMP Mon Feb 29 13:22:02 EST 2016 x86_64 x86_64 x86_64 GNU/Linux

Server2:
Client:
Version: 1.11.2
API version: 1.23
Go version: go1.5.4
Git commit: b9f10c9
Built: Wed Jun 1 21:23:11 2016
OS/Arch: linux/amd64

Server:
Version: 1.11.2
API version: 1.23
Go version: go1.5.4
Git commit: b9f10c9
Built: Wed Jun 1 21:23:11 2016
OS/Arch: linux/amd64

Containers: 2
Running: 0
Paused: 0
Stopped: 2
Images: 133
Server Version: 1.11.2
Storage Driver: devicemapper
Pool Name: vgdocker-thinpool
Pool Blocksize: 524.3 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file:
Metadata file:
Data Space Used: 9.417 GB
Data Space Total: 53.01 GB
Data Space Available: 43.6 GB
Metadata Space Used: 3.891 MB
Metadata Space Total: 557.8 MB
Metadata Space Available: 554 MB
Udev Sync Supported: true
Deferred Removal Enabled: true
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Library Version: 1.02.107-RHEL7 (2015-12-01)
Logging Driver: journald
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge null host
Kernel Version: 3.10.0-327.13.1.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.2 (Maipo)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.64 GiB
Name: d1xvoaxaxxxx.xxxxxx.com
ID: VCRC:ADFT:JD4C:COW4:KBSN:RRUV:SVZZ:4K3S:AT4X:GQMH:2FXB:R46D
Docker Root Dir: /app/docker-data
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

uname -a
Linux d1xvoaxaxxxx.xxxx.com 3.10.0-327.13.1.el7.x86_64 #1 SMP Mon Feb 29 13:22:02 EST 2016 x86_64 x86_64 x86_64 GNU/Linux

Image Id, while image is build in Server1 and before pushed to nexus:
zlpxxxx.vci.xxx.com:5104/xxxxx/play:release-1.0.0-play-v0.0.26 sha256:443007013ab89b81d56cd5ef190ec264d0811c432c8c2b7c44df2d1a0da779b3

Sudo docker images output after image is pulled on to server2 from nexus and retagged image also:
zlpxxxxx.vci.xxxx.com:5104/xxxxxx/play release-1.0.0-play-v0.0.26 443007013ab8 6 days ago 421 MB
zlpxxxxx.vci.xxx.com:5104/xxxxxx/play release-1.0.0-play-v0.0.26-release 443007013ab8 6 days ago 421 MB

Pushed tagged image to nexus
The push refers to a repository [zlpxxxxx.vci.xxx.com:5104/xxxxxx/play]
69ea06aea2d2: Layer already exists
89759c5cf405: Layer already exists
a60b21869669: Layer already exists
82b19ed7af22: Layer already exists
db1165172aa7: Layer already exists
eca3b1961920: Layer already exists
0e4a8a292bd7: Layer already exists
316c1485e3bf: Layer already exists
abbc79cd5eca: Layer already exists
13a574aedcac: Layer already exists
f5489c27d105: Layer already exists
cf8a0bf937ed: Layer already exists
011b303988d2: Layer already exists
release-1.0.0-play-v0.0.26-release: digest: sha256:0f6f87f091890b07d514c1c5491e438a8e9afea15aa0c7dc59eaf0f2d03fac0d size: 15325

Pulling image back
release-1.0.0-play-v0.0.26-release: Pulling from xxxxxx/play
3690ec4760f9: Already exists
482d30ba9981: Already exists
94f52be01b4d: Already exists
720272626c13: Already exists
99a94c7cdbb0: Already exists
340ce19466ff: Already exists
e04c172fbdbc: Already exists
d0a112731d68: Already exists
e424c2a01aab: Already exists
f6da60fba748: Already exists
1b282f5a514d: Already exists
9f2e62495ebb: Already exists
8adc8b347d66: Already exists
Digest: sha256:0f6f87f091890b07d514c1c5491e438a8e9afea15aa0c7dc59eaf0f2d03fac0d
Status: Downloaded newer image for zlpxxxxx.vci.xxxx.com:5104/xxxxxx/play:release-1.0.0-play-v0.0.26-release

Now image id differs after pulling back:
zlpxxxx.vci.xxx.com:5104/xxxxxx/play release-1.0.0-play-v0.0.26 443007013ab8 6 days ago 421 MB
zlpxxxx.vci.xxx.com:5104/xxxxx/play release-1.0.0-play-v0.0.26-release 444d858e3155 6 days ago 421 MB

Looking to know what is causing docker to allocate different new Image Id, to retagged image that i pulled back.

@thaJeztah
Copy link
Member

ping @tonistiigi ^^

@tonistiigi
Copy link
Member

@rajeshchowdary143
Please create a new issue for this and ping me on it.
Please post the registry version and daemon logs.
Please post output of following commands:

cat /var/lib/docker/image/devicemapper/imagedb/content/sha256/443007013ab8<expand to full id>
cat /var/lib/docker/image/devicemapper/imagedb/content/sha256/444d858e3155<expand to full id>

@thaJeztah
Copy link
Member

Ok, let me lock the conversation on this issue so that new reports of possible bugs are not reported here and don't get lost.

If you are running into this issue, and suspect there's a bug at hand, open a new issue and provide the information that's requested in the issue-template that is shown when you open the issue, at least;

  • provide the output of docker version
  • provide the output of docker info

For this issue, also provide the registry version and daemon logs (information as requested above #17749 (comment))

@moby moby locked and limited conversation to collaborators Feb 24, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants