loading/pulling big images fails on new filesystem

[ajohnsen]

Expected behavior

Being able to pull / load big images

Actual behavior

Fails the checksum / getting eof error

Information

Started failing after 17.12.0-ce-mac46.
I think it has something to do with the new file format because it started happening after i removed all data to get the .raw file.

Output trying to load a big docker image:
Version 17.12.0-ce-mac46:
docker load --input image.tar 5f70bf18a086: Loading layer [==================================================>] 1.024kB/1.024kB 2cca3e821cec: Loading layer [==================================================>] 204.3MB/204.3MB 23c2322da5cc: Loading layer [==================================================>] 116.2kB/116.2kB baf5e2bc5310: Loading layer [==================================================>] 116.2kB/116.2kB 81589c538b34: Loading layer [==================================================>] 273.1MB/273.1MB 50b548a61265: Loading layer [==================================================>] 7.702GB/7.702GB invalid diffID for layer 7: expected "sha256:50b548a61265c3e50b035586ec217a626e8298c59c49d08b23b03b0831f2c9f2", got "sha256:3e30ff6f14d5033b1a0dadb00c256a3027c2e867b325a9bcf33bc00c0b292ef6"

When trying to pull from registry:
a3ed95caeb02: Pull complete da71393503ec: Pull complete fb137735f4d4: Pull complete bcdde3ac9e50: Pull complete 90fe5c2bb701: Pull complete 619178efa443: Extracting [==================================================>] 3.142GB/3.142GB 49c8cc2c34eb: Download complete 23e3af8da16c: Download complete 182b3ad9d241: Download complete 292924bb2f5b: Download complete bc4a39626135: Download complete f63a276e4bad: Download complete cb0276f5dbe0: Download complete f169c694eea5: Download complete a129c193ba88: Download complete b76b23d256f2: Download complete d2b1c0b42e6f: Download complete 6cf272f995ee: Download complete 6b6a92671bad: Download complete docker: failed to register layer: Error processing tar file(exit status 1): unexpected EOF.

Running the same on version 17.09.1-ce-mac42:
docker load --input image.tar 5f70bf18a086: Loading layer [==================================================>] 1.024kB/1.024kB 2cca3e821cec: Loading layer [==================================================>] 204.3MB/204.3MB 23c2322da5cc: Loading layer [==================================================>] 116.2kB/116.2kB baf5e2bc5310: Loading layer [==================================================>] 116.2kB/116.2kB 81589c538b34: Loading layer [==================================================>] 273.1MB/273.1MB 50b548a61265: Loading layer [==================================================>] 7.702GB/7.702GB b3e603045e9e: Loading layer [==================================================>] 3.584kB/3.584kB 1542b88acbbb: Loading layer [==================================================>] 2.59GB/2.59GB 5a8310710f51: Loading layer [==================================================>] 245.5MB/245.5MB ab6c277080fd: Loading layer [==================================================>] 5.12kB/5.12kB 1590a0dc9193: Loading layer [==================================================>] 2.56kB/2.56kB 216516573799: Loading layer [==================================================>] 3.072kB/3.072kB 6881b7453ad2: Loading layer [==================================================>] 3.072kB/3.072kB 07e450d8aee3: Loading layer [==================================================>] 6.144kB/6.144kB 86d43f5fee3d: Loading layer [==================================================>] 6.144kB/6.144kB 3b8b46a1ec7b: Loading layer [==================================================>] 12.29kB/12.29kB ce0242956039: Loading layer [==================================================>] 119.1MB/119.1MB f30c871fa607: Loading layer [==================================================>] 3.072kB/3.072kB b16756f8a208: Loading layer [==================================================>] 3.072kB/3.072kB Loaded image

Steps to reproduce the behavior

1. load/pull big image
2. Checksum/pull fails

[rotorek]

Same issue here, SHA256 layer hash kept changing while I was trying to pull or load a big (15GB) image. After several tries when I succeeded to load the image, the container -- in my case Oracle DB reported corrupted DBF files and was unable to start.

It has something to do with the change of Docker's disk format from QCOW2 to RAW. I had to revert to Docker 17.09.1-ce-mac42 (21090) to get rid of the issue.

Seems to be connected with issue #2542.

[adah1972]

I have the same issue. The image loads well with Docker 17.09.1-ce-mac42 (21090):

gunzip < jupyter-cxx.tar.gz | docker load
875b3fe5d53d: Loading layer  2.646GB/2.646GB
2555e6054852: Loading layer  3.072kB/3.072kB
Loaded image: jupyter-cxx:latest

But it fails with the latest version:

gunzip < jupyter-cxx.tgz | docker load
875b3fe5d53d: Loading layer  2.646GB/2.646GB
invalid diffID for layer 0: expected "sha256:875b3fe5d53dce0c8ae611d04f389adf47032362bb9aba4d8973ab501c90c221", got "sha256:4fa827ab4d209213b5cb961855a7f681df6e8fbd107f16e9b584298a86dba193"

[docker-robott]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30d of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

[docker-robott]

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked