Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker push fails with "file integrity checksum failed" seemingly at random #33991

Closed
ColinChartier opened this issue Jul 6, 2017 · 15 comments
Closed

Comments

@ColinChartier
Copy link

@ColinChartier ColinChartier commented Jul 6, 2017

Docker save and docker push both fail for some images seemingly at random with "file integrity checksum failed"

I use a custom-made tool to build a bunch of docker images at once, and on basically every docker version I've used (CE 17.06.0, 1.12.1, 1.12.6, 1.11.1), I get an error where some of the built images can't be pushed or saved with the message file integrity checksum failed for "services/base_server/lib/apt/libgtk2.0/libasan2_5.4.0-6ubuntu1~16.04.4_amd64.deb" or equivalent (always a .deb file, but that might just be because the majority of files are .deb)

Steps to reproduce the issue:
I can't give repro steps outside of a single computer, the build system seems to work everywhere else.

Describe the results you received:
file integrity checksum failed for "services/base_server/lib/apt/libgtk2.0/libasan2_5.4.0-6ubuntu1~16.04.4_amd64.deb"

Additional information you deem important (e.g. issue happens only occasionally):
I've tried overlay2 and aufs, so this doesn't seem to be a graph driver bug. Others at my company also don't get this bug (with the same environment, OS, and docker version)

I tried upgrading my kernel, then my entire OS (ubuntu 14.04 to ubuntu 16.04) and multiple docker versions, but I might just be cursed.

Output of docker version:

Client:
 Version:      17.06.0-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:23:31 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.06.0-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:19:04 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 6
 Running: 4
 Paused: 0
 Stopped: 2
Images: 427
Server Version: 17.06.0-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 421
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfb82a876ecc11b5ca0977d1733adbe58599088a
runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.8.0-58-generic
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 62.85GiB
ID: 46HT:IF2S:6OKV:J7K3:VQUG:ZZTN:PPC6:ZJFR:HRO2:P5FR:3WN5:PFVC
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):

@ColinChartier

This comment has been minimized.

Copy link
Author

@ColinChartier ColinChartier commented Jul 6, 2017

Fixed: This was because one of my ram sticks was bad, and a single bit swap was happening on operations with a lot of writes.

I'm not going to close this because I think a (try a memory check) message would save a lot of headache if this ever happens again.

@thaJeztah

This comment has been minimized.

Copy link
Member

@thaJeztah thaJeztah commented Jul 10, 2017

Good to hear you resolved the issue.

I don't think we should add such a message; it's possible the checksum failed for other reasons (such as disk failure); mentioning just one possible cause (and doing a completely wild guess) may send users in the wrong direction.

I'll close this issue, but let me know if you feel strongly about adding more information to the error message.

@thaJeztah thaJeztah closed this Jul 10, 2017
@RoshanJeewantha

This comment has been minimized.

Copy link

@RoshanJeewantha RoshanJeewantha commented Sep 13, 2017

Hi,

I was having something similar when using docker push.

file integrity checksum failed for "lib/systemd/systemd-logind"

My image was over 100 MB and using Docker Version 17.06.2-ce-win27.

To fix that I had to increase the allocated memory for Docker engine. Go to Docker settings > Advanced.

Regards,
RJ

@glyph

This comment has been minimized.

Copy link

@glyph glyph commented Jan 16, 2018

I've also encountered this issue, using Docker for Mac:

Client:
 Version:	17.12.0-ce
 API version:	1.35
 Go version:	go1.9.2
 Git commit:	c97c6d6
 Built:	Wed Dec 27 20:03:51 2017
 OS/Arch:	darwin/amd64

Server:
 Engine:
  Version:	17.12.0-ce
  API version:	1.35 (minimum version 1.12)
  Go version:	go1.9.2
  Git commit:	c97c6d6
  Built:	Wed Dec 27 20:12:29 2017
  OS/Arch:	linux/amd64
  Experimental:	true

(with Experimental being both true and false).

Following @RoshanJeewantha's advice to increase the RAM allocated to the docker VM seems to have resolved the issue. Is the OOM killer nuking things during the build process? This is the only thing I can think of. (Extensive hardware tests indicate that the machines I'm running Docker on most definitely do not have bad RAM.)

@jamshid

This comment has been minimized.

Copy link
Contributor

@jamshid jamshid commented Jan 17, 2018

I don't what caused this in my case, don't think it's bad or insufficient memory. One of my base images just had a bad layer or checksum I guess, but it has otherwise been working fine.

I noticed the error because a push to a registry (even different registries) was failing for a particular image, always at the same layer, so I tried to just save the image and that also showed the error:

$ docker save -o /tmp/badrtjar.tar badrtjar:gateway-build
Error response from daemon: file integrity checksum failed for "usr/lib/jvm/java-1.8.0-openjdk-1.8.0.151-5.b12.el7_4.x86_64/jre/lib/rt.jar"

I tagged the bad image, created on Docker for Mac Edge Version 17.12.0-ce-mac45 (21669). Maybe there's some way to diagnose the problem?

Anyway I forced a rebuild of the base image to workaround this weird kinda disturbing problem. I'm worried there is a bug, that it's more than "bad hardware". Seems to turn up often e.g. https://forums.docker.com/t/file-integrity-checksum-failed-on-docker-save/14715/10.

@thaJeztah

This comment has been minimized.

Copy link
Member

@thaJeztah thaJeztah commented Jan 17, 2018

@jamshid may be worth reporting in the Docker for Mac issue tracker if you haven't done so yet; https://github.com/docker/for-mac/issues

@chicofilho

This comment has been minimized.

Copy link

@chicofilho chicofilho commented Jan 22, 2018

Had exactly the same problema. It worked after I deleted the images and rebuilt

@doudouOUC

This comment has been minimized.

Copy link

@doudouOUC doudouOUC commented Jan 27, 2018

I also met this poblem. if anyone have answers , please tell me.

@ankitshah009

This comment has been minimized.

Copy link

@ankitshah009 ankitshah009 commented Jan 31, 2018

Facing a similar problem and unable to solve the problem using the solutions provided in this thread.
Please inform on how to resolve this problem. Rebuilding is not an option as it took me days to reach this stage with the docker image.

@artx19

This comment has been minimized.

Copy link

@artx19 artx19 commented Feb 7, 2018

docker system prune -a solved the problem

@donghl

This comment has been minimized.

Copy link

@donghl donghl commented Jul 16, 2018

docker system prune -a

@jacobtomlinson

This comment has been minimized.

Copy link
Contributor

@jacobtomlinson jacobtomlinson commented Jul 25, 2018

Also having this issue. Seems to fail on a layer which is 3GB. I have allocated 6GB of RAM and already run docker system prune -a. Still no luck!

@thaJeztah

This comment has been minimized.

Copy link
Member

@thaJeztah thaJeztah commented Jul 25, 2018

@jacobtomlinson could you open a new ticket with details about your setup?

@vedrusss

This comment has been minimized.

Copy link

@vedrusss vedrusss commented Aug 29, 2018

I wasn't able to find the solution how to increase memory for docker engine in ubuntu 16.04 (no gui provided).
docker system prune -a looked very tricky to me, I knew it'll remove ALL my containers and images.
Having no another decisions what to do I tried it, lost my images. Downloaded new ones, build new ones and...
it helped: after 5-8 mins wait I've got my saved image :)

@jecsham-icte

This comment has been minimized.

Copy link

@jecsham-icte jecsham-icte commented Oct 17, 2019

Rebuild the image is a good solution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.