Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hash sum mismatch #23203

Closed
herrbischoff opened this issue Jun 2, 2016 · 90 comments
Closed

Hash sum mismatch #23203

herrbischoff opened this issue Jun 2, 2016 · 90 comments

Comments

@herrbischoff
Copy link

@herrbischoff herrbischoff commented Jun 2, 2016

Steps to reproduce the issue:

  1. Running apt-get update

Describe the results you received:
Running apt-get update on Debian Stretch just now results in

Err:2 https://apt.dockerproject.org/repo debian-stretch/main amd64 Packages
  Hash Sum mismatch

as well as

E: Failed to fetch https://apt.dockerproject.org/repo/dists/debian-stretch/main/binary-amd64/Packages.bz2  Hash Sum mismatch

I have cleaned the apt caches and tried again with the same result. Also, I'm not using a proxy.

Describe the results you expected:
No error.

@herrbischoff herrbischoff changed the title Has sum mismatch Hash sum mismatch Jun 2, 2016
@herrbischoff
Copy link
Author

@herrbischoff herrbischoff commented Jun 2, 2016

Appears to be related to #23202.

@mads79
Copy link

@mads79 mads79 commented Jun 2, 2016

Same problem on Debian Trusty

W: Failed to fetch https://apt.dockerproject.org/repo/dists/ubuntu-trusty/main/binary-amd64/Packages  Hash Sum mismatch

E: Some index files failed to download. They have been ignored, or old ones used instead.

@nielsvandermolen
Copy link

@nielsvandermolen nielsvandermolen commented Jun 2, 2016

Similar problem on Travis CI with Ubuntu package. It was working an hour ago.

https://travis-ci.org/goalgorilla/drupal_social/builds/134719276

W: There is no public key available for the following key IDs:
1397BC53640DB551
W: Failed to fetch https://apt.dockerproject.org/repo/dists/ubuntu-trusty/main/binary-amd64/Packages  Hash Sum mismatch
E: Some index files failed to download. They have been ignored, or old ones used instead.
@zjeraar
Copy link

@zjeraar zjeraar commented Jun 2, 2016

Same on Debian Jessie:
W: Failed to fetch https://apt.dockerproject.org/repo/dists/debian-jessie/main/binary-amd64/Packages Hash Sum mismatch

@zjeraar
Copy link

@zjeraar zjeraar commented Jun 2, 2016

Can easily be repoduced in a contatiner also:

FROM debian:8.4

RUN \
  apt-get update && \
  apt-get install -yq apt-transport-https ca-certificates && \
  apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D && \
  echo "deb https://apt.dockerproject.org/repo debian-jessie main" > /etc/apt/sources.list.d/docker.list && \
  apt-get update
@Vanuan
Copy link

@Vanuan Vanuan commented Jun 2, 2016

What about apt-get clean? Does it help?

@ivanfeob
Copy link

@ivanfeob ivanfeob commented Jun 2, 2016

@Vanuan No, already tried.

@GordonTheTurtle
Copy link

@GordonTheTurtle GordonTheTurtle commented Jun 2, 2016

USER POLL

The best way to get notified of updates is to use the Subscribe button on this page.

Please don't use "+1" or "I have this too" comments on issues. We automatically
collect those comments to keep the thread short.

The people listed below have upvoted this issue by leaving a +1 comment:

@ViGo5190

@mcapuccini
Copy link

@mcapuccini mcapuccini commented Jun 2, 2016

Same here!

Failed to fetch https://apt.dockerproject.org/repo/dists/ubuntu-trusty/main/binary-amd64/Packages  Hash Sum mismatch
@gillesdemey
Copy link

@gillesdemey gillesdemey commented Jun 2, 2016

Things we've tried so far:

Re-add the GPG key
curl -fsSL https://get.docker.com/gpg | sudo apt-key add -

Blow the lists cache
sudo rm -rf /var/lib/apt/lists/*

Apt clean
apt-clean

None of them have resolved the issue

@jraut
Copy link

@jraut jraut commented Jun 2, 2016

Tried to install via apt. Checksum mismatch with the following file:

https://apt.dockerproject.org/repo/dists/ubuntu-trusty/main/binary-amd64/Packages

Tried the following procedures which did not help:

apt-clean
sudo rm -rf /var/lib/apt/lists/*
@Vanuan
Copy link

@Vanuan Vanuan commented Jun 2, 2016

Maybe this would help apt-get -o Debug::pkgAcquire::Auth=true update figure out the issue.

mcapuccini pushed a commit to mcapuccini/SparkNow that referenced this issue Jun 2, 2016
@thoastbrot
Copy link

@thoastbrot thoastbrot commented Jun 2, 2016

Release contains:
MD5Sum:
49df2d605bb5914873fd826f7e7e8c6f 4917 Packages.bz2

InRelease contains:
b013253c327e2bc4be87825f02936344 4915 main/binary-amd64/Packages.bz2

the latter has been updated today, Date: Thu, 02 Jun 2016 11:06:54 UTC
while Release is from yesterday.

@gillesdemey
Copy link

@gillesdemey gillesdemey commented Jun 2, 2016

Running apt-get -o Debug::pkgAcquire::Auth=true update on Ubuntu 14.04 yields

[Waiting for headers]201 URI Done: bzip2:/var/lib/apt/lists/partial/apt.dockerproject.org_repo_dists_ubuntu-trusty_main_binary-amd64_Packages
RecivedHash: SHA512:d6ca1f74e876031161d1abd6cf9ad0b45f60b19876468cfcf9cacd4956dfd13be43147227a8daa5536f1455bb75b353b178942bc1843d11f0188d00117483912
ExpectedHash: SHA512:d07a3f2c42a9b213e3f03f2f11c08154512baa9fbbaed19f3601865634b82cfdde0e65151a24e523017f29ecfd08a1dfc0af2c2117b025c46d683160892b0de6


https://apt.dockerproject.org/repo/dists/ubuntu-trusty/main/binary-amd64/Packages: 
Computed Hash: SHA512:d6ca1f74e876031161d1abd6cf9ad0b45f60b19876468cfcf9cacd4956dfd13be43147227a8daa5536f1455bb75b353b178942bc1843d11f0188d00117483912  
Expected Hash: SHA512:d07a3f2c42a9b213e3f03f2f11c08154512baa9fbbaed19f3601865634b82cfdde0e65151a24e523017f29ecfd08a1dfc0af2c2117b025c46d683160892b0de6
@herrbischoff
Copy link
Author

@herrbischoff herrbischoff commented Jun 2, 2016

Relevant output of apt-get -o Debug::pkgAcquire::Auth=true update:

Got Codename: debian-stretch
Expecting Dist:
Transformed Dist:
Signature verification succeeded: /var/lib/apt/lists/partial/apt.dockerproject.org_repo_dists_debian-stretch_InRelease
Get:2 https://apt.dockerproject.org/repo debian-stretch/main amd64 Packages [4,941 B]
0% [Connecting to ftp.de.debian.org] [Connecting to security.debian.org] [Connecting to mirror.netcologne.de] [Connecting to packages.dotdeb.org] [Connecting to www.deb-multimedia.org] [Connecting to ftp-stud.hs-esslingen.de] [Connecting201 URI Done: https://apt.dockerproject.org/repo/dists/debian-stretch/main/binary-amd64/Packages.bz2
ReceivedHash:
    - SHA512:14844ddc767052951fb68eabc19a1935fb930c798d64fd86ace0dcce3aad2af887fc091ad90897a52f341f65dadac5f0dc31a35f9c70b5bcc582314187a336cf
    - SHA256:0cee3ef5330e133cc6dfbf3d34f118806ce685a1ded4210c5c4f7ef7b43e9867
    - SHA1:bcf84731c3d9fe4355ce73b3cd756decbf9b67cb
    - MD5Sum:c99614887831f4d020e682c8222fe49b
    - Checksum-FileSize:4933
ExpectedHash:
    - Checksum-FileSize:4941
    - SHA512:5de62937921a32be2e9cf14f65e6adda3499fd648f37ab5ccc9547a03d211be66c3a5cd15f272e5a3f0abc53fec3903f646410337917e4201bf2a7ed5ac8581d
    - SHA256:ebc0ec8921482f40bdcf1fa9a7f39b7bd198d81a769643723201c109b3b617ea
    - SHA1:a61818ebafdccbccdfdeee5e550b9241b8c32722
    - MD5Sum:9cd9390adc1849ba5923a70d92af1927
@nielsvandermolen
Copy link

@nielsvandermolen nielsvandermolen commented Jun 2, 2016

https://travis-ci.org/goalgorilla/drupal_social/builds/134730044

Get:11 https://apt.dockerproject.org ubuntu-trusty/main amd64 Packages
201 URI Done: https://apt.dockerproject.org/repo/dists/ubuntu-trusty/main/binary-amd64/Packages.bz2
RecivedHash: SHA512:36e068ae0288732c51bd971ee74b6d27c8707f4d11840afcca617884de82e8c533c5259d8d97bb297966424bc58ac219879f4f5d12c4abe073799bb658f4bd87
ExpectedHash: SHA512:d07a3f2c42a9b213e3f03f2f11c08154512baa9fbbaed19f3601865634b82cfdde0e65151a24e523017f29ecfd08a1dfc0af2c2117b025c46d683160892b0de6
@bmoorthamers
Copy link

@bmoorthamers bmoorthamers commented Jun 2, 2016

I get on Ubuntu Wily 15.10

E: Unable to locate package docker-engine

I got the same before on Ubuntu Xenial 16.04. Is docker even added to the Xenial repo yet?

@ThomasSteinbach
Copy link

@ThomasSteinbach ThomasSteinbach commented Jun 2, 2016

Relevant output of apt-get -o Debug::pkgAcquire::Auth=true update :

Got Codename: ubuntu-xenial
Expecting Dist: 
Transformed Dist: 
Signature verification succeeded: /var/lib/apt/lists/partial/apt.dockerproject.org_repo_dists_ubuntu-xenial_InRelease
Holen:12 https://apt.dockerproject.org/repo ubuntu-xenial/main amd64 Packages [1.712 B]
Ign:12 https://apt.dockerproject.org/repo ubuntu-xenial/main amd64 Packages
Holen:12 https://apt.dockerproject.org/repo ubuntu-xenial/main amd64 Packages [1.430 B]
Ign:12 https://apt.dockerproject.org/repo ubuntu-xenial/main amd64 Packages
Holen:12 https://apt.dockerproject.org/repo ubuntu-xenial/main amd64 Packages [4.815 B]
100% [12 Packages 4.815 B/4.815 B 100%]201 URI Done: https://apt.dockerproject.org/repo/dists/ubuntu-xenial/main/binary-amd64/Packages
ReceivedHash:
    - SHA512:c7883bb7a1d0b5162431576408644a85003be4601724b6f2db275cd4b603a61f8dcd924e80158c40413942519c8a528f7940ffbe5370daa4b0a0d867afe3163d
    - SHA256:de12840d76e571cb6f42e63ac570c59d5332d772fb295b6919d12214052bfa6b
    - SHA1:9f9c05d3b7d8ca13e9e03c4f0f12757816f02301
    - MD5Sum:65e1f5c451c230a091118b468c31bae7
    - Checksum-FileSize:4815
ExpectedHash:
    - Checksum-FileSize:4815
    - SHA512:2becf6c2b9aae5b6823ea6d9f12988e22905a87a9a03fed844a761698eee614899d7b039e081e0b330539e716918b75e87a96c287a5efbe9fc3e847d44657798
    - SHA256:f4ae20e2259740699fba3a79dd7fb557c472d172b578798071274f7ba4c400f3
    - SHA1:8f34563e8170c5698dc7ba04dd3cf4c8a93100cf
    - MD5Sum:31d143b7a15a8a38bc92a7559c995078
@thoastbrot
Copy link

@thoastbrot thoastbrot commented Jun 2, 2016

can we agree upon the fact that the hashsums are incorrect/repo needs administrative action?

@qzio
Copy link
Contributor

@qzio qzio commented Jun 2, 2016

I worked around this by downloading the new package manually and installing using dpkg

curl -OL https://apt.dockerproject.org/repo/pool/main/d/docker-engine/docker-engine_1.11.2-0~trusty_amd64.deb
dpkg -i docker-engine*.deb
@OJFord
Copy link

@OJFord OJFord commented Jun 2, 2016

Unfortunately dpkg installation doesn't seem to play well on Travis.

@herrbischoff
Copy link
Author

@herrbischoff herrbischoff commented Jun 2, 2016

Also, this is what happens to me when manually installing on Debian Stretch, using https://apt.dockerproject.org/repo/pool/main/d/docker-engine/docker-engine_1.11.2-0~stretch_amd64.deb:

$ sudo systemctl status docker.service
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2016-06-02 14:46:59 CEST; 58s ago
     Docs: https://docs.docker.com
 Main PID: 31269 (code=exited, status=1/FAILURE)

Jun 02 14:46:58 penny systemd[1]: Starting Docker Application Container Engine...
Jun 02 14:46:58 penny docker[31269]: time="2016-06-02T14:46:58.553905409+02:00" level=info msg="New containerd process, pid: 31293\n"
Jun 02 14:46:59 penny docker[31269]: time="2016-06-02T14:46:59.659258835+02:00" level=error msg="[graphdriver] prior storage driver \"aufs\" failed: driver not supported"
Jun 02 14:46:59 penny docker[31269]: time="2016-06-02T14:46:59.659395935+02:00" level=fatal msg="Error starting daemon: error initializing graphdriver: driver not supported"
Jun 02 14:46:59 penny systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Jun 02 14:46:59 penny docker[31269]: time="2016-06-02T14:46:59+02:00" level=info msg="stopping containerd after receiving terminated"
Jun 02 14:46:59 penny systemd[1]: Failed to start Docker Application Container Engine.
Jun 02 14:46:59 penny systemd[1]: docker.service: Unit entered failed state.
Jun 02 14:46:59 penny systemd[1]: docker.service: Failed with result 'exit-code'.

Update: As I somehow expected, this was an unrelated problem. I fixed it by running rm -rf /var/lib/docker/aufs after finding this. So the manual install works for me for the time being.

@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Jun 2, 2016

ping @mlaventure @tiborvass PTAL!

@akersj
Copy link

@akersj akersj commented Jun 2, 2016

ETA?

@rvetere
Copy link

@rvetere rvetere commented Jun 2, 2016

yes we need an ETA too, it's pretty urgent - our complete travis build chain is dead now -.-

@simos
Copy link

@simos simos commented Jun 2, 2016

Here are the relevant files for xenial,
https://apt.dockerproject.org/repo/dists/ubuntu-xenial/main/binary-amd64/

InRelease        02-Jun-2016 11:06  2.6K
Packages          02-Jun-2016  2:38  4.8K
Packages.bz2  02-Jun-2016  2:38  1.7K
Packages.gz    02-Jun-2016  2:38  1.4K
Release            02-Jun-2016  3:43  1.7K
Release.gpg    02-Jun-2016  3:43  801

We can see that these files have been regenerated earlier today.
The checksums (hashes) for these files should match what is inside the signed InRelease file of checksums.

In the InRelease file (https://apt.dockerproject.org/repo/dists/ubuntu-xenial/main/binary-amd64/InRelease), it says that this file was generated on Date: Thu, 02 Jun 2016 03:43:32 UTC . However, the timestamp as shown by the Web server is 02-Jun-2016 11:06.

Among the several causes for Hash Sum Mismatch, this one is about some weird update of InRelease with wrong checksums. In addition, InRelease lists the Release as being at 0 bytes long.

@bmoorthamers
Copy link

@bmoorthamers bmoorthamers commented Jun 2, 2016

@simos So this should work on Xenial now? I thought docker still did not work on Xenial and we had to go back to Wily. (then again, I am an Ubuntu user since today so what do I know)

@simos
Copy link

@simos simos commented Jun 2, 2016

@bmoorthamers You can manually check which repositories have mismatched hashes. See my post above. At least trusty, wily and xenial are currently (probably since earlier in the morning) affected.

@cpuguy83
Copy link
Collaborator

@cpuguy83 cpuguy83 commented Jun 2, 2016

@vadviktor Does not work at Docker nor maintain the project.

@quater
Copy link

@quater quater commented Jun 2, 2016

It appears to be working for Ubuntu Xenial now.

@okket
Copy link

@okket okket commented Jun 2, 2016

@oryband
Copy link

@oryband oryband commented Jun 2, 2016

for everybody raging over this downtime: here's a cute deer picture to calm down and pass the time in the meanwhile:

@mech422
Copy link

@mech422 mech422 commented Jun 2, 2016

Trusty appears to be back up

@shykes
Copy link
Contributor

@shykes shykes commented Jun 2, 2016

Hi everyone. I work at Docker.

First, my apologies for the outage. I consider our package infrastructure as critical infrastructure, both for the free and commercial versions of Docker. It's true that we offer better support for the commercial version (it's one if its features), but that should not apply to fundamental things like being able to download your packages.

The team is working on the issue and will continue to give updates here. We are taking this seriously.

Some of you pointed out that the response time and use of communication channels seem inadequate, for example the @dockerststus bot has not mentioned the issue when it was detected. I share the opinion but I don't know the full story yet; the post-mortem will tell us for sure what went wrong. At the moment the team is focusing on fixing the issue and I don't want to distract them from that.

Once the post-mortem identifies what went wrong, we will take appropriate corrective action. I suspect part of it will be better coordination between core engineers and infrastructure engineers (2 distinct groups within Docker).

Thanks and sorry again for the inconvenience.

@mech422
Copy link

@mech422 mech422 commented Jun 2, 2016

heh - got the catalog but the package is missing - guess I'll have another cofee :-)
@shykes Thanks for the update - lousy way to start your morning ...
Hope the day gets better from here

@oryband
Copy link

@oryband oryband commented Jun 2, 2016

I'm sad that my deer picture got less +1s than the official answer.

@Rocklviv
Copy link

@Rocklviv Rocklviv commented Jun 2, 2016

I have already installed docker with https://get.docker.com | sh without any error.
Look's like the guys from Docker fixed the issue,

@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Jun 2, 2016

We localized the cause of the issue, and if should be resolved now, please try again.

It may be needed to clear the apt-cache;

apt-get clean && apt-get update

@snario
Copy link

@snario snario commented Jun 2, 2016

Thanks for the fix @thaJeztah

@LouisKottmann
Copy link

@LouisKottmann LouisKottmann commented Jun 2, 2016

Well, that was quick for an unexpected issue, thanks.

@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Jun 2, 2016

@snario you're welcome; can't take credit for the fix, but happy to see it's been sorted out 😅

@rogaha
Copy link
Contributor

@rogaha rogaha commented Jun 2, 2016

👍

@jamescarr
Copy link
Contributor

@jamescarr jamescarr commented Jun 2, 2016

Sadly with this on the top of hacker news there's going to be billions of comments. Big thanks for the quick fix @thaJeztah.

I wonder if we should lock this thread before they show up.

@simos
Copy link

@simos simos commented Jun 2, 2016

Up until now there have been workarounds (either grab the .deb and install with dpkg, temporarily switch to the testing repository, etc). These are not permanent solutions.

A fix means that the source of this problem is resolved and we can mark this issue as Solved.

As posted earlier, you can use a script to get an audio notification as soon as the main docker repositories are fixed,
https://gist.github.com/simos/7ee8258ec17101e44bbfa93606694ede
Other than that, there is not much to do.

@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Jun 2, 2016

@simos see my earlier comment; #23203 (comment) the issue should be resolved

@denibertovic
Copy link
Contributor

@denibertovic denibertovic commented Jun 2, 2016

@thaJeztah I verified that the issue has been resolved. Tested on Ubuntu 15.10. Thanks to all the other Docker folks that helped resolve this issue quickly.

@icecrime
Copy link
Contributor

@icecrime icecrime commented Jun 2, 2016

Thank you all for the reports: we're very sorry for this. We're looking into the details and the timeline of events that lead to this, and we'll make sure it doesn't happen again.

I'm closing the issue, but of course feel free to let me know if you see any remaining quirks.

@icecrime icecrime closed this Jun 2, 2016
@Niceplace
Copy link

@Niceplace Niceplace commented Jun 2, 2016

Ubuntu 14.04 Here, issue solved !

@jalawrence
Copy link

@jalawrence jalawrence commented Jun 2, 2016

Probably shouldn't be surprised, but it is shocking how many people risk their infrastructure with hard dependencies on external repos. I don't even do that with my home systems.

And then complain about Docker having a single point of failure?

@gggeek
Copy link

@gggeek gggeek commented Jun 2, 2016

@jalawrence Docker is the tip of the iceberg...
Did you hear about the recent problems with node.js and one dev pulling out one single package?
I am pretty sure that most php developers using Composer - the defacto package manager for that platform - also do not store complete copies of all their site's dependencies, and the fact that there has been no mishaps so far is more luck than anything.
The problem is that everybody and their dog now depends on $world, and caching all the dependencies locally is a sisyphean task. Shall I cache all of debian, all of packagist, all of cpan, all of rubygems, all of npm within a reverse proxy at my own expenses?
And then: if github, bitbucket or travis are down, what will my developers be able to do anyway? Do I want back to the day when I had to host all of that?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet