Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sudo apt-get update fails on GCE Trusty #6881

Closed
BanzaiMan opened this issue Nov 16, 2016 · 37 comments
Closed

sudo apt-get update fails on GCE Trusty #6881

BanzaiMan opened this issue Nov 16, 2016 · 37 comments
Assignees
@BanzaiMan
Labels
build environment travis-build

Comments

@BanzaiMan
Copy link
Contributor

BanzaiMan commented Nov 16, 2016 •
edited

We have received a few reports of sudo apt-get update failing due to TLS handshake error with apt.dockerproject.org.

Affected repositories would have:

sudo: required
dist: trusty

or

sudo: required
services:
  - docker

The error would manifest in this way:

W: Failed to fetch https://apt.dockerproject.org/repo/dists/ubuntu-trusty/main/binary-amd64/Packages  gnutls_handshake() failed: Handshake failed

E: Some index files failed to download. They have been ignored, or old ones used instead.

This is probably related to moby/moby#16941. (Recent comments indicate that the Docker team is aware of the issue and is working with AWS to investigate further.)

We plan to add a temporary fix until the issue is resolved.
@BanzaiMan BanzaiMan self-assigned this Nov 16, 2016
@BanzaiMan BanzaiMan changed the title sudo apt-get update fails sudo apt-get update fails on GCE Nov 16, 2016
@BanzaiMan BanzaiMan changed the title sudo apt-get update fails on GCE sudo apt-get update fails on GCE Trusty Nov 16, 2016
@jskeet jskeet mentioned this issue Nov 16, 2016
Merged
@BanzaiMan
Copy link
Contributor Author

BanzaiMan commented Nov 16, 2016 •
edited

UPDATE: The following workaround may not work. See comments below.

For those who want to fix it in their build right now, the one we are planning as a hot fix amounts to the following:

before_install:
  - echo "54.230.142.136 apt.dockerproject.org" | sudo tee -a /etc/hosts
  ⋮ # rest

@BanzaiMan
Copy link
Contributor Author

moby/moby#28474 is the current issue on Docker.

@The-Compiler
Copy link

What about updating Travis-CI status in the meantime so people are aware of this?

@BanzaiMan
Copy link
Contributor Author

@The-Compiler Thanks. We'll do that shortly.

@solarce
Copy link

solarce commented Nov 16, 2016

We've opened a status incident to help inform people of the issue, https://www.traviscistatus.com/incidents/gmd6p074w061

@BanzaiMan
Copy link
Contributor Author

Sorry, the fix earlier had a bad command. Instead of | sudo tee, you'd need | sudo tee -a. I fixed the comment above.

@BanzaiMan BanzaiMan mentioned this issue Nov 16, 2016
Closed
BanzaiMan added a commit to travis-ci/travis-build that referenced this issue Nov 16, 2016
@BanzaiMan
On sudo-enabled builds, add numeric IP address for apt.dockerproject.org
df203d4 
There is an issue with docker team's DNS where TLS handshake fails when
we try to update packate index from apt.dockerproject.org.

See
travis-ci/travis-ci#6881 and
moby/moby#28474 for more details.
@miketheman miketheman mentioned this issue Nov 16, 2016
Closed
@jonathansantilli
Copy link

@BanzaiMan Thanks but your workaround seems not to be working either :(

@BanzaiMan
Copy link
Contributor Author

@jonathansantilli Oh, no! I'll take a deeper look.

@richelbilderbeek
Copy link

richelbilderbeek commented Nov 16, 2016 •
edited

I can second that @BanzaiMan ! Here is the output of my build log:

too_bad

Good luck!

@richelbilderbeek richelbilderbeek mentioned this issue Nov 16, 2016
Closed
@BanzaiMan
Copy link
Contributor Author

Seems that switching off https in favor of http appears to be more reliable.

@BanzaiMan
Copy link
Contributor Author

I spoke too soon. apt.dockerproject.org is not listening on port 80.

@jonnor
Copy link

jonnor commented Nov 16, 2016

I was affected by this from a "sudo apt-get install ..." line in before_install. Changing to use the Apt addon instead solved this case: imgflo/imgflo-server@575337a

Probably will not help with the case of actually installing Docker.

@BanzaiMan
Copy link
Contributor Author

We are going to disable the Docker repository for now, until we can resolve the issue.

This way, sudo apt-get install would work, as long as the packages do not come from apt.dockerproject.org.

Thanks for your understanding.

BanzaiMan added a commit to travis-ci/travis-build that referenced this issue Nov 16, 2016
@BanzaiMan
Remove apt source for apt.dockerproject.org
edec9c2 
There is an issue with this APT source preventing use of `apt`.

See
travis-ci/travis-ci#6881 for more details.
@BanzaiMan BanzaiMan mentioned this issue Nov 16, 2016
Merged
BanzaiMan added a commit to travis-ci/travis-build that referenced this issue Nov 16, 2016
@BanzaiMan
Remove apt source for apt.dockerproject.org
2d85eaf 
There is an issue with this APT source preventing use of `apt`.

See
travis-ci/travis-ci#6881 for more details.
BanzaiMan added a commit to travis-ci/travis-build that referenced this issue Nov 16, 2016
@BanzaiMan
Remove apt source for apt.dockerproject.org
11be1d5 
There is an issue with this APT source preventing use of `apt`.

See
travis-ci/travis-ci#6881 for more details.
lzjqsdd added a commit to lzjqsdd/scikit-learn-doc-cn that referenced this issue Nov 16, 2016
@lzjqsdd
Fix travisCI Error
9d3f127 
travis-ci/travis-ci#6881

`sudo apt-get update` fails on GCE Trusty
@thaJeztah
Copy link

Just got informed that AWS identified a possible cause and is rolling out updates. It may take some time for those to be available everywhere. Hopefully the issue will be resolved by that.

BanzaiMan added a commit to BanzaiMan/travis_production_test that referenced this issue Nov 16, 2016
@BanzaiMan
run apt-get update on GCE Trusty
9fbf8d7 
See travis-ci/travis-ci#6881
@BanzaiMan
Copy link
Contributor Author

@thaJeztah Thank you for the update and for keeping us in the loop.

@lzjqsdd lzjqsdd mentioned this issue Nov 16, 2016
Merged
@jwhui jwhui mentioned this issue Nov 16, 2016
Closed
@BanzaiMan
Copy link
Contributor Author

Hi, everyone.

We've deployed a fix to remove the APT source for apt.dockerproject.org to production. apt-get commands should work now.

Please let us know if you continue to see problems.

Thank you for your patience.

@tobiaskley
Copy link

Hi, I still see an error, but I am not certain if it is related. It sounds similar. Thanks for considering this and best regards, Tobias.

@trosendal
Copy link

Me too:
https://api.travis-ci.org/jobs/176393653/log.txt?deansi=true

@BanzaiMan
Copy link
Contributor Author

@tobiaskley Yours is related, but slightly different, because the issue lies with an r-project.org source. This is tracked separately with #6882.

@mingp
Copy link

mingp commented Nov 16, 2016

We continue to encounter this issue in one of our projects, now occurring with a different APT source. This is trying to install NodeJS from the NodeSource repos.

Step 2 : RUN curl -sL https://deb.nodesource.com/setup_4.x | bash -
 ---> Running in ad0e31b95c0c
## Installing the NodeSource Node.js v4.x LTS Argon repo...
## Populating apt-get cache...
+ apt-get update
Get:1 http://security.debian.org jessie/updates InRelease [63.1 kB]
Ign http://deb.debian.org jessie InRelease
Get:2 http://deb.debian.org jessie-updates InRelease [145 kB]
Get:3 http://deb.debian.org jessie Release.gpg [2373 B]
Get:4 http://security.debian.org jessie/updates/main amd64 Packages [408 kB]
Get:5 http://deb.debian.org jessie Release [148 kB]
Get:6 http://deb.debian.org jessie-updates/main amd64 Packages [17.6 kB]
Get:7 http://deb.debian.org jessie/main amd64 Packages [9064 kB]
Fetched 9848 kB in 2s (3727 kB/s)
Reading package lists...
## Installing packages required for setup: apt-transport-https lsb-release...
+ apt-get install -y apt-transport-https lsb-release > /dev/null 2>&1
## Confirming "jessie" is supported...
+ curl -sLf -o /dev/null 'https://deb.nodesource.com/node_4.x/dists/jessie/Release'
## Adding the NodeSource signing key to your keyring...
+ curl -s https://deb.nodesource.com/gpgkey/nodesource.gpg.key | apt-key add -
OK
## Creating apt sources list file for the NodeSource Node.js v4.x LTS Argon repo...
+ echo 'deb https://deb.nodesource.com/node_4.x jessie main' > /etc/apt/sources.list.d/nodesource.list
+ echo 'deb-src https://deb.nodesource.com/node_4.x jessie main' >> /etc/apt/sources.list.d/nodesource.list
## Running `apt-get update` for you...
+ apt-get update
Ign https://deb.nodesource.com jessie InRelease
Hit http://security.debian.org jessie/updates InRelease
Ign https://deb.nodesource.com jessie Release.gpg
Ign https://deb.nodesource.com jessie Release
Ign http://deb.debian.org jessie InRelease
Hit http://deb.debian.org jessie-updates InRelease
Get:1 http://security.debian.org jessie/updates/main amd64 Packages [408 kB]
Hit http://deb.debian.org jessie Release.gpg
Hit http://deb.debian.org jessie Release
Get:2 http://deb.debian.org jessie-updates/main amd64 Packages [17.6 kB]
Err https://deb.nodesource.com jessie/main Sources
  gnutls_handshake() failed: Handshake failed
Err https://deb.nodesource.com jessie/main amd64 Packages
  gnutls_handshake() failed: Handshake failed
Get:3 http://deb.debian.org jessie/main amd64 Packages [9064 kB]
W: Failed to fetch https://deb.nodesource.com/node_4.x/dists/jessie/main/source/Sources  gnutls_handshake() failed: Handshake failed
W: Failed to fetch https://deb.nodesource.com/node_4.x/dists/jessie/main/binary-amd64/Packages  gnutls_handshake() failed: Handshake failed

@solarce
Copy link

solarce commented Nov 16, 2016

@tobiaskley @trosendal that is Failed to fetch https://cloud.r-project.org/bin/linux/ubuntu/precise/Sources gnutls_handshake() failed: A TLS fatal alert has been received. which is due to cloud.r-project.org also being hosted on AWS CloudFront.

Ultimately the issue is with the CDN provider and we're waiting on them to fix the issue for everyone

@thaJeztah
Copy link

@mingp same issue for nodesource.com they are tracking this in nodesource/distributions#388

@solarce
Copy link

solarce commented Nov 16, 2016

@mingp that is Err https://deb.nodesource.com jessie/main Sources gnutls_handshake() failed: Handshake failed which is due to deb.nodesource.com also being hosted on AWS CloudFront.

Ultimately the issue is with the CDN provider and we're waiting on them to fix the issue for everyone

@dewittpe dewittpe mentioned this issue Nov 16, 2016
Closed
@mingp
Copy link

mingp commented Nov 16, 2016

@thaJeztah @solarce Thank you for the clarification. We'll go follow the other issue.

@tobiaskley
Copy link

@BanzaiMan @solarce thank you for explaining it so very well to me. If I understand correctly, there is nothing I can do about it now? Best, Tobias.

@BanzaiMan
Copy link
Contributor Author

@tobiaskley Not very much, unfortunately. You might want to alert r-project.org maintainers to the issue, if you wish.

@BanzaiMan BanzaiMan mentioned this issue Nov 16, 2016
Closed
@eddelbuettel
Copy link

For what it is worth my Rcpp build on Travis now succeeded. Yay.

I use a personal variant of the 'older' R Travis setup using explicit apt-get calls to Ubuntu, CRAN, launchpad, .... All fine now. Thanks all!

@shouze
Copy link

shouze commented Nov 16, 2016

@BanzaiMan ok looks like it's been fixed for docker, would you please revert your patch please?

@solarce
Copy link

solarce commented Nov 16, 2016

@shouze Hiro is working on deploying that travis-ci/travis-build#892

@BanzaiMan
Copy link
Contributor Author

Hi, everyone. apt.dockerproject.org has been restored with travis-ci/travis-build#892.

This has been deployed to production.

Let us know if you continue to have problems.

Again, thank you for your patience.

@BanzaiMan
Copy link
Contributor Author

I should've asked the opposite. Let us know if you no longer see the problem. (So that we can close this issue!)

@shouze
Copy link

shouze commented Nov 16, 2016

It's working perfectly on an in progress build with 10 jobs, all the 10 jobs have successfully installed docker 1.12.3! ;)

@BanzaiMan
Copy link
Contributor Author

@shouze Thanks for the confirmation. I'm closing this now.

@The-Compiler
Copy link

@puolukkamaenpuu ...huh?

@jenkoian jenkoian mentioned this issue Mar 30, 2017
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@BanzaiMan BanzaiMan

Labels
build environment travis-build
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
13 participants
@solarce @BanzaiMan @jonnor @shouze @The-Compiler @eddelbuettel @jonathansantilli @thaJeztah @richelbilderbeek @trosendal @mingp @tobiaskley and others