Skip to content
This repository has been archived by the owner on Jan 22, 2024. It is now read-only.

Problem with repositories for Ubuntu 16.04 #571

Closed
sb5518 opened this issue Dec 13, 2017 · 46 comments
Closed

Problem with repositories for Ubuntu 16.04 #571

sb5518 opened this issue Dec 13, 2017 · 46 comments

Comments

@sb5518
Copy link

sb5518 commented Dec 13, 2017

When trying to create a docker image with nvidia-docker build the following issue arises:

W: GPG error: http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64 Release: The following signatures were invalid: BADSIG F60F4B3D7FA2AF80 cudatools cudatools@nvidia.com
W: The repository 'http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64 Release' is not signed.
W: GPG error: http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64 Release: The following signatures were invalid: BADSIG F60F4B3D7FA2AF80 cudatools cudatools@nvidia.com
W: The repository 'http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64 Release' is not signed.
E: Failed to fetch http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/Packages.gz Hash Sum mismatch
E: Failed to fetch http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/Packages.gz Hash Sum mismatch
E: Some index files failed to download. They have been ignored, or old ones used instead.

This occurred after compiling an image that was working perfectly until two days ago, and in different environments.
@flx42
Copy link
Member

flx42 commented Dec 13, 2017

Can you try again @sb5518? We released CUDA 9.1 recently on this repository, and sometimes we hit issues with our CDN when a new release happens.

@sb5518
Copy link
Author

sb5518 commented Dec 14, 2017

Unfortunately I just tried again in two different environments and the issue persists. On one of the environments it shows the previous issue and in the other one:

E: Failed to fetch http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/Packages.gz Hash Sum mismatch
E: Some index files failed to download. They have been ignored, or old ones used instead.

@flx42
Copy link
Member

flx42 commented Dec 14, 2017

Which country are you in? It might be a CDN issue since I'm not seeing that from the US.

@flx42
Copy link
Member

flx42 commented Dec 20, 2017

Ping @sb5518

@flx42
Copy link
Member

flx42 commented Jan 3, 2018

No update, closing. If it persists, try contacting a local NVIDIA representative, as it might be a CDN issue related to your region, I can't help you any further.

@flx42 flx42 closed this as completed Jan 3, 2018
@YankeeMarco
Copy link

nvidia technical staff

@Pingmin
Copy link

Pingmin commented Jan 10, 2018 •
edited
Loading

I just got this too on Ubuntu 16.04 by performing "apt-get update" from Shanghai, China :

Error: http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64 Release.gpg
The following signatures were invalid： BADSIG F60F4B3D7FA2AF80 cudatools cudatools@nvidia.com

=== Chinese ===

错误: http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64 Release.gpg
下列签名无效： BADSIG F60F4B3D7FA2AF80 cudatools cudatools@nvidia.com
......
W: 校验数字签名时出错。此仓库未被更新，所以仍然使用此前的索引文件。GPG 错误：http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64 Release: 下列签名无效： BADSIG F60F4B3D7FA2AF80 cudatools cudatools@nvidia.com
W: 无法下载 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/Release.gpg 下列签名无效： BADSIG F60F4B3D7FA2AF80 cudatools cudatools@nvidia.com
W: 部分索引文件下载失败。如果忽略它们，那将转而使用旧的索引文件。

@flx42
Copy link
Member

flx42 commented Jan 10, 2018

@Pingmin can you try again in a few hours? If it persists, I will ask for a CDN cache flush.

@vincentbissonnette
Copy link

I also have the exact same issue here, on nvidia/cuda:8.0 and 9.0 docker images for Ubuntu 16.04.
I am located in France.

@flx42
Copy link
Member

flx42 commented Jan 12, 2018

@vincentbissonnette Use nvidia/cuda:8.0-devel or nvidia/cuda:9.0-devel

@vincentbissonnette
Copy link

I am indeed using the devel images in both cases. Sorry that was unclear.

@flx42
Copy link
Member

flx42 commented Jan 12, 2018

@vincentbissonnette can you post the log?

@vincentbissonnette
Copy link

Log from apt-get update :

Hit:1 http://archive.ubuntu.com/ubuntu xenial InRelease
Hit:2 http://ppa.launchpad.net/george-edison55/cmake-3.x/ubuntu xenial InRelease
Get:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB]                      
Get:4 http://archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB]                               
Get:5 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB]                               
Ign:6 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  InRelease
Get:7 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Release [564 B]
Get:8 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Release.gpg [819 B]
Get:9 http://archive.ubuntu.com/ubuntu xenial-updates/universe Sources [234 kB]
Get:10 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [903 kB]
Err:8 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Release.gpg      
  The following signatures were invalid: BADSIG F60F4B3D7FA2AF80 cudatools <cudatools@nvidia.com>
Get:11 http://archive.ubuntu.com/ubuntu xenial-updates/restricted amd64 Packages [13.1 kB]        
Get:12 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [735 kB]
Get:13 http://security.ubuntu.com/ubuntu xenial-security/universe Sources [56.8 kB]
Get:14 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 Packages [18.5 kB]
Get:15 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [539 kB]
Get:16 http://security.ubuntu.com/ubuntu xenial-security/restricted amd64 Packages [12.7 kB]
Get:17 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [245 kB]
Get:18 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages [3492 B]
Fetched 3069 kB in 1s (1781 kB/s)   
Reading package lists... Done
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Release: The following signatures were invalid: BADSIG F60F4B3D7FA2AF80 cudatools <cudatools@nvidia.com>
W: Failed to fetch http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/Release.gpg  The following signatures were invalid: BADSIG F60F4B3D7FA2AF80 cudatools <cudatools@nvidia.com>
W: Some index files failed to download. They have been ignored, or old ones used instead.

I have also tried the usual steps of deleting the /var/lib/apt/lists folder and updating again, as well as apt-get clean, the issue still remains.

@flx42
Copy link
Member

flx42 commented Jan 15, 2018

@Pingmin @vincentbissonnette what's the output of docker run --rm -ti nvidia/cuda:9.0-devel apt-key list?

@vincentbissonnette
Copy link

Output:

/etc/apt/trusted.gpg
--------------------
pub   1024D/437D05B5 2004-09-12
uid                  Ubuntu Archive Automatic Signing Key <ftpmaster@ubuntu.com>
sub   2048g/79164387 2004-09-12

pub   4096R/C0B21F32 2012-05-11
uid                  Ubuntu Archive Automatic Signing Key (2012) <ftpmaster@ubuntu.com>

pub   4096R/EFE21092 2012-05-11
uid                  Ubuntu CD Image Automatic Signing Key (2012) <cdimage@ubuntu.com>

pub   1024D/FBB75451 2004-12-30
uid                  Ubuntu CD Image Automatic Signing Key <cdimage@ubuntu.com>

pub   4096R/7FA2AF80 2016-06-24
uid                  cudatools <cudatools@nvidia.com>

@vincentbissonnette
Copy link

However, the issue appears to have been fixed, as I'm now unable to reproduce the error.
apt-get update now performs the update as usual:

Get:1 http://archive.ubuntu.com/ubuntu xenial InRelease [247 kB]
Get:2 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB]     
Get:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB]         
Ign:5 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  InRelease
Get:6 http://archive.ubuntu.com/ubuntu xenial/universe Sources [9802 kB]
Get:7 http://security.ubuntu.com/ubuntu xenial-security/universe Sources [56.8 kB]                    
Get:8 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [539 kB]                                         
Ign:9 http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64  InRelease
Get:10 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Release [564 B]  
Get:11 http://security.ubuntu.com/ubuntu xenial-security/restricted amd64 Packages [12.7 kB]
Get:12 http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64  Release [564 B]
Get:13 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Release.gpg [801 B]
Get:14 http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64  Release.gpg [801 B]
Get:15 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [245 kB]                            
Get:16 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages [3492 B]                
Get:17 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Packages [109 kB]                  
Get:18 http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64  Packages [14.4 kB]
Get:19 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages [1558 kB]
Get:20 http://archive.ubuntu.com/ubuntu xenial/restricted amd64 Packages [14.1 kB]
Get:21 http://archive.ubuntu.com/ubuntu xenial/universe amd64 Packages [9827 kB]
Get:22 http://archive.ubuntu.com/ubuntu xenial/multiverse amd64 Packages [176 kB]
Get:23 http://archive.ubuntu.com/ubuntu xenial-updates/universe Sources [234 kB]
Get:24 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [903 kB]
Get:25 http://archive.ubuntu.com/ubuntu xenial-updates/restricted amd64 Packages [13.1 kB]
Get:26 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [735 kB]
Get:27 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 Packages [18.5 kB]
Get:28 http://archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages [5162 B]
Get:29 http://archive.ubuntu.com/ubuntu xenial-backports/universe amd64 Packages [7146 B]
Fetched 24.8 MB in 4s (5605 kB/s)                           
Reading package lists... Done

Any idea on what causes this issue ?

@flx42
Copy link
Member

flx42 commented Jan 15, 2018

I'm not sure, our CDN can get into a bad state sometimes, it's difficult to debug since it seems to be region specific.
Please file a new issue if it happens again.

@pwyq pwyq mentioned this issue Jan 20, 2018
Closed
@flx42 flx42 mentioned this issue Jan 22, 2018
Closed
@Pingmin
Copy link

Pingmin commented Feb 26, 2018 •
edited
Loading

@flx42 Sorry, I just see these messages right now. It seems like it's OK now and I have updated CUDA to v9.1.85-1 for some days.

@riless
Copy link

riless commented Mar 9, 2018 •
edited
Loading

I have the same Issue with nvidia/cuda:9.0-cudnn7-runtime-ubuntu16.04 (from France)

Get:18 http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64  Packages [17.9 kB]
Err:18 http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64  Packages
  Hash Sum mismatch

@flx42
Copy link
Member

flx42 commented Mar 9, 2018

I've circulated the information internally, since I don't maintain this package repository.

@riless
Copy link

riless commented Mar 12, 2018

I've retried today, and now it works.

@classicsong
Copy link

I have the same problem these days:
E: GPG error: http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1404/x86_64 Release: The following signatures were invalid: NODATA 1 NODATA 2

@flx42
Copy link
Member

flx42 commented May 27, 2018

@classicsong try setting the repository as https and check if it changes something.
You will need the apt-transport-https package.

@kyeongsoo
Copy link

@flx42 Changing 'http' to 'https' works for me. Great thanks! In fact, I have been suffering from this issue since last month or so.

@jageshmaharjan
Copy link

jageshmaharjan commented Jun 1, 2018 •
edited
Loading

I think it's a region-specific error, some region doesn't allow access on 'http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64'
The above link will redirect to this link "http://114.80.80.238:8089/ceshi/index.html". The content is in the Chinese language the translated version is:

Tips: The site is temporarily inaccessible

According to the relevant laws and regulations of the Ministry of Industry and Information, because your website has not yet been filed, the website cannot be accessed.

According to the Order No. 33 of the Ministry of Industry and Information Technology of the People's Republic of China: Websites that do not have record numbers cannot be solved
Analysis and opening, otherwise it is necessary to bear the relevant legal responsibility.
        	 
Please be sure that all enterprises and institutions must complete the record and open the website. Thank you for your support!
	
Century Connect Record Consultation Contact Information

Registration Advisory Telephone: 021-36359595

What we could do is to edit on "sudo nano /etc/apt/sources.list.d/cuda.list" file and replace
"http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64" with
"https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64"

@peterborkuti
Copy link

peterborkuti commented Jun 9, 2018 •
edited
Loading

What helped in my case:

  1. install apt-transport-https
  2. Setting link from http: to https: in /etc/apt/sources.list.d/cuda.list
  3. updating apt-key
    check the https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/ page and find the .pub file and update apt-key with that .pub file's content, like this way:
wget -qO - https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub | sudo apt-key add -

@goodloop
Copy link

Thanks @peterborkuti. Step 1 really helped me.

@8rV1n 8rV1n mentioned this issue Jun 15, 2018
Closed
@reubenjohn
Copy link

@peterborkuti Needed to perform an additional Step 3: sudo apt clean before sudo apt update worked to fix "problems were found: Metadata files have errors".

@htgeis
Copy link

htgeis commented Sep 20, 2018

It happened again. I have try the way mentioned above but it all failed. The same image worked in yesterday.

@flx42
Copy link
Member

flx42 commented Sep 20, 2018

@htgeis the CUDA repositories are being updated right now, check again in ~1h.

@shridharkini
Copy link

nvidia/cuda:9.0-cudnn7-devel-ubuntu16.04 has the same issues still.

@nuzhet
Copy link

nuzhet commented Sep 26, 2018

I am getting this error today. It was working fine yesterday:

E: Failed to fetch https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/Packages.gz Hash Sum mismatch

@khurramHazen
Copy link

same issue as @nuzhet

@flx42
Copy link
Member

flx42 commented Sep 26, 2018

@nuzhet I can reproduce, I'm filing an issue internally.

@khurramHazen
Copy link

khurramHazen commented Sep 26, 2018 •
edited
Loading

@flx42 Do you think using 9.1-cudnn7-devel-ubuntu16.04 will help?

Edit: No it will not.

@gpascualg
Copy link

gpascualg commented Sep 26, 2018 •
edited
Loading

I can also reproduce with nvidia/cuda:9.2-cudnn7-devel-ubuntu18.04 and nvidia/cuda:9.2-cudnn7-devel-ubuntu16.04

Error on 18.04 is somewhat different and references 1604:

E: Failed to fetch https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/Packages.gz  File has unexpected size (29068 != 30835). Mirror sync in progress? [IP: 192.229.221.58 443]
   Hashes of expected file:
    - Filesize:30835 [weak]
    - SHA256:5900f4f7f9ea42522a4d2ef07c58e8492bea610380c47e803d75b6a1a30cdebe
    - SHA1:96cb67d3d73ebfe35b732bb46cde0e3784adf168 [weak]
    - MD5Sum:292e5eb4f76dd981bf199fa6760146b0 [weak]
   Release file created at: Tue, 25 Sep 2018 18:11:32 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead.

@marcoaleixo
Copy link

I'm also having this error using this Dockerfile - https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/dockerfiles/dockerfiles/nvidia-jupyter.Dockerfile

I'm from Brazil.

@pzawistowski
Copy link

Same situation from Poland.

@mgolub2
Copy link

mgolub2 commented Sep 26, 2018

Same issue in USA NW

@rpengms
Copy link

rpengms commented Sep 26, 2018

+1.

@mfojtak
Copy link

mfojtak commented Sep 26, 2018

Same problem when building docker image.

Failed to fetch https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/Packages.gz  Hash Sum mismatch

@flx42
Copy link
Member

flx42 commented Sep 26, 2018

Yeah, I agree there is an issue, but I don't control this repository. I'm pushing for a resolution.
In the meantime, if your driver is recent enough (driver 410), you can use the CUDA 10.0 image on ubuntu 18.04: nvidia/cuda:10.0-base-ubuntu18.04
The 18.04 repository is not affected.

@mirekphd
Copy link

@flx42 It's gone, seems to be resolved.

@khurramHazen
Copy link

It is resolved in nvidia/cuda:9.0-cudnn7-devel-ubuntu16.04 too.

@flx42
Copy link
Member

flx42 commented Sep 26, 2018

Cool! Thanks for confirming (and reporting!).

@NVIDIA NVIDIA locked as resolved and limited conversation to collaborators Sep 26, 2018
@flx42
Copy link
Member

flx42 commented Sep 26, 2018

Locking this issue, please file a new one next time there is a problem with the repo :)

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests