Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Public GPG key error #258

Open
vamman opened this issue Apr 28, 2022 · 57 comments
Open

Public GPG key error #258

vamman opened this issue Apr 28, 2022 · 57 comments

Comments

@vamman
Copy link

vamman commented Apr 28, 2022

Where's the public GPG key? Was working fine for weeks. Today this error.

W: GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC
E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease' is not signed.

@rrlamichhane
Copy link

Seeing the exact issue as well.

@zestrells
Copy link

Also seeing this issue and trying to debug it now.

@klueska
Copy link
Contributor

klueska commented Apr 28, 2022

https://forums.developer.nvidia.com/t/notice-cuda-linux-repository-key-rotation/212772

@topinfrassi01
Copy link

topinfrassi01 commented Apr 28, 2022

@klueska I'm starting from the tensorflow/tensorflow:2.7.0-gpu image, I tried following this blog and so I added the following lines at the start of my file (after FROM tensorflow...)

RUN apt-get install -y --no-install-recommends wget
RUN apt-key del 7fa2af80
RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb
RUN dpkg -i cuda-keyring_1.0-1_all.deb
RUN apt-get update

When running apt-get update, I have the following error :

NVIDIA/nvidia-docker#11 [ 8/13] RUN apt-get update
NVIDIA/nvidia-docker#11 sha256:c1afeb0121b560d8f8cf7247423fec777c038dd0f561407a5757db6c7e71ce82
NVIDIA/nvidia-docker#11 0.503 E: Conflicting values set for option Signed-By regarding source https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /: /usr/share/keyrings/cuda-archive-keyring.gpg !=
NVIDIA/nvidia-docker#11 0.503 E: The list of sources could not be read.
NVIDIA/nvidia-docker#11 ERROR: executor failed running [/bin/bash -c apt-get update]: exit code: 100

In order to add as much information as possible, here is the content of the /etc/apt/sources.list.d folder :

  • cuda.list
  • nvidia-ml.list

I tried deleting both these files before running the previous lines, then I get the following error :

NVIDIA/nvidia-docker#12 7.636 E: Failed to fetch https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/Packages.gz File has unexpected size (631054 != 481481). Mirror sync in progress? [IP: 152.199.20.126 443]
NVIDIA/nvidia-docker#12 7.636 Hashes of expected file:
NVIDIA/nvidia-docker#12 7.636 - Filesize:481481 [weak]
NVIDIA/nvidia-docker#12 7.636 - SHA256:8556d67c6d380c957f05057f448d994584a135d7ed75e5ae6bb25c3fc1070b0b
NVIDIA/nvidia-docker#12 7.636 - SHA1:c5ea9556407a3b5daec4aac530cd038e9b490441 [weak]
NVIDIA/nvidia-docker#12 7.636 - MD5Sum:a5513131dbd2d4e50f185422ebb43ac9 [weak]
NVIDIA/nvidia-docker#12 7.636 Release file created at: Mon, 25 Apr 2022 23:27:19 +0000
NVIDIA/nvidia-docker#12 7.636 E: Some index files failed to download. They have been ignored, or old ones used instead.

@GeneralErrorOK
Copy link

GeneralErrorOK commented Apr 28, 2022

I have the exact same issue. Added this to my Dockerfile:
WORKDIR /tmp RUN apt-key del 7fa2af80 && \ apt install wget && \ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-keyring_1.0-1_all.deb && \ dpkg -i cuda-keyring_1.0-1_all.deb

And then, when running apt update and apt install:

E: Conflicting values set for option Signed-By regarding source https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /: /usr/share/keyrings/cuda-archive-keyring.gpg !=
E: The list of sources could not be read.

@estan
Copy link

estan commented Apr 28, 2022

Same here, someone pushed the wrong button at NVIDIA.

@rrlamichhane
Copy link

https://forums.developer.nvidia.com/t/notice-cuda-linux-repository-key-rotation/212772

I rotated my keys, but still getting errors.

Commands in Dockerfile:

RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb
RUN dpkg -i cuda-keyring_1.0-1_all.deb
RUN apt update && apt install -y cuda-compat-11-4

Error:

NVIDIA/nvidia-docker#8 7.228 E: Failed to fetch https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/Packages.gz  File has unexpected size (631054 != 481481). Mirror sync in progress? [IP: 152.195.19.142 443]
NVIDIA/nvidia-docker#8 7.228    Hashes of expected file:
NVIDIA/nvidia-docker#8 7.228     - Filesize:481481 [weak]
NVIDIA/nvidia-docker#8 7.228     - SHA256:8556d67c6d380c957f05057f448d994584a135d7ed75e5ae6bb25c3fc1070b0b
NVIDIA/nvidia-docker#8 7.228     - SHA1:c5ea9556407a3b5daec4aac530cd038e9b490441 [weak]
NVIDIA/nvidia-docker#8 7.228     - MD5Sum:a5513131dbd2d4e50f185422ebb43ac9 [weak]
NVIDIA/nvidia-docker#8 7.228    Release file created at: Mon, 25 Apr 2022 23:27:19 +0000

@rrlamichhane
Copy link

NVIDIA/nvidia-docker#1633

@wilkesreid
Copy link

It's pretty disappointing that hardly a day's notice is considered enough warning before breaking everyone's ability to install.

@ryan00
Copy link

ryan00 commented Apr 28, 2022

Argggg, borked build club member signin

this seems to work for me
RUN rm /etc/apt/sources.list.d/cuda.list
RUN rm /etc/apt/sources.list.d/nvidia-ml.list
RUN apt-key del 7fa2af80
RUN apt-get update && apt-get install -y --no-install-recommends wget
RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-keyring_1.0-1_all.deb
RUN dpkg -i cuda-keyring_1.0-1_all.deb

@jtran1999
Copy link

it seems the keys changed today.

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin
sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600
sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub
sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /"
sudo apt-get update
sudo apt-get -y install cuda

@zestrells
Copy link

zestrells commented Apr 29, 2022

This is what I used in the Dockerfile to fix this

    && apt-key del 7fa2af80 \
    && curl -L -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-keyring_1.0-1_all.deb \
    && dpkg -i cuda-keyring_1.0-1_all.deb \

@sanjass
Copy link

sanjass commented Apr 29, 2022

This worked for me for a dockerfile, I used the alternative approach on the official notice but I had to remove old source files and I also needed to fetch 2 keys, one for "cuda" and one for "machine learning"
Put these after the FROM command

RUN rm /etc/apt/sources.list.d/cuda.list
RUN rm /etc/apt/sources.list.d/nvidia-ml.list
RUN apt-key del 7fa2af80
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/3bf863cc.pub
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/7fa2af80.pub

Make sure you change ubuntu1804/x86_64 to your distro/arch. Also, note the first two lines are necessary as the base image might have bad keys (you might get an error The list of sources could not be read without them)

@GeneralErrorOK
Copy link

RUN rm /etc/apt/sources.list.d/cuda.list

I can confirm. This is also the line that I needed to add to get it to work. Only the alternative method from the official announcement seems to work in a Docker build.

This is what I added at the start of my Dockerfile: (after FROM)

RUN rm /etc/apt/sources.list.d/cuda.list && \ apt-key del 7fa2af80 && \ apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/3bf863cc.pub

I still think it's strange for NVIDIA to break everyone's build pipelines with only a few hours notice.

@roman-vynar
Copy link

Same here, ubuntu2004 repo is broken because the previous key 7fa2af80.pub stopped working.
The right one can be found here ubuntu1804/x86_64/3bf863cc.pub 😐

@LordNetro
Copy link

This worked for me, I used the alternative approach on the official notice but I had to fetch another 2 keys, one for "cuda" and one for "machine learning"

RUN rm /etc/apt/sources.list.d/cuda.list
RUN rm /etc/apt/sources.list.d/nvidia-ml.list
RUN apt-key del 7fa2af80
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/3bf863cc.pub
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/7fa2af80.pub

This worked for me, but remember to change the arguments of the command depending on the version of your distribution. Thanks!

@kasaikou
Copy link

This may solved for me (base image: nvidia/cuda:11.3.1-cudnn8-devel-ubuntu20.04)

RUN \
    # Update nvidia GPG key
    rm /etc/apt/sources.list.d/cuda.list && \
    rm /etc/apt/sources.list.d/nvidia-ml.list && \
    apt-key del 7fa2af80 && \
    apt-get update && apt-get install -y --no-install-recommends wget && \
    wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb && \
    dpkg -i cuda-keyring_1.0-1_all.deb && \
    apt-get update

@cwichka
Copy link

cwichka commented Apr 29, 2022

This one worked for me here

RUN rm /etc/apt/sources.list.d/cuda.list
RUN apt-key del 7fa2af80
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/7fa2af80.pub

@AlanSavio25
Copy link

Argggg, borked build club member signin

this seems to work for me RUN rm /etc/apt/sources.list.d/cuda.list RUN rm /etc/apt/sources.list.d/nvidia-ml.list RUN apt-key del 7fa2af80 RUN apt-get update && apt-get install -y --no-install-recommends wget RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-keyring_1.0-1_all.deb RUN dpkg -i cuda-keyring_1.0-1_all.deb

This solved it for me. Thanks!

@wkcn
Copy link

wkcn commented Apr 29, 2022

Argggg, borked build club member signin

this seems to work for me RUN rm /etc/apt/sources.list.d/cuda.list RUN rm /etc/apt/sources.list.d/nvidia-ml.list RUN apt-key del 7fa2af80 RUN apt-get update && apt-get install -y --no-install-recommends wget RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-keyring_1.0-1_all.deb RUN dpkg -i cuda-keyring_1.0-1_all.deb

It works for me. Thank you!

The files /etc/apt/sources.list.d/cuda.list and /etc/apt/sources.list.d/nvidia-ml.list should be removed before installing cuda-keyring, otherwise it raises the error The list of sources could not be read.

rosbo referenced this issue in Kaggle/docker-python Apr 29, 2022
Context: https://github.com/NVIDIA/nvidia-docker/issues/1631

The next `deeplearning-platform-release/tf2-gpu.2-6` should remove the
need for this workaround.

http://b/230864778
rosbo referenced this issue in Kaggle/docker-python Apr 29, 2022
Context: https://github.com/NVIDIA/nvidia-docker/issues/1631

The next `deeplearning-platform-release/tf2-gpu.2-6` should remove the
need for this workaround.

http://b/230864778
@mynkyu
Copy link

mynkyu commented Jun 8, 2022

Same Issue,

# https://github.com/NVIDIA/nvidia-container-toolkit/issues/258
RUN apt-key del 7fa2af80
RUN apt-key del 3bf863cc
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/3bf863cc.pub
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub

Adding this lines above apt-get update solved the issue. Thanks @Xcompanygames

@harshkakashaniya
Copy link

This worked for me for a dockerfile, I used the alternative approach on the official notice but I had to remove old source files and I also needed to fetch 2 keys, one for "cuda" and one for "machine learning" Put these after the FROM command

RUN rm /etc/apt/sources.list.d/cuda.list
RUN rm /etc/apt/sources.list.d/nvidia-ml.list
RUN apt-key del 7fa2af80
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/3bf863cc.pub
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/7fa2af80.pub

Make sure you change ubuntu1804/x86_64 to your distro/arch. Also, note the first two lines are necessary as the base image might have bad keys (you might get an error The list of sources could not be read without them)

In my case I had to comment second line

RUN rm /etc/apt/sources.list.d/cuda.list
# RUN rm /etc/apt/sources.list.d/nvidia-ml.list
RUN apt-key del 7fa2af80
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/3bf863cc.pub
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/7fa2af80.pub

@nectario
Copy link

nectario commented Jul 12, 2022

I am getting this error:

NVIDIA/nvidia-docker#6 2.655 Get:18 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  Packages [580 kB]
NVIDIA/nvidia-docker#6 2.656 Err:18 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  Packages
NVIDIA/nvidia-docker#6 2.656   File has unexpected size (580845 != 580040). Mirror sync in progress? [IP: 152.195.19.142 443]
NVIDIA/nvidia-docker#6 2.656   Hashes of expected file:
NVIDIA/nvidia-docker#6 2.656    - Filesize:580040 [weak]
NVIDIA/nvidia-docker#6 2.656    - SHA256:dc8abbaf470d3ee626f4f1f4d2871a98d8dc8f770bc592676f0d1f60637e0c2d
NVIDIA/nvidia-docker#6 2.656    - SHA1:109bfa4e5c415731aa44b1c1caae3f19754c5406 [weak]
NVIDIA/nvidia-docker#6 2.656    - MD5Sum:bf8f928cb55b1c9e4c158a69ee52b5d8 [weak]
NVIDIA/nvidia-docker#6 2.656   Release file created at: Mon, 11 Jul 2022 19:01:54 +0000
NVIDIA/nvidia-docker#6 2.996 Fetched 9797 kB in 2s (4170 kB/s)
NVIDIA/nvidia-docker#6 2.996 Reading package lists...
NVIDIA/nvidia-docker#6 4.132 E: Failed to fetch https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/Packages.gz  File has unexpected size (580845 != 580040). Mirror sync in progress? [IP: 152.195.19.142 443]
NVIDIA/nvidia-docker#6 4.132    Hashes of expected file:
NVIDIA/nvidia-docker#6 4.132     - Filesize:580040 [weak]
NVIDIA/nvidia-docker#6 4.132     - SHA256:dc8abbaf470d3ee626f4f1f4d2871a98d8dc8f770bc592676f0d1f60637e0c2d
NVIDIA/nvidia-docker#6 4.132     - SHA1:109bfa4e5c415731aa44b1c1caae3f19754c5406 [weak]
NVIDIA/nvidia-docker#6 4.132     - MD5Sum:bf8f928cb55b1c9e4c158a69ee52b5d8 [weak]
NVIDIA/nvidia-docker#6 4.132    Release file created at: Mon, 11 Jul 2022 19:01:54 +0000
NVIDIA/nvidia-docker#6 4.132 E: Some index files failed to download. They have been ignored, or old ones used instead.

@TheBlackPlague
Copy link

This one worked for me here

RUN rm /etc/apt/sources.list.d/cuda.list
RUN apt-key del 7fa2af80
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/7fa2af80.pub

This worked for me. I just had the runtime, so /etc/apt/sources.list.d/nvidia-ml.list didn't exist for me.

Thank you ❤️

@daMichaelB
Copy link

I had the same problem building my Docker image with

FROM nvidia/cuda:11.4.0-base-ubuntu20.04.

I just had to delete the old nvidia/cuda image that was downloaded on my server months ago and with that force Docker to fetch the latest update. Worked without any problem.

fversaci referenced this issue in fversaci/cassandra-dali-plugin Aug 22, 2022
- Improved setup.py for different python versions
- Fix for broken nvidia dockerfile
  https://github.com/NVIDIA/nvidia-docker/issues/1631
@gauenk
Copy link

gauenk commented Oct 2, 2022

I had an issue with running sudo apt-key del 7fa2af80. Specifically, my machine prints: Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). The deprecation of this apt-key command will not allow me to add the cuda using the install guide.

Since apt-key won't work, I searched and found a great answer about why apt-key is deprecated, and how to manually install the keys. I used these steps to finally install cuda successfully and painlessly. I am sharing them below, since I didn't see it in my googling:

  1. Download the 3bf863cc.pub file
  2. Copy the file to a save place
sudo cp 3bf863cc.pub /etc/apt/keyrings/
  1. Tell "apt" about cuda and your new key
sudo echo "deb [signed-by=/etc/apt/keyrings/3bf863cc.pub] https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/ /" /etc/apt/sources.list.d/cuda.list
  1. Update & Install
sudo apt-get update
sudo apt-get install cuda

I am running Ubuntu 22.02 on x86, so the URLs can be modified with your version and arch accordingly.

Note: I did remove all my other cuda-related souces.list.d files before all of this. I am not sure if it's important but I do not think it is.

@Khoa-NT
Copy link

Khoa-NT commented Dec 2, 2022

Thank you @gauenk but I would like to update for the 3rd step Tell "apt" about cuda and your new key:

sudo echo "deb [signed-by=/etc/apt/keyrings/3bf863cc.pub] https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/ /" | sudo tee -a /etc/apt/sources.list.d/cuda.list

Adding sudo tee -a following this link

@Damianosso
Copy link

@klueska I'm starting from the tensorflow/tensorflow:2.7.0-gpu image, I tried following this blog and so I added the following lines at the start of my file (after FROM tensorflow...)

RUN apt-get install -y --no-install-recommends wget RUN apt-key del 7fa2af80 RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb RUN dpkg -i cuda-keyring_1.0-1_all.deb RUN apt-get update

When running apt-get update, I have the following error :

NVIDIA/nvidia-docker#11 [ 8/13] RUN apt-get update
NVIDIA/nvidia-docker#11 sha256:c1afeb0121b560d8f8cf7247423fec777c038dd0f561407a5757db6c7e71ce82
NVIDIA/nvidia-docker#11 0.503 E: Conflicting values set for option Signed-By regarding source https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /: /usr/share/keyrings/cuda-archive-keyring.gpg !=
NVIDIA/nvidia-docker#11 0.503 E: The list of sources could not be read.
NVIDIA/nvidia-docker#11 ERROR: executor failed running [/bin/bash -c apt-get update]: exit code: 100

In order to add as much information as possible, here is the content of the /etc/apt/sources.list.d folder :

* cuda.list

* nvidia-ml.list

I tried deleting both these files before running the previous lines, then I get the following error :

NVIDIA/nvidia-docker#12 7.636 E: Failed to fetch https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/Packages.gz File has unexpected size (631054 != 481481). Mirror sync in progress? [IP: 152.199.20.126 443]
NVIDIA/nvidia-docker#12 7.636 Hashes of expected file:
NVIDIA/nvidia-docker#12 7.636 - Filesize:481481 [weak]
NVIDIA/nvidia-docker#12 7.636 - SHA256:8556d67c6d380c957f05057f448d994584a135d7ed75e5ae6bb25c3fc1070b0b
NVIDIA/nvidia-docker#12 7.636 - SHA1:c5ea9556407a3b5daec4aac530cd038e9b490441 [weak]
NVIDIA/nvidia-docker#12 7.636 - MD5Sum:a5513131dbd2d4e50f185422ebb43ac9 [weak]
NVIDIA/nvidia-docker#12 7.636 Release file created at: Mon, 25 Apr 2022 23:27:19 +0000
NVIDIA/nvidia-docker#12 7.636 E: Some index files failed to download. They have been ignored, or old ones used instead.

OK

@viiika
Copy link

viiika commented Jan 9, 2024

This may solved for me (base image: nvidia/cuda:11.3.1-cudnn8-devel-ubuntu20.04)

RUN \
    # Update nvidia GPG key
    rm /etc/apt/sources.list.d/cuda.list && \
    rm /etc/apt/sources.list.d/nvidia-ml.list && \
    apt-key del 7fa2af80 && \
    apt-get update && apt-get install -y --no-install-recommends wget && \
    wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb && \
    dpkg -i cuda-keyring_1.0-1_all.deb && \
    apt-get update

This manner solves my problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests