GPG error "public key is not available" in Ubuntu 20.04 CUDA 11.4.0 image while building

[wilkesreid]

1. Issue or feature description

The following Dockerfile does not build today (April 28, 2022), even though it built successfully yesterday:

FROM    nvidia/cuda:11.4.0-runtime-ubuntu20.04
RUN     apt-get update

The error is the following:

GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC
The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease' is not signed.

2. Steps to reproduce the issue

Create the above dockerfile and attempt to docker build it.

3. Information to attach (optional if deemed irrelevant)

I am running Docker version 20.10.14, build a224086 on WSL 2 Ubuntu 20.04 on Windows 10 Pro, Version 21H2, OS Build 19044.1645

nvidia-smi on my host machine:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.65       Driver Version: 471.96       CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:0A:00.0  On |                  N/A |
|  0%   55C    P0    61W / 250W |   1515MiB /  8192MiB |    ERR!      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

[wilkesreid]

Possible duplicate of #258

[klueska]

https://forums.developer.nvidia.com/t/notice-cuda-linux-repository-key-rotation/212772

[wilkesreid]

The instructions in that notice do not work in the docker image.

FROM  nvidia/cuda:11.4.0-runtime-ubuntu20.04
...
RUN   apt-key del 7fa2af80
ADD   https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb .
RUN   dpkg -i cuda-keyring_1.0-1_all.deb
...
RUN   apt-get update

Results in

Conflicting values set for option Signed-By regarding source https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /: /usr/share/keyrings/cuda-archive-keyring.gpg !=

[memray]

The instructions in that notice do not work in the docker image.

FROM  nvidia/cuda:11.4.0-runtime-ubuntu20.04
...
RUN   apt-key del 7fa2af80
ADD   https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb .
RUN   dpkg -i cuda-keyring_1.0-1_all.deb
...
RUN   apt-get update

Results in

Conflicting values set for option Signed-By regarding source https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /: /usr/share/keyrings/cuda-archive-keyring.gpg !=

Tried following the methods in announcement and didn't work either. Well done Nvidia.

[memray]

A workaround seems to do the trick for me. Add those lines before apt-get update

RUN rm /etc/apt/sources.list.d/cuda.list
RUN rm /etc/apt/sources.list.d/nvidia-ml.list

Also check out discussions here

[jtran1999]

I noticed different keys available to day for the repo.

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin
sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600
sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub
sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /"
sudo apt-get update
sudo apt-get -y install cuda

[zestrells]

This is what I used in the dockerfile to fix this issue.

    && apt-key del 7fa2af80 \
    && curl -L -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-keyring_1.0-1_all.deb \
    && dpkg -i cuda-keyring_1.0-1_all.deb \

[mirekphd]

None of these hacks above are sufficiently reliable yet, as NVIDIA is still working on the changes. Some latest CUDA and Ubuntu versions are already working (images such as CUDA 11.6 for Ubuntu 20.04 can be rebuild from their code at Gitlab), but others (older CUDA/Ubuntu versions such as CUDA 11.2) may still fail.

So given that CUDA 11.1 .. 11.6 toolkits are compatible with the same drivers version (>=450.80.02) it should be possible to adapt NVIDIA's Dockerfile with the latest CUDA version (11.6) and your OS of choice:
https://gitlab.com/nvidia/container-images/cuda/-/tree/master/dist/11.6.2

More info: https://gitlab.com/nvidia/container-images/cuda/-/issues/158

[372046933]

nvidia/cuda:11.2.1-base-ubuntu20.04 is updated this afternoon. I pull the updated image and the pub key problem disappeared

[kinue00]

A workaround seems to do the trick for me. Add those lines before apt-get update

RUN rm /etc/apt/sources.list.d/cuda.list
RUN rm /etc/apt/sources.list.d/nvidia-ml.list

Also check out discussions here

seems working for me (kaldi's official dockerfile ubuntu1804)

[VictorZuanazzi]

We are having the same problem with the image nvidia/cuda:11.2.2-cudnn8-devel-ubuntu20.04 and 11.3. Any overview of which images are fixed?

[zjuPeco]

RUN apt-key del 7fa2af80
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64/7fa2af80.pub

This works for me

[Co0perator]

This may add nothing productive to the conversation but I thought it worth mentioning
Most people seem to be getting this error in Ubuntu 20.04 Docker containers, I'm getting it on Ubuntu 20.04 Desktop.

[ghost]

RUN apt-key del 7fa2af80
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64/7fa2af80.pub

This works for me

This worked for me as well.

[LordNex]

having the same issue on a 4gb Jetson Nano

[zanedurante]

I found this issue searching up my problem on google (working with a GCP instance with nvidia GPU, no docker). Eventually following the steps here worked for me: https://developer.nvidia.com/blog/updating-the-cuda-linux-gpg-repository-key/

[aycaecemgul]

I found this issue searching up my problem on google (working with a GCP instance with nvidia GPU, no docker). Eventually following the steps here worked for me: https://developer.nvidia.com/blog/updating-the-cuda-linux-gpg-repository-key/

This is the only solution that worked for ubuntu 20.04 desktop.

[StatsGary]

A workaround seems to do the trick for me. Add those lines before apt-get update

RUN rm /etc/apt/sources.list.d/cuda.list
RUN rm /etc/apt/sources.list.d/nvidia-ml.list

Also check out discussions here

This worked for me.

[pxorig]

try this:

FROM nvidia/cuda:10.1-cudnn8-devel-ubuntu18.04
ENV LANG C.UTF-8
RUN  rm -rf /var/lib/apt/lists/* \
         /etc/apt/sources.list.d/cuda.list \
          /etc/apt/sources.list.d/nvidia-ml.list 
RUN apt-get update 

[ri-cao]

A workaround seems to do the trick for me. Add those lines before apt-get update

RUN rm /etc/apt/sources.list.d/cuda.list
RUN rm /etc/apt/sources.list.d/nvidia-ml.list

Also check out discussions here

This trick works for me. Thank you! My system is Ubuntu 22.04. The base singularity image was built on Ubuntu 18.04

[PolarisLight]

A workaround seems to do the trick for me. Add those lines before apt-get update

RUN rm /etc/apt/sources.list.d/cuda.list
RUN rm /etc/apt/sources.list.d/nvidia-ml.list

Also check out discussions here

Thanks, this works for me.

[karray]

A workaround seems to do the trick for me. Add those lines before apt-get update

RUN rm /etc/apt/sources.list.d/cuda.list
RUN rm /etc/apt/sources.list.d/nvidia-ml.list

Also check out discussions here

I encountered a similar issue with the nvidia/cuda Docker container, but the specific files mentioned in the workaround were not present in my container. Instead, the relevant file in my case was /etc/apt/sources.list.d/cuda-ubuntu2204-x86_64.list.

Therefore, it is advisable to check the destination of the apt source list before attempting to delete them.

[lehcode]

Working Docker solution is as follows, also with persistence across services being built;

ADD "https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin" "/tmp/cuda-ubuntu2004.pin"
ADD "https://developer.download.nvidia.com/compute/cuda/11.3.1/local_installers/cuda-repo-ubuntu2004-11-3-local_11.3.1-465.19.01-1_amd64.deb" "/tmp/cuda-repo-ubuntu2004-11-3-local_11.3.1-465.19.01-1_amd64.deb"
ADD "https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/7fa2af80.pub" "/etc/apt/7fa2af80.pub"

RUN cat /etc/apt/7fa2af80.pub | apt-key add - && \
    dpkg -i /tmp/cuda-repo-ubuntu2004-11-3-local_11.3.1-465.19.01-1_amd64.deb && \
    apt update && apt -y install cuda && \