Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ubuntu kubernetes-xenial public key is not available: NO_PUBKEY B53DC80D13EDEF05 #2862

Closed
khteh opened this issue Jan 17, 2023 · 73 comments
Closed
Labels
area/release-eng Issues or PRs related to the Release Engineering subproject kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/release Categorizes an issue or PR as relevant to SIG Release.

Comments

@khteh
Copy link

khteh commented Jan 17, 2023

What happened:

$ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
100  1210  100  1210    0     0  39402      0 --:--:-- --:--:-- --:--:-- 40333
OK
$ sudo apt update -y
Get:1 http://mirror.0x.sg/ubuntu kinetic InRelease [267 kB]
Hit:2 https://dl.google.com/linux/chrome/deb stable InRelease                                                                                                                
Hit:3 http://mirror.0x.sg/ubuntu kinetic-security InRelease                                                                                                                                                       
Hit:4 http://mirror.0x.sg/ubuntu kinetic-updates InRelease                                                                                                                                                        
Hit:5 http://mirror.0x.sg/ubuntu kinetic-backports InRelease                                                                                                                                                      
Hit:6 https://packages.treasuredata.com/4/ubuntu/jammy jammy InRelease                                                                                                                                            
Get:7 https://download.docker.com/linux/ubuntu impish InRelease [48.9 kB]                                                                                                                                         
Hit:8 https://deb.nodesource.com/node_18.x jammy InRelease                                                                                                                                                        
0% [Waiting for headers] [Connecting to ppa.launchpadcontent.net (185.125.190.52)] [Connecting to ftp.postgresql.org (217.196.149.55)]                                                 
Get:9 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [8,993 B]                                                     
Err:9 https://packages.cloud.google.com/apt kubernetes-xenial InRelease                                                             
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY B53DC80D13EDEF05
Hit:10 https://ftp.postgresql.org/pub/pgadmin/pgadmin4/apt/jammy pgadmin4 InRelease
Hit:11 https://ppa.launchpadcontent.net/mati75/gimp30/ubuntu kinetic InRelease
Fetched 325 kB in 1s (317 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://packages.cloud.google.com/apt kubernetes-xenial InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY B53DC80D13EDEF05
W: Failed to fetch https://apt.kubernetes.io/dists/kubernetes-xenial/InRelease  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY B53DC80D13EDEF05
W: Some index files failed to download. They have been ignored, or old ones used instead.

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

@khteh khteh added area/release-eng Issues or PRs related to the Release Engineering subproject kind/bug Categorizes issue or PR as related to a bug. sig/release Categorizes an issue or PR as relevant to SIG Release. labels Jan 17, 2023
@dims
Copy link
Member

dims commented Jan 17, 2023

xref: #2860

@saschagrunert
Copy link
Member

Might this be a transient failure? I'm not able to reproduce with plain Ubuntu:22.10:

podman run -it ubuntu:22.10 sh -c '\
    apt-get update && \
    apt-get install -y apt-transport-https ca-certificates curl && \
    curl -fsSLo /etc/apt/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && \
    echo "deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | tee /etc/apt/sources.list.d/kubernetes.list && \
    apt-get update && \
    apt-get install -y kubelet'
…
Setting up kubelet (1.26.0-00) ...

@lukeweller
Copy link

lukeweller commented Jan 18, 2023

I'm seeing the same issue on my Ubuntu 20.04.5 machine:

$ sudo apt update             
...
Err:2 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY B53DC80D13EDEF05
...
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://packages.cloud.google.com/apt kubernetes-xenial InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY B53DC80D13EDEF05
W: Failed to fetch https://apt.kubernetes.io/dists/kubernetes-xenial/InRelease  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY B53DC80D13EDEF05
W: Some index files failed to download. They have been ignored, or old ones used instead.

@khteh
Copy link
Author

khteh commented Jan 18, 2023

Might this be a transient failure? I'm not able to reproduce with plain Ubuntu:22.10:

podman run -it ubuntu:22.10 sh -c '\
    apt-get update && \
    apt-get install -y apt-transport-https ca-certificates curl && \
    curl -fsSLo /etc/apt/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && \
    echo "deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | tee /etc/apt/sources.list.d/kubernetes.list && \
    apt-get update && \
    apt-get install -y kubelet'
…
Setting up kubelet (1.26.0-00) ...

This works for me.

@saschagrunert
Copy link
Member

@lukeweller could you provide us a reproducer here?

@aronneberg
Copy link

aronneberg commented Jan 18, 2023

I think the reproducer would be running apt update in an existing installation without manually updating the key. I ran into the same problem and just curling the new key fixed it for me.
Note that the download location in the Kubernetes help pages changed and you might have to run
sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
instead or update the location of the key file.

@khteh I can reproduce the apt-key behaviour, but it might be unrelated to the Kubernetes key. If you manually put the key in /etc/apt/trusted.gpg.d/ apt-key list shows it correctly and the apt update works.

@lukeweller
Copy link

lukeweller commented Jan 18, 2023

@saschagrunert This line is failing on my machine: curl -fsSLo /etc/apt/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg

$ curl -fsSLo /etc/apt/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
curl: (23) Failed writing body (0 != 1210)

The recommendation from @aronneberg worked for me. After running sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg, I am able to apt update normally.

@saschagrunert
Copy link
Member

@lukeweller @aronneberg thank you for the clarification! I don't see much we can do here from the project perspective since we don't own the keys. If they changed, then on purpose I assume. 🤔

@N3X14
Copy link

N3X14 commented Jan 19, 2023

I used the information provided here: https://cloud.google.com/compute/docs/troubleshooting/known-issues#keyexpired-2

Specifically:

Command (Ubuntu):
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -

Command (Debian):
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

Worked fine on the latest Debian Bullseye + Ubuntu 22.04.5 releases.

Cheers

@fabio-blanco
Copy link

I'm on a Ubuntu 20.04.5 machine and the @aronneberg workaround worked for me while @N3X14 workaround don't.

@jaxbulsara
Copy link

jaxbulsara commented Jan 29, 2023

I think the reproducer would be running apt update in an existing installation without manually updating the key. I ran into the same problem and just curling the new key fixed it for me. Note that the download location in the Kubernetes help pages changed and you might have to run sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg instead or update the location of the key file.

Thanks, this worked for me. The error I was getting was (Linux Mint 20.3 / Ubuntu 20.04):

W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://packages.cloud.google.com/apt kubernetes-xenial InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY B53DC80D13EDEF05
W: Failed to fetch https://apt.kubernetes.io/dists/kubernetes-xenial/InRelease  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY B53DC80D13EDEF05
W: Some index files failed to download. They have been ignored, or old ones used instead.

@jtele2
Copy link

jtele2 commented Jan 31, 2023

I have the same issue and none of the workarounds are working. I've tried all the commands listed here and in the Kubectl installation. I am still getting this error upon running apt-get update:

Get:4 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [8993 B]
Err:4 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY B53DC80D13EDEF05
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.5 LTS"

@Zmicier14
Copy link

Zmicier14 commented Feb 3, 2023

@jtele2 thanks for reference for for the kubectl installation.

W: Failed to fetch https://apt.kubernetes.io/dists/kubernetes-xenial/InRelease  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY B53DC80D13EDEF05

After these commands:
sudo curl -fsSLo /etc/apt/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
the warning has gone.

@emmjam
Copy link

emmjam commented Feb 27, 2023

The provided url, https://packages.cloud.google.com/apt/doc/apt-key.gpg, produces a 500 error

@dims
Copy link
Member

dims commented Feb 27, 2023

Please see the email thread - https://groups.google.com/a/kubernetes.io/g/dev/c/gR5xDzzIcW8/m/SK5ARWkGAQAJ

and the work around in that thread or here - kubernetes/k8s.io#4837 (comment)

/close

@k8s-ci-robot
Copy link
Contributor

@dims: Closing this issue.

In response to this:

Please see the email thread - https://groups.google.com/a/kubernetes.io/g/dev/c/gR5xDzzIcW8/m/SK5ARWkGAQAJ

and the work around in that thread or here - kubernetes/k8s.io#4837 (comment)

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@emilpeychev
Copy link

I keep getting
curl: (23) Failed writing body (0 != 1177
What could the problem be?

@Adam111p
Copy link

follow
https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/#install-using-native-package-management

@nyetwurk
Copy link

nyetwurk commented May 3, 2023

sudo mkdir -p /etc/apt/keyrings
echo "deb [signed-by=/etc/apt/keyrings/kubernetes.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes.gpg

From below

On Ubuntu 20.04.6, there is no /etc/apt/keyrings directory to start with

Please instead read this advice (quoting from the docs):

Note: In releases older than Debian 12 and Ubuntu 22.04, /etc/apt/keyrings does not exist by default. You can create this directory if you need to, making it world-readable but writeable only by admins.

For the Debian project explanation of where to put the trust data, read https://wiki.debian.org/DebianRepository/UseThirdParty#OpenPGP_certificate_distribution

Please also avoid recommending that Kubernetes users change the system-level package trust within /etc/apt/trusted.gpg.d. We'd much rather follow Debian's recommended approach, which also states:

The certificate MUST NOT be placed in /etc/apt/trusted.gpg.d or loaded by apt-key add.

@hmv-workspace
Copy link

FIXED!

sudo apt update

Err:2 https://packages.cloud.google.com/apt kubernetes-xenial InRelease The following signatures couldn't be verified because the public key is not available: NO_PUBKEY B53DC80D13EDEF05

sudo curl -fsSLo /etc/apt/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg

curl: (23) Failed writing body (0 != 1210)

sudo mkdir /etc/apt/keyrings
sudo curl -fsSLo /etc/apt/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
sudo apt update

Get:7 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [65.7 kB]

It seems curl is unable to create the keyrings folder inside the /etc/apt/ directory. I just created it manually and it worked!

@jhedgpeth
Copy link

I was also successful following the advice here with one small adjustment, per my specific circumstances.
I am on debian 12. I made /etc/apt/keyrings directory - permissions 0755
The curl command populated gpg file as expected, yet apt update returned NO_PUBKEY error.
I noted that due to my umask the gpg file was created with 0640 permissions. Changing the file to 0644 solved the issue for me.

extra step that I needed:

chmod 0644 /etc/apt/keyrings/kubernetes-archive-keyring.gpg

four posterity :)

Xynnn007 added a commit to Xynnn007/enclave-cc that referenced this issue Jun 1, 2023
As
kubernetes/release#2862 (comment),
public key release of kubernetes have been changed.

Signed-off-by: Xynnn007 <xynnn@linux.alibaba.com>
MoritzWeber0 added a commit to DSD-DBS/capella-collab-manager that referenced this issue Jun 1, 2023
The GPG used in the official documentation is expired.
However, there is another official source with a new GPG key available.
This commit changes the source to fetch the new GPG key.

kubernetes/release#2862
fidencio added a commit to fidencio/kata-containers that referenced this issue Jun 1, 2023
The current method has been failing every now and then, and was reported
on kubernetes/release#2862.

Ding poked me and suggested to do this change here, so here we go. :-)

Fixes: kata-containers#7006

Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
fidencio pushed a commit to confidential-containers/enclave-cc that referenced this issue Jun 1, 2023
As
kubernetes/release#2862 (comment),
public key release of kubernetes have been changed.

Signed-off-by: Xynnn007 <xynnn@linux.alibaba.com>
@Redw0od
Copy link

Redw0od commented Jun 13, 2023

Ran into this issue today while making a debian container image.
None of the comments I read here worked, here's what worked for me:

mkdir -p /etc/apt/keyrings
echo "deb [signed-by=/etc/apt/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | gpg --dearmor -o /etc/apt/keyrings/cloud.google.gpg

@pauljones0
Copy link

The error I saw:

Get:2 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [8,993 B]
Err:2 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY B53DC80D13EDEF05
Reading package lists... Done
W: GPG error: https://packages.cloud.google.com/apt kubernetes-xenial InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY B53DC80D13EDEF05
E: The repository 'https://apt.kubernetes.io kubernetes-xenial InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.

The fix was almost the same as the debian comment above:

sudo mkdir -p /etc/apt/keyrings
echo "deb [signed-by=/etc/apt/keyrings/kubernetes.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes.gpg

SethBodine pushed a commit to SethBodine/audit-tools that referenced this issue Jun 22, 2023
… (comment))

2. Fixed prowler v3
3. Fixed Azure CLI build issues
4. Updated scripts and source files
5. Disabled aws-list-all for the time being
@cq-liu
Copy link

cq-liu commented Jun 28, 2023

Please see the email thread - https://groups.google.com/a/kubernetes.io/g/dev/c/gR5xDzzIcW8/m/SK5ARWkGAQAJ

and the work around in that thread or here - kubernetes/k8s.io#4837 (comment)

/close

was using the google url and worked fine, suddenly failed recently. changed to this k8s url, back to work

fidencio added a commit to fidencio/kata-containers that referenced this issue Jun 28, 2023
The current method has been failing every now and then, and was reported
on kubernetes/release#2862.

Ding poked me and suggested to do this change here, so here we go. :-)

Fixes: kata-containers#7006

Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
(cherry picked from commit 26f7520)
@yunwu-cn
Copy link

@ffabretti19
Copy link

ffabretti19 commented Jul 8, 2023

This script worked for me on Ubuntu 22.04.2 LTS using WSL

curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --yes --dearmor -o /usr/share/keyrings/kubernetes-archive-keyring.gpg

echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list > /dev/null

andalike added a commit to andalike/kubernetes-v2 that referenced this issue Jul 18, 2023
@jdboris
Copy link

jdboris commented Jul 18, 2023

This script worked for me on Ubuntu 22.04.2 LTS using WSL

curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --yes --dearmor -o /usr/share/keyrings/kubernetes-archive-keyring.gpg

echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list > /dev/null

This is the only thing that worked for me on Ubuntu 23.04 on an AWS EC2 instance. It looks like I was only missing the arch option.

@mfhaln
Copy link

mfhaln commented Jul 19, 2023

Note that:
curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-archive-keyring.gpg
results in curl (60) error:
curl: (60) SSL: no alternative certificate subject name matches target host name 'packages.cloud.google.com'

after long googling, it turns out curl doesn't recognise the host with 'https://' specified. instead below worked (Ubuntu 22.04):
curl -fsSL packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-archive-keyring.gpg

@chancez
Copy link
Member

chancez commented Jul 19, 2023

@mfhaln your issue is more likely due to this bug shipped in a recent Ubuntu package update https://bugs.launchpad.net/ubuntu/+source/curl/+bug/2028170

dominikzorgnotti added a commit to dominikzorgnotti/script-junkyard that referenced this issue Jul 20, 2023
cavcrosby added a commit to cavcrosby/homelab-cm that referenced this issue Aug 19, 2023
The k8s_kubeadm role now processes the Kubernetes's (Google's) GPG key
by dearmoring the file before installing it onto the system. This was
done because before this, I was getting an error much like
kubernetes/release#2862. This solution follows
the recommendations from the PR that derived from that issue.

Dearmoring was not done originally because Kubernetes v1.24
documentation (at the time and as of now) for installing kubeadm does
not instruct this when downloading the key.
cavcrosby added a commit to cavcrosby/homelab-cm that referenced this issue Aug 19, 2023
The k8s_kubeadm role now processes the Kubernetes's (Google's) GPG key
by dearmoring the file before installing it onto the system. This was
done because before this, I was getting an error much like
kubernetes/release#2862. This solution follows
the recommendations from the PR that derived from this issue.

Dearmoring was not done originally because Kubernetes v1.24
documentation (at the time and as of now) for installing kubeadm does
not instruct this when downloading the key.
@sbisig
Copy link

sbisig commented Aug 31, 2023

following command also worked for me (having same issue trying to install google-cloud-cli):

curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --yes --dearmor -o /usr/share/keyrings/cloud.google
.gpg

@conorab
Copy link

conorab commented Sep 1, 2023

In my case, the guides I had used in the past (https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/) defined the APT repository similarly to the following:
deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main

However, the instructions I read to fix this issue said to download the new key and import it to APT as follows:
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

This makes no difference since APT has been told to fetch the key from /etc/apt/keyrings/kubernetes-apt-keyring.gpg explicitly. To fix this, delete (maybe make a backup first) /etc/apt/keyrings/kubernetes-apt-keyring.gpg and change /etc/apt/sources.list.d/kubernetes.list to the following:
deb https://apt.kubernetes.io/ kubernetes-xenial main

Then run the following to have APT update the repository:
apt update

@xmudrii
Copy link
Member

xmudrii commented Sep 1, 2023

Hello folks,

The legacy repositories (apt.kubernetes.io and yum.kubernetes.io) are deprecated. We don't have a way to fix this for these legacy repositories. Instead, we strongly recommend migrating to the new community-owned repositories. Please see the following announcements for more details:

Given that there's nothing else we can do here, I'm going to lock this issue. If you encounter any issue with the new community-owned repositories, please open a new issue.

@kubernetes kubernetes locked as resolved and limited conversation to collaborators Sep 1, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area/release-eng Issues or PRs related to the Release Engineering subproject kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/release Categorizes an issue or PR as relevant to SIG Release.
Projects
None yet
Development

No branches or pull requests