Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Installation on CentOS Stream 9 aborts due to failed GPG check #5588

Closed
1player opened this issue May 20, 2022 · 29 comments
Closed

Installation on CentOS Stream 9 aborts due to failed GPG check #5588

1player opened this issue May 20, 2022 · 29 comments

Comments

@1player
Copy link

1player commented May 20, 2022

Environmental Info:

Node(s) CPU architecture, OS, and Version: CentOS Stream 9 on x86_64

Describe the bug:

Ran curl -sfL https://get.k3s.io | sh - as root, fails during package installation with:

Downloading Packages:
(1/3): container-selinux-2.183.0-1.el9.noarch.rpm                                                             313 kB/s |  47 kB     00:00
(2/3): policycoreutils-python-utils-3.3-5.el9.noarch.rpm                                                      433 kB/s |  74 kB     00:00
(3/3): k3s-selinux-1.1-1.el8.noarch.rpm                                                                        28 kB/s |  20 kB     00:00
----------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                         162 kB/s | 142 kB     00:00
Rancher K3s Common (stable)                                                                                   4.6 kB/s | 2.4 kB     00:00
Importing GPG key 0xE257814A:
 Userid     : "Rancher (CI) <ci@rancher.com>"
 Fingerprint: C8CF F216 4551 26E9 B9C9 18BE 925E A29A E257 814A
 From       : https://rpm.rancher.io/public.key
Key imported successfully
Import of key(s) didn't help, wrong key(s)?
Problem opening package k3s-selinux-1.1-1.el8.noarch.rpm. Failing package is: k3s-selinux-1.1-1.el8.noarch
 GPG Keys are configured as: https://rpm.rancher.io/public.key
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: GPG check FAILED

Steps To Reproduce:

Run curl -sfL https://get.k3s.io | sh - on a CentOS Stream 9 or other RPM distribution.

@1player 1player changed the title Install script on CentOS Stream 9 fails due to GPG check Install script on CentOS Stream 9 aborts due to failed GPG check May 20, 2022
@1player
Copy link
Author

1player commented May 20, 2022

Given that the repo config uses https://rpm.rancher.io/public.key for the GPG key, is it out of date or have the RPM packages been compromised?

@1player 1player changed the title Install script on CentOS Stream 9 aborts due to failed GPG check Installation on CentOS Stream 9 aborts due to failed GPG check May 20, 2022
@StefanNienhuis
Copy link

Same issue on RHEL 9. Interestingly I was able to perform a successful install on RHEL 9 earlier.

@hbjydev
Copy link

hbjydev commented May 24, 2022

When I try and run dnf install --nogpgcheck k3s-selinux I get packages does not verify: Header V4 RSA/SHA1 Signature, key ID e257814a: BAD -- this could be due to RHEL 9 deprecating and no longer enabling SHA1 out of the box

@hbjydev
Copy link

hbjydev commented May 24, 2022

In fact, yes, after running update-crypto-policies --set DEFAULT:SHA1 (see https://access.redhat.com/articles/6846411), installing k3s-selinux works just fine.

This is a repo/packaging problem, I think Rancher needs to sign their packages with a SHA256 or SHA521 key instead of SHA1.

@brandond
Copy link
Member

brandond commented May 24, 2022

Yeah, since we only technically support EL7 and EL8 we haven't updated the signing hash algorithms to account for the deprecation of SHA-1. We'll take a look at addressing this for our next release cycle.

Note that this has to do with the hash algorithm used when signing the packages, there is nothing wrong with the keys themselves.

@hbjydev
Copy link

hbjydev commented May 24, 2022

Yeah, sorry, that's what I meant. :)

@mplinuxgeek
Copy link

Another affected user here, but just to add to @hbjydev's comment about updating crypto policies, SHA1 can be disabled after install like this update-crypto-policies --set DEFAULT:NO-SHA1

@hbjydev
Copy link

hbjydev commented May 25, 2022

Another affected user here, but just to add to @hbjydev's comment about updating crypto policies, SHA1 can be disabled after install like this update-crypto-policies --set DEFAULT:NO-SHA1

It certainly could be, but I don't think it's wise until they fix the package signatures, because doing this will just break any updates to the package until it's sorted out at the repo end.

For a dev or test box, sure, but I can't recommend if you're in production.

@brandond
Copy link
Member

brandond commented May 25, 2022

For what it's worth, we don't update the selinux rpm very often.

@hbjydev
Copy link

hbjydev commented May 25, 2022

@brandond That's fair, yeah, but I'd still rather not need to turn it on and off either side, hence I'm just cautioning that it's not a great solution. :)

@1player
Copy link
Author

1player commented May 26, 2022

Yeah, since we only technically support EL7 and EL8 we haven't updated the signing hash algorithms to account for the deprecation of SHA-1. We'll take a look at addressing this for our next release cycle.

Not to put any pressure, but when is the next release cycle due? I can wait for a few days to install k3s if it's due shortly, otherwise I'll have to try one of the workaround I'd prefer to avoid.

@mplinuxgeek
Copy link

Another affected user here, but just to add to @hbjydev's comment about updating crypto policies, SHA1 can be disabled after install like this update-crypto-policies --set DEFAULT:NO-SHA1

It certainly could be, but I don't think it's wise until they fix the package signatures, because doing this will just break any updates to the package until it's sorted out at the repo end.

For a dev or test box, sure, but I can't recommend if you're in production.

I'm not sure anyone is using CentOS 9 Stream in production, however RHEL 9 went GA last week so it won't be long before it starts making its way into prod environments and as this issue also affects RHEL 9 people are going to be eager to test things out, I'm already doing test deployments on RHEL 9 which other than the SHA1 issue and another minor issue not directly related to k3s has gone rather well so far.

The problem here is as far back as 2005 SHA-1 was considered not secure, so its kind of surprising to see a project in 2022 still signing packages with SHA-1. Collision attacks on SHA-1 are feasible in 2022, there's an interesting answer posted to stackexchange that calculates the time required for a successful attack using current generation GPU's, and before anyone scoff's at the costs of running multiple high end GPU's consider that crypto miners are running small farms with enough RTX30xx's and/or RX 6x00's to generate successful attacks in a few months from their own homes.

Please don't take this as a personal attack, its not, I do understand that things get overlooked sometimes.

@brandond
Copy link
Member

brandond commented May 26, 2022

The rpmsign command on el7 and el8 still only uses SHA-1 by default; it's not something we've gone out of our way to hold back.

Upstream Kubernetes patches for this month just came out yesterday and we are working on validating those again the current support matrix; validating el9 will probably be slotted in for next months cycle but we may be able to get el9 packages out for testing sooner than that.

Not to be too blunt about it, but if you're super paranoid about running untrusted software (or software with 'weak' signatures) you're probably not using containers, right? Image signing still isn't really a thing so all you have is TLS to verify the source of your pulls...

@vwbusguy
Copy link

vwbusguy commented May 26, 2022

"Not to be too blunt about it, but if you're super paranoid about running untrusted software (or software with 'weak' signatures) you're probably not using containers, right?"

It's more complicated than that. The script overrides the .repo file, so even setting gpgcheck=0 won't fix it. Supply chain attacks are also a very real threat to enterprise environments, but ignoring that doesn't provide an obvious workaround here. I downloaded the updated rpm and installed it with rpm -Uvh --nosignature and was able to continue that way but it is not a great upgrade experience to have to go that far into what is supposed to be a "here, run this script again" as the docs present.

@vwbusguy
Copy link

vwbusguy commented May 26, 2022

I'm not sure anyone is using CentOS 9 Stream in production,

Yes, we are, with k3s for several months now, and it's actually been working great. This recent upgrade was my first real snag.

@mplinuxgeek
Copy link

Not to be too blunt about it, but if you're super paranoid about running untrusted software (or software with 'weak' signatures) you're probably not using containers, right? Image signing still isn't really a thing so all you have is TLS to verify the source of your pulls...

Not paranoid, just "aware", in my job I have to answer to security people and customers if I do things like enable a deprecated crypto algorithm.

@1player
Copy link
Author

1player commented May 31, 2022

Work around until this is fixed:

Run the script once to get the rancher repo added. Then:

dnf install container-selinux
dnf download k3s-selinux
rpm -Uvh --nosignature ./k3s-selinux-1.1-1.el8.noarch.rpm

Then run the install script again, this time it'll go to completion.

@hbjydev
Copy link

hbjydev commented Jun 5, 2022

Any movement on this at all?

@askb
Copy link

askb commented Jun 8, 2022

Been hitting a similar issue with CentOS streams 8. Is this related?

    openstack: TASK [Install sigul client for EL 8 (Streams)] *********************************
    openstack: Wednesday 08 June 2022  20:51:03 +1000 (0:00:00.026)       0:12:18.049 ********
    openstack: fatal: [default]: FAILED! => {
    openstack:     "changed": false
    openstack: }
    openstack:
    openstack: MSG:
    openstack:
    openstack: Failed to validate GPG signature for sigul-1.1-1.el8.infra.x86_64
   openstack: TASK [Install base packages] ***************************************************
    openstack: Wednesday 08 June 2022  20:33:16 +1000 (0:00:01.530)       0:07:03.762 ********
    openstack: fatal: [default]: FAILED! => {
    openstack:     "changed": false
    openstack: }
    openstack:
    openstack: MSG:
    openstack:
    openstack: Failed to validate GPG signature for epel-release-8-15.el8.noarch
    openstack:
    openstack: PLAY RECAP *********************************************************************
    openstack: default                    : ok=7    changed=2    unreachable=0    failed=1    skipped=3    rescued=0    ignored=0

@brandond
Copy link
Member

brandond commented Jun 8, 2022

Stream 8 is basically EL9 but worse, so yes.

We will address this once our current in-flight releases are complete.

@vwbusguy
Copy link

vwbusguy commented Jun 9, 2022

Steam 8 is definitely not Stream 9 and if it's broken there, it is soon to be in RHEL, AlmaLinux, and Rocky. CentOS Stream just did you a favor in giving you a heads up for the scope of impact soon for a significant portion of the project's end users.

@brandond
Copy link
Member

brandond commented Jun 10, 2022

An updated package is now available in the testing channel. You should now be able to do:

curl -ks https://get.k3s.io | INSTALL_K3S_CHANNEL=testing sh -
[root@2975a4d1a4bf /]# cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="9.0 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.0"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux 9.0 (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/9/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.0
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.0"

[root@2975a4d1a4bf /]# curl -ks https://get.k3s.io | INSTALL_K3S_SKIP_START=1 INSTALL_K3S_CHANNEL=testing sh -
[INFO]  Finding release for channel testing
[INFO]  Using v1.24.1-rc5+k3s1 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.24.1-rc5+k3s1/sha256sum-amd64.txt
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.24.1-rc5+k3s1/k3s
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
Updating Subscription Management repositories.
Unable to read consumer identity
Subscription Manager is operating in container mode.

This system is not registered with an entitlement server. You can use subscription-manager to register.

Rancher K3s Common (testing)                                                                                                                                 3.6 kB/s | 3.0 kB     00:00
Dependencies resolved.
=============================================================================================================================================================================================
 Package                                    Architecture                          Version                                    Repository                                                 Size
=============================================================================================================================================================================================
Installing:
 k3s-selinux                                noarch                                1.2-2.el8                                  rancher-k3s-common-testing                                 20 k

Transaction Summary
=============================================================================================================================================================================================
Install  1 Package

Total download size: 20 k
Installed size: 94 k
Downloading Packages:
k3s-selinux-1.2-2.el8.noarch.rpm                                                                                                                             111 kB/s |  20 kB     00:00
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                        110 kB/s |  20 kB     00:00
Rancher K3s Common (testing)                                                                                                                                 7.1 kB/s | 2.4 kB     00:00
Importing GPG key 0xD161F542:
 Userid     : "Rancher (CI) <ci@rancher.com>"
 Fingerprint: 856A 0069 529C A63B 21AA 4E0A 089F A20E D161 F542
 From       : https://rpm-testing.rancher.io/public.key
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                     1/1
  Running scriptlet: k3s-selinux-1.2-2.el8.noarch                                                                                                                                        1/1
  Installing       : k3s-selinux-1.2-2.el8.noarch                                                                                                                                        1/1
  Running scriptlet: k3s-selinux-1.2-2.el8.noarch                                                                                                                                        1/1
  Verifying        : k3s-selinux-1.2-2.el8.noarch                                                                                                                                        1/1
Installed products updated.

Installed:
  k3s-selinux-1.2-2.el8.noarch

Complete!

@vwbusguy
Copy link

This is now fixed for me. I upgraded to 1.23.8 on CentOS Stream 9 normally with the regular update script today without any workarounds.

@nemith
Copy link

nemith commented Aug 7, 2022

Stream 8 is basically EL9 but worse, so yes.

No. Stream 8 is used for EL8 and Stream 9 is used for EL9. The confusion on the "rolling release" is strong, but really simple. Updates to Centos 8 will show up as minor upgrade for EL8 and similar for Cent9 -> EL9.

@KyleSanderson
Copy link

Any update on this?

Downloading Packages:
(1/9): k3s-selinux-1.1-1.el8.noarch.rpm                                                                                                                          33 kB/s |  20 kB     00:00
(2/9): container-selinux-2.188.0-1.el9_0.noarch.rpm                                                                                                             243 kB/s |  47 kB     00:00
(3/9): python3-setools-4.4.0-4.el9.x86_64.rpm                                                                                                                   478 kB/s | 547 kB     00:01
(4/9): python3-audit-3.0.7-101.el9_0.2.x86_64.rpm                                                                                                               162 kB/s |  83 kB     00:00
(5/9): checkpolicy-3.3-1.el9.x86_64.rpm                                                                                                                         273 kB/s | 339 kB     00:01
(6/9): policycoreutils-python-utils-3.3-6.el9_0.noarch.rpm                                                                                                      149 kB/s |  68 kB     00:00
(7/9): python3-setuptools-53.0.0-10.el9.noarch.rpm                                                                                                              297 kB/s | 841 kB     00:02
(8/9): python3-libsemanage-3.3-2.el9.x86_64.rpm                                                                                                                 222 kB/s |  80 kB     00:00
(9/9): python3-policycoreutils-3.3-6.el9_0.noarch.rpm                                                                                                           612 kB/s | 2.0 MB     00:03
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                           737 kB/s | 4.0 MB     00:05
Problem opening package k3s-selinux-1.1-1.el8.noarch.rpm
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'yum clean packages'.
Error: GPG check FAILED

@mickare
Copy link

mickare commented Aug 14, 2022

I'm having the same issue on CentOS Stream 9.
@KyleSanderson I think we'll have to wait for the next release of k3s-selinux.

@brandond The testing channel did not work for me.
@1player Thx, that workaround is working.

# dnf install --nogpgcheck k3s-selinux
Last metadata expiration check: 0:10:50 ago on Sun 14 Aug 2022 12:43:08 PM UTC.
Dependencies resolved.
================================================================================================================================================================
 Package                                        Architecture             Version                              Repository                                   Size
================================================================================================================================================================
Installing:
 k3s-selinux                                    noarch                   1.1-1.el8                            rancher-k3s-common-stable                    20 k
Installing dependencies:
 container-selinux                              noarch                   3:2.189.0-1.el9                      appstream                                    49 k
 policycoreutils-python-utils                   noarch                   3.4-2.el9                            appstream                                    75 k

Transaction Summary
================================================================================================================================================================
Install  3 Packages

Total size: 143 k
Installed size: 268 k
Is this ok [y/N]: y
Downloading Packages:
[SKIPPED] container-selinux-2.189.0-1.el9.noarch.rpm: Already downloaded                                                                                       
[SKIPPED] policycoreutils-python-utils-3.4-2.el9.noarch.rpm: Already downloaded                                                                                
[SKIPPED] k3s-selinux-1.1-1.el8.noarch.rpm: Already downloaded                                                                                                 
Running transaction check
Transaction check succeeded.
Running transaction test
RPM: warning: Signature not supported. Hash algorithm SHA1 not available.
RPM: warning: Signature not supported. Hash algorithm SHA1 not available.
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: Transaction test error:
  package k3s-selinux-1.1-1.el8.noarch does not verify: Header V4 RSA/SHA1 Signature, key ID e257814a: BAD

@nemith
Copy link

nemith commented Aug 15, 2022

Using testing and the Centos 8 packages worked for me on Centos 9, however I am not using the k3s install script but just set this via ansible:

[bbennett@kube01]~% cat /etc/yum.repos.d/rancher-k3s-common-testing.repo 
[rancher-k3s-common-testing]
async = 1
baseurl = https://rpm-testing.rancher.io/k3s/testing/common/centos/8/noarch
gpgkey = https://rpm-testing.rancher.io/public.key
name = Rancher K3 Common (testing)

@brandond
Copy link
Member

the updated k3s-selinux package should now be available in other channels.

@nemith
Copy link

nemith commented Aug 16, 2022

Reverted back to stable and everything is good. Any chance on packaging specifically for centos9? Probably not a big deal now unless se policies ore dependencies drift more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

10 participants