GPG key expires on March the 30th

[NotMyFault]

Service(s)

pkg.jenkins.io

Summary

➜  ~ curl -s https://pkg.jenkins.io/debian/jenkins.io.key | gpg2 --show-keys
pub   rsa4096 2020-03-30 [SC] [expires: 2023-03-30]
      62A9756BFD780C377CF24BA8FCEF32E745F2C3D5
uid                      Jenkins Project <jenkinsci-board@googlegroups.com>
sub   rsa4096 2020-03-30 [E] [expires: 2023-03-30]

I assume this is not covered by #3323, given the issue covers the code signing certificate only.
Don't hesitate to correct me if I'm wrong :P

Reproduction steps

No response

[dduportal]

You are correct @NotMyFault : the GPG key expiration is NOT covered by #3323.

• The current GPG key is used for
  • The release process of Jenkins Core
  • The packaging process to provide signing of metadata for APT, Yum and other packages
  • Please note that the public key is also present in the pkg.origin.jenkins.io machine (used to be managed as code but not anymore): jenkins-infra/jenkins-infra@aaef138#diff-61780a237304a0a4857da29c6a07c5211700546f8f41ca10ff8783727f1df0ca

Backstory: the current GPG key (3 year valid, ending the 30 March 2023) was generated as part of #1881, to handover the Jenkins releases from Kohsuke's fromer GPG key. Reference here: https://groups.google.com/g/jenkins-infra/c/c7m_72vUTXE/m/37zw9GXhAQAJ.

We can either extend the expiration date or create a new key: both will have the same impact on users who will need to import the new/extended key:

• GPG key used to sign debian packages is expired cli/cli#6175
• https://www.hashicorp.com/security (section "PGP Public Key") for instance.

⚠️ How much time for the new key? 3 years? 8 years ? something else ?

[NotMyFault]

⚠️ How much time for the new key? 3 years? 8 years ? something else ?

Extending the existing key's life span by 3 years sounds solid to me. Not too long, but not too short either, to renew keys on the consumer's end too frequently.

The changelogs starting with 2.398 and 2.387.2 onwards have to declare that keys have changed and importing the new ones is mandatory.

[olblak]

I would also extend the gpg key for three years. If you plan to regenerate a new one then you'll need to communicate about it

[dduportal]

I would also extend the gpg key for three years. If you plan to regenerate a new one then you'll need to communicate about it

Extending the gpg key also requires communication because end users need to reimport it to get extended date alas

[dduportal]

A new GPG key was generated for the Jenkins project with the following attributes:

• Name: Jenkins Project
• Email: jenkinsci-board@googlegroups.com
• Expiration date: 26 March 2026 (3 years)
• Encoding: ed25519 (Default for GNUPG 2.4.x). ⚠️ We'll fall back to rsa4096 if the package fails to be generated (or maybe upgrade the package tooling...)

This new key has been inserted into the Azure Release Vault, with a secret name jenkins-release-pgp-2023 (Different than the previous secret one so we can shift to one or the other).

Next step:

• Add the key to pkg.origin.jenkins.io (Puppet)
• Add a PR to jenkins-infra/release to specify the new key for the Weekly line
• Share the public key here
• Prepare the list of communication media (cc @MarkEWaite)
• Cleanup the old key once expired (next week)

[dduportal]

Adding the new public key (with another name to avoid replacing the current one) to pkg.jenkins.io: jenkins-infra/jenkins-infra#2720

[dduportal]

Adding the new key for the upcoming weekly: jenkins-infra/release#352

[dduportal]

Communication channels:

• Jenkins blog jenkins.io PR-6205
• Changelogs:
  • Weekly (this week): https://www.jenkins.io/changelog/ jenkins.io PR-6201 as an improvement to jenkins.io PR-6171
  • LTS (next week): https://www.jenkins.io/changelog-stable/ jenkins.io PR-6173
• Packages pages (it looks like each package as a copy of the key + there is a .key.info file that might be used by the HTML static files) jenkins-infra puppet PR 2720:
  • Debian/Ubuntu
    • Weekly: https://pkg.origin.jenkins.io/debian/ (old key at https://pkg.jenkins.io/debian/jenkins.io.key, new key will be at https://pkg.jenkins.io/debian/jenkins.io-2023.key)
    • LTS
  • Red Hat, etc.
• Jenkins.io:
• https://www.jenkins.io/download/verify/ jenkins.io PR-6211
• install pages jenkins.io PR-6205
• Add the blog post to the Jumbotron for more visibility jenkins.io PR-6204
• Community.jenkins.io: post
• Gitter channels:
  • jenkins/jenkins
• Developer mailing list?

[dduportal]

Proposal for a new key comes from the need for clarity for end users. It can be confusing to "only" check the key ID and not a guranatee that the "extended" key is imported and used properly.

As the end users will have to import something one time, ensuring that we have a new key, with a new ID and a new time windows looked clearer: either you got the new key or not.
It was inspired by https://docs.datadoghq.com/agent/guide/linux-agent-2022-key-rotation/?tab=debianubuntu.

However it might be a mistake: please express yourself here with the rationale if you do not agree with this proposal (I do not want to decide alone :) )

[NotMyFault]

+1 for the strategy proposed

[saper]

Will ed25519 work on the current platforms? Just tried to import some dummy ed25519 GPG key on Almalinux 8 (compatible with the current Red Hat Enteprise Linux 8):

$ sudo rpm -vvv --import /tmp/k.asc
ufdio:       1 reads,    17154 total bytes in 0.000010 secs
ufdio:       1 reads,     5442 total bytes in 0.000004 secs
ufdio:       1 reads,    17154 total bytes in 0.000009 secs
ufdio:       1 reads,      393 total bytes in 0.000003 secs
D: loading keyring from pubkeys in /var/lib/rpm/pubkeys/*.key
D: couldn't find any keys in /var/lib/rpm/pubkeys/*.key
D: loading keyring from rpmdb
D: opening  db environment /var/lib/rpm cdb:0x401
D: opening  db index       /var/lib/rpm/Packages 0x400 mode=0x0
D: locked   db index       /var/lib/rpm/Packages
D: opening  db index       /var/lib/rpm/Name 0x400 mode=0x0
D:  read h#     405 
Header SHA1 digest: OK
D: added key gpg-pubkey-3abb34f8-5ffd890e to keyring
D: added subkey 0 of main key gpg-pubkey-3abb34f8-5ffd890e to keyring
D:  read h#     459 
Header SHA1 digest: OK
D: added key gpg-pubkey-2f86d6a1-5cf7cefb to keyring
D:  read h#     923 
Header SHA1 digest: OK
D: added key gpg-pubkey-37576165-5b22521c to keyring
D: added subkey 0 of main key gpg-pubkey-37576165-5b22521c to keyring
D:  read h#     935 
Header SHA1 digest: OK
D: added key gpg-pubkey-442df0f8-608c8351 to keyring
D:  read h#    2031 
Header SHA1 digest: OK
D: added key gpg-pubkey-8b318951-6418c21b to keyring
D:  read h#    2032 
Header SHA1 digest: OK
D: added key gpg-pubkey-45f2c3d5-5e81efb9 to keyring
D: added subkey 0 of main key gpg-pubkey-45f2c3d5-5e81efb9 to keyring
D: Using legacy gpg-pubkey(s) from rpmdb
error: /tmp/k.asc: key 1 import failed.
D: closed   db index       /var/lib/rpm/Packages
D: closed   db index       /var/lib/rpm/Name
D: closed   db environment /var/lib/rpm

[MarkEWaite]

Will ed25519 work on the current platforms? Just tried to import some dummy ed25519 GPG key on Almalinux 8 (compatible with the current Red Hat Enteprise Linux 8):

I think this is a valid concern. CentOS 7 fails to import the key and reports that "key 1 is not an armored public key:

-----BEGIN PGP PUBLIC KEY BLOCK-----

mDMEZCGZwhYJKwYBBAHaRw8BAQdAj+SvDzMg4Tm8uXlPCd9jwlXcSLz+pi9m0lNI
drxWZsi0MkplbmtpbnMgUHJvamVjdCA8amVua2luc2NpLWJvYXJkQGdvb2dsZWdy
b3Vwcy5jb20+iJkEExYKAEEWIQQ3MIGS553OPjgzoyGU/YhtqYdhrAUCZCGZwgIb
AwUJBaOagAULCQgHAgIiAgYVCgkICwIEFgIDAQIeBwIXgAAKCRCU/YhtqYdhrCEU
AP4zrgb6fJ6pRWFlWDVkzZmSA4q92zl888KU/mWsoBEEhgD/ZKcwYndT59Evl/GC
JuTummn9Zo4YysPII0f+bjotzQu4OARkIZnCEgorBgEEAZdVAQUBAQdAjMg9dCFK
7jurd6h1im+P3TRYqC1KJQ41Sgt6t9/g7QQDAQgHiH4EGBYKACYWIQQ3MIGS553O
PjgzoyGU/YhtqYdhrAUCZCGZwgIbDAUJBaOagAAKCRCU/YhtqYdhrDm7AP0UOgZa
N+CnMa2FPO+iwL3C+A65yVb6qxwrnmO9plRoMwEA8yGE7z+iJ1//KTzUHdIkEIfg
m/FHxngbpEuDQ9H1cgQ=
=++Bw
-----END PGP PUBLIC KEY BLOCK----

Same message is reported on Red Hat Enterprise Linux 8 and Rocky Linux 9.

@dduportal I think that we need an RSA key instead of the ED25519 key.

[dduportal]

ack, thanks @MarkEWaite @saper , this is a valid concern!

Updated the key to rsa/4096:

• feat(weekly) add new 2023 GPG key release#352 (comment) (key ID in release properties)
• feat(pkgrepo) add the new GPG 2023 Jenkins public key jenkins-infra#2720 (comment) (public key + added the “key info” file)

[saper]

There seems to be some disagreement in the OpenPGP working group regarding packet format related to Ed25519 signatures (compare https://www.ietf.org/archive/id/draft-koch-openpgp-2015-rfc4880bis-01.txt vs https://www.ietf.org/archive/id/draft-ietf-openpgp-crypto-refresh-08.txt).

[MarkEWaite]

I placed the jenkins.io-2023.key files on pkg.jenkins.io manually with a little bit of shell script. Permissions on the files match the permissions on the jenkins.io.key files. I placed the jenkins.io-2023.key.info files in the same directories that already had jenkins.io.key.info files.

I realize that the automation will probably replace all those files. I think that would be great. I wanted the instructions in the blog post to be workable immediately, without waiting for the automation to run.

Discussion on the blog post will happen in the community.jenkins.io post.

[dduportal]

I placed the jenkins.io-2023.key files on pkg.jenkins.io manually with a little bit of shell script. Permissions on the files match the permissions on the jenkins.io.key files. I placed the jenkins.io-2023.key.info files in the same directories that already had jenkins.io.key.info files.

@MarkEWaite , the puppet agent on pkg.origin.jenkins.io VM keeps changing the file jenkins.io.key back to its old values (as specified in the puppet management).

Currently checking the weekly Debian and Redhat repositories contents

[MarkEWaite]

I think it is OK that jenkins.io.key is set to the previous value, isn't it?

The first interesting lines of the Debian key that I download from https://pkg.jenkins.io/debian/jenkins.io.key looks like this:

-----BEGIN PGP PUBLIC KEY BLOCK-----

mQINBF6B77kBEACZoUU41uYVDbagtNQrNQsnbx7UkRdu2rdUZLHryTOKv4InT33Z

The first interesting lines of the jenkins.io-2023.key file are different. They look like this:

-----BEGIN PGP PUBLIC KEY BLOCK-----

mQINBGQhzisBEAC7yUhIqVCcyCXJWeZZf/BA6/+KguDQpycck0xUomj5ogT1+lwJ

[dduportal]

@MarkEWaite puppet reports

(/Stage[main]/Profile::Pkgrepo/File[/var/www/pkg.jenkins.io/debian/jenkins.io.key]/content) content changed '{md5}294157f3149d77a4d6da166086faea4f' to '{md5}179ce6653d4a3bdfe318d71c6a83d8ea' (corrective)

on each run, while we got the following:

$ md5sum ./dist/profile/files/pkgrepo/*                  
179ce6653d4a3bdfe318d71c6a83d8ea  ./dist/profile/files/pkgrepo/jenkins-ci.org.key
294157f3149d77a4d6da166086faea4f  ./dist/profile/files/pkgrepo/jenkins.io-2023.key

It means that "something" keeps changing the content of the file jenkins-ci.org.key to the "new" GPG key content and I'm not what is doing that so I wondered if it was a crontab or a script you applied earlier this week?

If not, then I'll check the update center scripts

[dduportal]

Ooooh caught the culprit:

It's https://github.com/jenkins-infra/mirror-scripts/blob/49abc7c6272d67450ee4c4c1eb57f9c5dab6fb2a/sync.sh#L67-L69 which runs regularly:

$ /var/www/pkg.jenkins.io.staging# md5sum debian/*.key
294157f3149d77a4d6da166086faea4f  debian/jenkins.io-2023.key
294157f3149d77a4d6da166086faea4f  debian/jenkins.io.key

Gotta fix it tomorrow.

[dduportal]

• Fixed the keys on pkg.origin's staging directory:

$ md5sum /var/www/pkg.jenkins.io.staging/*/*.key
294157f3149d77a4d6da166086faea4f  /var/www/pkg.jenkins.io.staging/debian/jenkins.io-2023.key
179ce6653d4a3bdfe318d71c6a83d8ea  /var/www/pkg.jenkins.io.staging/debian/jenkins.io.key
294157f3149d77a4d6da166086faea4f  /var/www/pkg.jenkins.io.staging/debian-stable/jenkins.io-2023.key
179ce6653d4a3bdfe318d71c6a83d8ea  /var/www/pkg.jenkins.io.staging/debian-stable/jenkins.io.key
294157f3149d77a4d6da166086faea4f  /var/www/pkg.jenkins.io.staging/redhat/jenkins.io-2023.key
179ce6653d4a3bdfe318d71c6a83d8ea  /var/www/pkg.jenkins.io.staging/redhat/jenkins.io.key
294157f3149d77a4d6da166086faea4f  /var/www/pkg.jenkins.io.staging/redhat-stable/jenkins.io-2023.key
179ce6653d4a3bdfe318d71c6a83d8ea  /var/www/pkg.jenkins.io.staging/redhat-stable/jenkins.io.key

• New key jenkins.io-2023.key added in the Azure Blob storage binary-core-packages (used for core releases: https://github.com/jenkins-infra/release/blob/b4950a3453755533938bb725e4d0338fdb089549/PodTemplates.d/package-linux.yaml#L27)

[slide]

The debian-stable repo is not signed correctly, neither the old nor the new keys work

W: GPG error: https://pkg.jenkins.io/debian-stable binary/ Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY FCEF32E745F2C3D5
E: The repository 'https://pkg.jenkins.io/debian-stable binary/ Release' is not signed.

[NotMyFault]

There is no release of an LTS release (stable repository) yet, which is signed with the new key.

The first one is 2.387.2, released on the 5th of April.

[slide]

So, the stable repo is just going to not work until next Wednesday?

[NotMyFault]

I'd just skip the GPG check for now.

[ghost]

I am a little bit concerned about that answer since this will cause configuration management systems, that in our case has a formula for installing jenkins, to fail for a week. Similar we have tests for installs of Jenkins slave images that are now failing. Sure we can ignore the failing CM and tests, but we don't as we think that is bad practice.

[MarkEWaite]

I am a little bit concerned about that answer since this will cause configuration management systems, that in our case has a formula for installing jenkins, to fail for a week.

Yes, it is unfortunate that we didn't prepare far enough in advance so that there would not be a one week window where debian-stable did not have a package available with a valid PGP signature. The last time we made this transition, we planned more thoroughly and the outage was 24 hours or less. We'll host a retrospective to identify the mistakes we made and plan ways to prevent those mistakes for future key expiration.

Similar we have tests for installs of Jenkins slave images that are now failing. Sure we can ignore the failing CM and tests, but we don't as we think that is bad practice.

I'm curious how this would cause Jenkins agent images to fail. Can you explain further how you're configuring Jenkins agents in a way that was disrupted by the expiration of the GPG key that signs the Debian package of the Jenkins controller?

[dduportal]

• The LTS release 2.387.2 is out, signed with the new GPG key ✅ Tested on a fresh Debian (latest) container:

Click to see the output

$ docker run --rm -ti debian
Unable to find image 'debian:latest' locally
latest: Pulling from library/debian
3e440a704568: Pull complete 
Digest: sha256:7b991788987ad860810df60927e1adbaf8e156520177bd4db82409f81dd3b721
Status: Downloaded newer image for debian:latest

root@394d03e6a42b:/# apt-get install sudo curl gpg -y 
# ...

root@394d03e6a42b:/#   curl -fsSL https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key | sudo tee \
    /usr/share/keyrings/jenkins-keyring.asc > /dev/null

root@394d03e6a42b:/# echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] \
    https://pkg.jenkins.io/debian-stable binary/ | sudo tee \
    /etc/apt/sources.list.d/jenkins.list > /dev/null
# ...
# Installed 

• FIxed an issue in pkg.origin.jenkins.io (same as GPG key expires on March the 30th #3457 (comment) but for stable)

=> this issue is closable from infra point of view.

[dduportal]

• 2.387.2 is deployed with success on the Jenkins Infra Update ci.jenkins.io, trusted.ci, cert.ci and release.ci to latest LTS version 2.387.2 #3490
• The acceptance tests were updated for LTS Debian (https://github.com/jenkins-infra/acceptance-tests/tree/install-lts-debian-package) and LTS Redhat (https://github.com/jenkins-infra/acceptance-tests/tree/install-lts-redhat-rpm) with success
• The instructions at https://pkg.jenkins.io/redhat-stable/ and https://pkg.jenkins.io/debian-stable/ are up to date with the new key as expected

=> closing the issue. Feel free to reopen with a comment if you see any issue.