rpm --import does not replace old keys with new keys

[andrewclausen]

Encountering the problem with Google Linux Signing keys

When I try to install new Google Linux Signing keys in Fedora 37 (with the latest stable release of rpm, namely 4.18.1), it fails. However, when I delete the old (pre-installed) keys, and try again, it works. Here is what I typed:

wget https://dl.google.com/linux/linux_signing_key.pub
sudo rpm --import linux_signing_key.pub
rpm -qi gpg-pubkey-d38b4796-* | gpg --show-keys --with-subkey-fingerprint
rpm -qa gpg-pubkey* --qf '%{NAME}-%{VERSION}-%{RELEASE} %{PACKAGER}\n' | grep 'linux-packages-keymaster@google.com' | sed 's/ .*$//' | sudo xargs rpm -e
sudo rpm --import linux_signing_key.pub
rpm -qi gpg-pubkey-d38b4796-* | gpg --show-keys --with-subkey-fingerprint

Here is the output. Notice that the first attempt at importing the keys fails silently, whereas the second attempt (after deleting old keys) succeeds.

[user@disp8080 ~]$ wget https://dl.google.com/linux/linux_signing_key.pub
--2023-07-21 23:26:12--  https://dl.google.com/linux/linux_signing_key.pub
Resolving dl.google.com (dl.google.com)... 142.250.178.14, 2a00:1450:4009:81f::200e
Connecting to dl.google.com (dl.google.com)|142.250.178.14|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 14605 (14K) [application/octet-stream]
Saving to: ‘linux_signing_key.pub’

linux_signing_key.p 100%[===================>]  14.26K  --.-KB/s    in 0s      

2023-07-21 23:26:14 (43.0 MB/s) - ‘linux_signing_key.pub’ saved [14605/14605]

[user@disp8080 ~]$ sudo rpm --import linux_signing_key.pub
[user@disp8080 ~]$ rpm -qi gpg-pubkey-d38b4796-* | gpg --show-keys --with-subkey-fingerprint
gpg: directory '/home/user/.gnupg' created
gpg: keybox '/home/user/.gnupg/pubring.kbx' created
pub   rsa4096 2016-04-12 [SC]
      EB4C1BFD4F042F6DDDCCEC917721F63BD38B4796
uid                      Google Inc. (Linux Packages Signing Authority) <linux-packages-keymaster@google.com>
sub   rsa4096 2016-04-12 [S] [expired: 2019-04-12]
      3B068FB4789ABE4AEFA3BB491397BC53640DB551
sub   rsa4096 2017-01-24 [S] [expired: 2020-01-24]
      3E50F6D3EC278FDEB655C8CA6494C6D6997C215E

[user@disp8080 ~]$ rpm -qa gpg-pubkey* --qf '%{NAME}-%{VERSION}-%{RELEASE} %{PACKAGER}\n' | grep 'linux-packages-keymaster@google.com' | sed 's/ .*$//' | sudo xargs rpm -e
[user@disp8080 ~]$ sudo rpm --import linux_signing_key.pub
[user@disp8080 ~]$ rpm -qi gpg-pubkey-d38b4796-* | gpg --show-keys --with-subkey-fingerprint
pub   rsa4096 2016-04-12 [SC]
      EB4C1BFD4F042F6DDDCCEC917721F63BD38B4796
uid                      Google Inc. (Linux Packages Signing Authority) <linux-packages-keymaster@google.com>
sub   rsa4096 2016-04-12 [S] [expired: 2019-04-12]
      3B068FB4789ABE4AEFA3BB491397BC53640DB551
sub   rsa4096 2017-01-24 [S] [expired: 2020-01-24]
      3E50F6D3EC278FDEB655C8CA6494C6D6997C215E
sub   rsa4096 2019-07-22 [S] [expired: 2022-07-21]
      2F528D36D67B69EDF998D85778BD65473CB3BD13
sub   rsa4096 2021-10-26 [S] [expires: 2024-10-25]
      8461EFA0E74ABAE010DE66994EB27DB2A3B88B8B
sub   rsa4096 2023-02-15 [S] [expires: 2026-02-14]
      A5F483CD733A4EBAEA378B2AE88979FB9B30ACF2

The root of the problem

Google's Linux Signing keys have an unusual configuration: they retain the same master key, but replace the subkeys which do the actual signing. Since rpm identifies the whole collection of keys by the master key, the new key looks like the pre-existing key. So it ignores it.

Work around

The work around -- as alluded to above -- is to delete the old keys before trying to import the new ones. This can be done with:

rpm -qa gpg-pubkey* --qf '%{NAME}-%{VERSION}-%{RELEASE} %{PACKAGER}\n' | grep 'linux-packages-keymaster@google.com' | sed 's/ .*$//' | sudo xargs rpm -e

Discussion

This bug is potentially quite serious. Millions of Linux users use Google Chrome, and right now, there is no way for them to install or upgrade it. They will just see that signature verification fails. Most people will give up, or worse, disable signature checks. Those that try to check the signature manually will be thwarted by the same bug again.

This means there is a serious usability impact (not being able to install Google Chrome), as well as a serious security impact (not getting security updates, or installing unverified updates for Google Chrome).

Solution

Please either fix rpm so that it correctly imports keys (e.g. by using a timestamp as the key version), or deprecate the use of subkeys.

[andrewclausen]

I should mention that I am not the first person to encounter this problem -- I imagine many people are affected. See for example this Reddit thread: https://www.reddit.com/r/chrome/comments/13s799o/googlechromebeta_1140573545_rpm_invalid_signature/

[nwalfield]

Looking at the code, it seems:

• rpmcliImportPubkeys
• calls doImport
• calls rpmtsImportPubkey
• calls rpmKeyringAddKey

which checks if the certificate's key id is known, and if so, don't do anything.

OpenPGP certificates are made up of packets. It is possible to add new packets without updating other packets (e.g., a new subkey, a new self signature that extends the certificate's expiration time). So it is not easy to generate a linear version number for an OpenPGP certificate.

I think we need to extend rpm's pgp interface to add a function to merge two versions of a certificate.

[andrewclausen]

@nwalfield, merging certificates sounds like a relatively hard problem to solve in general. In this situation, if the user types

sudo rpm --import linux_signing_key.pub

then they probably just wants to replace the pre-existing key/certificate with the supplied one.

Can you think of any use cases where this would cause a problem?

[nwalfield]

@nwalfield, merging certificates sounds like a relatively hard problem to solve in general.

Can you explain what you are thinking or worried about here? The implementation to merge certificates in Sequoia starts here. We basically turn the two certificates into arrays of packets, and merge the two arrays. Then we canonicalize the result, which reorders and dedups the packets. I admit it is a few lines of code, but I think it is a stretch to say that it is a hard problem.

Can you think of any use cases where this would cause a problem?

Yes. If the new certificate is missing some components that the existing version has, signatures that could once be verified may no longer be verifiable.

[andrewclausen]

Can you explain what you are thinking or worried about here?

I'm not especially worried. It just sounds like a lot of work! But it sounds like you have already done the work, so there is no problem.

Can you think of any use cases where this would cause a problem?

Yes. If the new certificate is missing some components that the existing version has, signatures that could once be verified may no longer be verifiable.

I suppose there might be some situations where that is helpful. But it sounds dangerous as a default behaviour. Old keys would never get retired / revoked. This could leave users vulnerable to attack. Otherwise, what's the point of rotating keys?

[nwalfield]

I suppose there might be some situations where that is helpful. But it sounds dangerous as a default behaviour. Old keys would never get retired / revoked. This could leave users vulnerable to attack. Otherwise, what's the point of rotating keys?

I see. If I understand you correctly, you're suggesting that the old component would be dropped. Are you suggesting that people resign their old messages when they retire an signing subkey (that sounds like a lot of work)? If not, how would a third party verify an old signature?

[andrewclausen]

I suppose there might be some situations where that is helpful. But it sounds dangerous as a default behaviour. Old keys would never get retired / revoked. This could leave users vulnerable to attack. Otherwise, what's the point of rotating keys?

I see. If I understand you correctly, you're suggesting that the old component would be dropped. Are you suggesting that people resign their old messages when they retire an signing subkey (that sounds like a lot of work)? If not, how would a third party verify an old signature?

Yes, I think the old component should be dropped by default.

No, I don't think old packages should be re-signed by the new key. Rather, I think that the key master ought to keep old keys inside the certificate unless they want to revoke it (e.g. due to an attack). The old keys would still have expiry dates, so RPM would complain about old signatures -- but this can be overriden.

To reiterate, I think the common scenarios would be one of:

1. The key master keeps a long history of keys (as subkeys) inside the certificate. This is what Google does.

2. The key master publishes keys with separate certificates, so they can be installed simultaneously. I think this is the more common practice.

[nwalfield]

Is a "key master" the entity that controls the key?

I think we are using the word certificate in different ways. According to RFC 4880, a certificate in OpenPGP (as a first approximation) is a Public Key packet, zero or more direct key signatures, zero or more User ID packets with any accompanying self signatures, and zero or more Subkey packets with any accompanying self signatures. Is that how you are using the word certificate? Can you explain what it means for "a key master" to "publish keys with separate certificates"?

[andrewclausen]

Is a "key master" the entity that controls the key?

I think we are using the word certificate in different ways. According to RFC 4880, a certificate in OpenPGP (as a first approximation) is a Public Key packet, zero or more direct key signatures, zero or more User ID packets with any accompanying self signatures, and zero or more Subkey packets with any accompanying self signatures. Is that how you are using the word certificate?

You are right, I am not on top of OpenPGP terminology, and I suspect much of the documentation floating around (including GPG's own manual) is not entirely aligned with it. Thank you for being patient with me!

By key master, I mean the person (or team of people) who manage the signing keys at Google.

I think I was using the word certificate inconsistently, but mostly in the same way you are. I think it's helpful to use examples to make 100% sure we are talking about the same thing. Examples of certificates include:

• Each item listed in the output of rpm -qa gpg-pubkey* contains a single PGP public key block. As far as I know, each of these contains a single certificate. For example, one of these on my system is gpg-pubkey-38ab71f4-60242b08. The contents -- including subkeys -- can be viewed by typing:

  $ rpm -qi gpg-pubkey-38ab71f4-60242b08 | gpg --show-keys --with-subkey-fingerprint
  pub   rsa4096 2021-02-10 [SCE]
        53DED2CB922D8B8D9E63FD18999F7CBF38AB71F4
  uid                      Fedora (36) <fedora-36-primary@fedoraproject.org>
  

• Another example is the Google's Linux signing key, which is actually a collection of public keys bundled inside a single certificate. For the record, its contents are:

  $ cat linux_signing_key.pub | gpg --show-keys --with-subkey-fingerprint
  pub   dsa1024 2007-03-08 [SC]
        4CCA1EAF950CEE4AB83976DCA040830F7FAC5991
  uid                      Google, Inc. Linux Package Signing Key <linux-packages-keymaster@google.com>
  sub   elg2048 2007-03-08 [E]
        9534C9C4130B4DC9927992BF4F30B6B4C07CB649
  
  pub   rsa4096 2016-04-12 [SC]
        EB4C1BFD4F042F6DDDCCEC917721F63BD38B4796
  uid                      Google Inc. (Linux Packages Signing Authority) <linux-packages-keymaster@google.com>
  sub   rsa4096 2016-04-12 [S] [expired: 2019-04-12]
        3B068FB4789ABE4AEFA3BB491397BC53640DB551
  sub   rsa4096 2017-01-24 [S] [expired: 2020-01-24]
        3E50F6D3EC278FDEB655C8CA6494C6D6997C215E
  sub   rsa4096 2019-07-22 [S] [expired: 2022-07-21]
        2F528D36D67B69EDF998D85778BD65473CB3BD13
  sub   rsa4096 2021-10-26 [S] [expires: 2024-10-25]
        8461EFA0E74ABAE010DE66994EB27DB2A3B88B8B
  sub   rsa4096 2023-02-15 [S] [expires: 2026-02-14]
        A5F483CD733A4EBAEA378B2AE88979FB9B30ACF2
  

Can you explain what it means for "a key master" to "publish keys with separate certificates"?

My first example above -- the Fedora 36 signing key -- is distributed in its own certificate. Fedora 37 has a separate key and a separate certificate. This separation means that the user can install any combination of them at their leisure. If for some reason they want to check old signatures, they can do so by installing old certificates containing old keys. Since Fedora publishes keys in separate certificates, there is no need to merge Fedora's certificates.

On the other hand, Google publishes many keys within a single certificate (the second example). If a new version of the certificate removes some old keys, this would prevent the user from verifying old signatures. For example, the key that was issued on 2016-04-12 (and expired in 2019) might get removed from future versions of this certificate. If this happens, then the user would have no obvious way of verifying packages signed by this key. Your proposal of merging the new certificate with previously installed ones is one way of addressing this problem. But I think it comes with a serious downside that the user has no way removing revoked keys. If the 2016-04-12 key gets compromised, your proposal might leave the user vulnerable to attacker-signed packages. (The fact that they key has expired might help, but it's not the end of the story; e.g. what if a more recent key gets compromised?)

Sorry for the long letter. But hopefully my writing (and my thinking!) is a bit clearer now.

[nwalfield]

Is a "key master" the entity that controls the key?
I think I was using the word certificate inconsistently, but mostly in the same way you are. I think it's helpful to use examples to make 100% sure we are talking about the same thing. Examples of certificates include:

* Each item listed in the output of `rpm -qa gpg-pubkey*` contains a single PGP public key block.  As far as I know, each of these contains a single certificate.  For example, one of these on my system is `gpg-pubkey-38ab71f4-60242b08`.  The contents -- including subkeys -- can be viewed by typing:
  ```
  $ rpm -qi gpg-pubkey-38ab71f4-60242b08 | gpg --show-keys --with-subkey-fingerprint
  pub   rsa4096 2021-02-10 [SCE]
        53DED2CB922D8B8D9E63FD18999F7CBF38AB71F4
  uid                      Fedora (36) <fedora-36-primary@fedoraproject.org>
  ```

Yes.

* Another example is the [Google's Linux signing key](https://dl.google.com/linux/linux_signing_key.pub), which is actually a collection of public keys bundled inside a single certificate.  For the record, its contents are:
  ```
  $ cat linux_signing_key.pub | gpg --show-keys --with-subkey-fingerprint
  pub   dsa1024 2007-03-08 [SC]
        4CCA1EAF950CEE4AB83976DCA040830F7FAC5991
  uid                      Google, Inc. Linux Package Signing Key <linux-packages-keymaster@google.com>
  sub   elg2048 2007-03-08 [E]
        9534C9C4130B4DC9927992BF4F30B6B4C07CB649
  
  pub   rsa4096 2016-04-12 [SC]
        EB4C1BFD4F042F6DDDCCEC917721F63BD38B4796
  uid                      Google Inc. (Linux Packages Signing Authority) <linux-packages-keymaster@google.com>
  sub   rsa4096 2016-04-12 [S] [expired: 2019-04-12]
        3B068FB4789ABE4AEFA3BB491397BC53640DB551
  sub   rsa4096 2017-01-24 [S] [expired: 2020-01-24]
        3E50F6D3EC278FDEB655C8CA6494C6D6997C215E
  sub   rsa4096 2019-07-22 [S] [expired: 2022-07-21]
        2F528D36D67B69EDF998D85778BD65473CB3BD13
  sub   rsa4096 2021-10-26 [S] [expires: 2024-10-25]
        8461EFA0E74ABAE010DE66994EB27DB2A3B88B8B
  sub   rsa4096 2023-02-15 [S] [expires: 2026-02-14]
        A5F483CD733A4EBAEA378B2AE88979FB9B30ACF2
  ```

This is not how the term certificate is commonly used in the OpenPGP ecosystem. This file includes two certificates: 4CCA1EAF950CEE4AB83976DCA040830F7FAC5991 and EB4C1BFD4F042F6DDDCCEC917721F63BD38B4796.

Can you explain what it means for "a key master" to "publish keys with separate certificates"?

My first example above -- the Fedora 36 signing key -- is distributed in its own certificate. Fedora 37 has a separate key and a separate certificate. This separation means that the user can install any combination of them at their leisure. If for some reason they want to check old signatures, they can do so by installing old certificates containing old keys. Since Fedora publishes keys in separate certificates, there is no need to merge Fedora's certificates.

You've unfortunately lost me again. In OpenPGP, a key is not separate from a certificate. A key is a component.

On the other hand, Google publishes many keys within a single certificate (the second example). If a new version of the certificate removes some old keys, this would prevent the user from verifying old signatures. For example, the key that was issued on 2016-04-12 (and expired in 2019) might get removed from future versions of this certificate. If this happens, then the user would have no obvious way of verifying packages signed by this key. Your proposal of merging the new certificate with previously installed ones is one way of addressing this problem. But I think it comes with a serious downside that the user has no way removing revoked keys. If the 2016-04-12 key gets compromised, your proposal might leave the user vulnerable to attacker-signed packages. (The fact that they key has expired might help, but it's not the end of the story; e.g. what if a more recent key gets compromised?)

I don't understand why a user would want to remove a revoked key. If it is revoked, the user should just import the revocation certificate and then it can't be used to verify packages any more.

[andrewclausen]

* Another example is the [Google's Linux signing key](https://dl.google.com/linux/linux_signing_key.pub), which is actually a collection of public keys bundled inside a single certificate.  For the record, its contents are:

$ cat linux_signing_key.pub | gpg --show-keys --with-subkey-fingerprint
pub dsa1024 2007-03-08 [SC]
4CCA1EAF950CEE4AB83976DCA040830F7FAC5991
uid Google, Inc. Linux Package Signing Key linux-packages-keymaster@google.com
sub elg2048 2007-03-08 [E]
9534C9C4130B4DC9927992BF4F30B6B4C07CB649

pub rsa4096 2016-04-12 [SC]
EB4C1BFD4F042F6DDDCCEC917721F63BD38B4796
uid Google Inc. (Linux Packages Signing Authority) linux-packages-keymaster@google.com
sub rsa4096 2016-04-12 [S] [expired: 2019-04-12]
3B068FB4789ABE4AEFA3BB491397BC53640DB551
sub rsa4096 2017-01-24 [S] [expired: 2020-01-24]
3E50F6D3EC278FDEB655C8CA6494C6D6997C215E
sub rsa4096 2019-07-22 [S] [expired: 2022-07-21]
2F528D36D67B69EDF998D85778BD65473CB3BD13
sub rsa4096 2021-10-26 [S] [expires: 2024-10-25]
8461EFA0E74ABAE010DE66994EB27DB2A3B88B8B
sub rsa4096 2023-02-15 [S] [expires: 2026-02-14]
A5F483CD733A4EBAEA378B2AE88979FB9B30ACF2

This is not how the term certificate is commonly used in the OpenPGP ecosystem. This file includes two certificates: 4CCA1EAF950CEE4AB83976DCA040830F7FAC5991 and EB4C1BFD4F042F6DDDCCEC917721F63BD38B4796.

Thank you for your patience!

They are bundled together into a single file, and into a single (virtual?) RPM package gpg-pubkey-38ab71f4-60242b08. Is there standard terminology for this situation?

Can you explain what it means for "a key master" to "publish keys with separate certificates"?

My first example above -- the Fedora 36 signing key -- is distributed in its own certificate. Fedora 37 has a separate key and a separate certificate. This separation means that the user can install any combination of them at their leisure. If for some reason they want to check old signatures, they can do so by installing old certificates containing old keys. Since Fedora publishes keys in separate certificates, there is no need to merge Fedora's certificates.

You've unfortunately lost me again. In OpenPGP, a key is not separate from a certificate. A key is a component.

What I mean is that the Fedora 36 signing key can be installed separately from the Fedora 37 signing key. They are in separate virtual RPM packages, and they have separate keys and certificates. This contrasts with the Google situation where it's different versions of the same certificate.

On the other hand, Google publishes many keys within a single certificate (the second example). If a new version of the certificate removes some old keys, this would prevent the user from verifying old signatures. For example, the key that was issued on 2016-04-12 (and expired in 2019) might get removed from future versions of this certificate. If this happens, then the user would have no obvious way of verifying packages signed by this key. Your proposal of merging the new certificate with previously installed ones is one way of addressing this problem. But I think it comes with a serious downside that the user has no way removing revoked keys. If the 2016-04-12 key gets compromised, your proposal might leave the user vulnerable to attacker-signed packages. (The fact that they key has expired might help, but it's not the end of the story; e.g. what if a more recent key gets compromised?)

I don't understand why a user would want to remove a revoked key. If it is revoked, the user should just import the revocation certificate and then it can't be used to verify packages any more.

Good point. I guess I was worried that the key master might not distribute a revocation certificate, or that DNF / RPM might not acquire and process the revocation certificate correctly. Has this been tested?

[DemiMarie]

@andrewclausen The Sequoia backend should process revocation certificates correctly. The internal backend does not, which is one of multiple reasons it is deprecated.

[andrewclausen]

@andrewclausen The Sequoia backend should process revocation certificates correctly. The internal backend does not, which is one of multiple reasons it is deprecated.

It's not just a question of the backend. For example, how would the revocation certificates be distributed?

My main point here is: security is hard, and users (like me) definitely appreciate seeing that everything has been thought through.

If so, I think @nwalfield's proposal should work well.

[heyakyra]

This blocks Google Chrome updates on Fedora Silverblue entirely:

fedora-silverblue/issue-tracker#408 (comment)

[pmatilai]

I agree --import needs some sort of "update" operation.

But any solution that may be developed in the future is not going to help the now of Chrome users as there's no way to get it deployed to all the (enterprise) distros out there. Google needs to use keys that are compatible with how existing rpm works, no matter how inconventient or braindead it may be at this time (it is). In practise this means using a new (top-level) key so the gpg-pubkey "version" changes.