Rework RPM signing key handling

[Lalufu]

This is an attempt to address
openzfs/zfs#14344. It reworks the zfs-release RPM to

• Always include all signing keys that are in active use on any release
• Provide versioned symlinks for the supported releases, pointing to the appropriate key
• Modifies the .repo files to point to a versioned symlink by utilizing the $releasever variable provided by yum/dnf

This will allow smooth key transitions between different major releases. The same mechanism is used by other major repos, including Fedora itself.

Tested by:

• Take a F36 system without ZFS
• Run dnf install -y https://zfsonlinux.org/fedora/zfs-release-2-2$(rpm --eval "%{dist}").noarch.rpm according to the installation instructions on https://openzfs.github.io/openzfs-docs/Getting%20Started/Fedora/index.html#installation
• Install zfs packages
• Attempt to upgrade the system to F37 by issuing dnf system-upgrade --releasever=37 download
• Observe failure to validate F37 ZFS packages
• Install zfs-release file built from this MR
• Attempt to upgrade the system to F37 by issuing dnf system-upgrade --releasever=37 download again
• Observe the upgrade process asking to import new key:

ZFS on Linux for Fedora 37                                                                                                      3.3 MB/s | 3.4 kB     00:00
Importing GPG key 0x9DB84141:
 Userid     : "OpenZFS <release@openzfs.org>"
 Fingerprint: 7DC7 299D CF7C 7FD9 CD87 701B A599 FD5E 9DB8 4141
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-openzfs-fedora-37
Is this ok [y/N]: y
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Complete!

Also tested by installing zfs-release file built from this MR on a F36 system without ZFS, and observing the correct key being imported.

[tonyhutter]

I think this PR is commendable as it's trying to solve a real-world problem. Unfortunately the devil is in the details.

One drawback is that we would would need to update zfs-release.spec on every Fedora release, even though the key is the same. That requires a PR to be opened against the zfsonlinux.github.com repo on every Fedora release, which is an added maintenance burden. Considering the old SHA1 (160bit) key worked for us for 9 years, it's possible the new SHA512 (512bits) key is going to last us way longer. If that's the case, we'd be needlessly updating the spec file every 6 months for each Fedora release. I'm not convinced that it's worth it.

We would also need to make the ZFS RPMs dependent on the zfs-release RPM. Otherwise if you upgrade from F36 -> F37, it's possible dnf would try to install the ZFS RPMs first before the zfs-release RPM (with the updated key), giving you a key-signing error. To get around that we would need to update the zfs spec file to add a dependency on zfs-release, and then re-build the latest ZFS RPMs (which for argument's sake, we'll call zfs-2.1.7-2). Then, a user would would need to do the upgrade from Fedora 3X with zfs-2.1.7-2 to Fedora 37 for everything to work seamlessly. If you had an older installation like Fedora 34 that you wanted to upgrade to F37, or if you try to do an update from F36 -> F37 with the older <= zfs-2.1.7 RPMs, it would not work correctly. There may be additional gotchas with zfs requiring zfs-release that I'm not considering. For example, it would be slightly annoying for people building their own ZFS RPMs locally to have a dependency on zfs-release, which they wouldn't technically need.

[Lalufu]

Thanks for the reply, I'll see if I can address some of the concerns. For context, as we're talking about major version upgrades, in the following I will assume that I have a F36 system, and I want to upgrade to F37, and I'll use those designators to talk about the "current" and "future" system.

On Tue, Jan 03, 2023 at 01:59:28PM -0800, Tony Hutter wrote: One drawback is that we would would need to update `zfs-release.spec` on every Fedora release, even though the key is the same. That requires a PR to be opened against the zfsonlinux.github.com repo on every Fedora release, which is an added maintenance burden.

This is correct, and unfortunately, with this proposal, there is no real way around this. Given that Fedora updates twice a year with some commendable regularity this would require a change at least twice a year to the `zfs-release` file. If this is an unreasonable burden I cannot readily say. One could maybe automate this, but that sounds like enough work to make manual changes for the future 10 years viable.

We would also need to make the ZFS RPMs dependent on the zfs-release RPM. Otherwise if you upgrade from F36 -> F37, it's possible dnf would try to install the ZFS RPMs first before the zfs-release RPM (with the updated key), giving you a key-signing error.

There might be a misunderstanding here. A system-upgrade from F36 to F37 requires the F36 system to already have all signing keys required to validate the F37 packages before the upgrade starts. This is one of the reasons `dnf system-upgrade` warns you that you should have a fully updated F36 system before attempting to upgrade to F37, to make sure all the required keys are there. In other words, the contents of the F37 `zfs-release` file have no influence at all on the system-upgrade process. The install order of ZFS packages and the zfs-release file during the actual upgrade process to F37 does not matter, and neither would the ZFS packages have to depends on any particular version of zfs-release, or depend on zfs-release at all.

If you had an older installation like Fedora 34 that you wanted to upgrade to F37, or if you try to do an update from F36 -> F37 with the older <= zfs-2.1.7 RPMs, it would not work correctly.

It's funny you should mention this, as I did actually test this on a F34 system (I had an old VM sitting around that was convenient). I installed the "old" zfs-release.fc34.noarch file that is still available on zfsonlinux.org, installed ZFS, updated to the zfs-release file from this MR, and started a `dnf system-upgrade --releasever=37`. That correctly imported the ZFS F37 key, downloaded the ZFS F37 packages, validated them, and went into the upgrade process. (Side note: I do not believe F34 ever got the Fedora signing keys for F37, so this update is not viable in general, as one could not verify the base system packages. One would have to go through F36 first. Still, the general principle holds) One elephant in the room that this particular MR does not address is that the ZFS project, as of today, does not distribute the `zfs-releass` RPM in the ZFS repo. There is no automatic way for users of ZFS to get new keys. This is not strictly required for this MR, but I believe it would be convenient for the users of the project if this changed. With both this MR and the `zfs-release` RPM distributed thought the ZFS repo, the user experience over time would look like this: * User has a F36 system without ZFS * User installs `zfs-release` file following instructions on the web page. Let us assume this `zfs-release` file only contains keys for F36 and earlier * User installs zfs, maybe occasionally updating to a new ZFS version as it becomes available in the repo * The Fedora project branches off F37 (it's still a few months to an actual release) * The zfs project updates the `zfs-release` file with a key for F37 (this may or may not be the same key as used for F36) * User updates their system and gets the new `zfs-release` file. Nothing really changes for them * F37 gets released * The ZFS project releases F37 RPMs of ZFS * User does a `dnf system-upgrade`, dnf can validate the F37 ZFS RPMs with the keys from the `zfs-release` RPM that is already installed

[tonyhutter]

Ok, I see what your saying. So the user would be required to be fully updated on their current Fedora version before upgrading (which is the normal procedure anyway). That would get the new ZFS keys installed via the updated zfs-release for the next version before doing the upgrade. We'd probably only put the updated zfs-release RPM in the F36/F37 repos since F35 and older are EOL (and upgrading from Fedora two or more versions back becomes more risky anyway).

Maybe I can automate updating the spec file with a script or something, We already need to checkin the rebuilt zfs-releases RPMs with each new Fedora (like 13a7720), so we can roll the spec file changes in with those commits.

I don't think there's anything else you need to do with this PR for now. Let me try building the new RPMs and stick them in a test repo and see how the upgrades work.

[tonyhutter]

@Lalufu just to manage expectations - it's going to be a few months before I'll be able to get to creating/testing/signing these new packages. I also see that Fedora 38 is coming out Apr 18 2023, so let's me use that as a target date to get this out the door. Please leave this PR open in the meantime. I'll assign it to myself so it doesn't get lost in the shuffle.

[Lalufu]

On Thu, Jan 26, 2023 at 01:35:01PM -0800, Tony Hutter wrote: @Lalufu just to manage expectations - it's going to be a few months before I'll be able to get to creating/testing/signing these new packages. I also see that Fedora 38 is coming out Apr 18 2023, so let's me use that as a target date to get this out the door. Please leave this PR open in the meantime. I'll assign it to myself so it doesn't get lost in the shuffle.

No worries, this is fine.

[tonyhutter]

I think we can pull this in now. I built RPMs from this patch and installed them on centos[7-9] and fedora[36-38] without issue.

[tonyhutter]

@Lalufu would you mind adding a Signed-off-by line to the commit and I can pull it in?

[Lalufu]

On Wed, Apr 26, 2023 at 03:47:24PM -0700, Tony Hutter wrote: @Lalufu would you mind adding a `Signed-off-by` line to the commit and I can pull it in?

Sure, but I'm not quite sure how to do that? The PR seems to have been merged with `master`?

[tonyhutter]

It's ok, it's not merged with master yet.

To add a signed-off by line:

1. git commit --amend to modify your commit message.
2. Add your Signed-off-by: line to the bottom of the commit. For example:

Author: Tony Hutter <hutter2@llnl.gov>
Date:   Wed Apr 19 17:24:33 2023 -0700

    Add zfs-2.1.11 release
    
    Signed-off-by: Tony Hutter <hutter2@llnl.gov>

3. Force push it back (git push --force)

[Lalufu]

OK, I was confused, apperently the master branch in my fork already had the change.

Rebased, amended the commit.

[tonyhutter]

@Lalufu thanks, I just merged it