Packages can fail to build due to expired signing keys in old keyrings

[fergus-dall]

I recently set up a rebuilderd instance running on archlinux core, and discovered 54 packages fail to build because one or more of the packages in their build environment fails the GPG signature check. All the packages failing validation have valid signatures from the perspective of the current archlinux-keyring package, but in the old archlinux-keyring packages used in the builds, the signing keys have expired.

Specifically, the following two keys:
60411304C09D36628340EEFFCEB167EFB5722BD6 (eschwartz) expired on 2020-10-17, extended to 2022-07-10 in keyring version 20200817
E550FFFFD485E264A1717E30901C1C320EB0D45D (demize) expired on 2020-12-10, extended to 2022-02-03 in keyring version 20200422

Signed the following five packages:
asciidoc 9.0.2-1 (eschwartz)
autoconf-archive 1:2019.01.06-1 (eschwartz)
ncompress 4.2.4.6-1 (eschwartz)
pkgconf 1.6.3-2 (demize)
python2-typing 3.6.6-1 (demize)

Which are used along with pre-extension archlinux-keyring versions in the builds of the following packages:
automake 1.16.2-3
b43-fwcutter 019-3
base 2-2
ca-certificates 20181109-4
ca-certificates-utils 20181109-4
cracklib 2.9.7-2
crda 4.14-3
cronie 1.5.5-1
db 5.3.28-5
dbus 1.12.20-1
dbus-docs 1.12.20-1
diffutils 3.7-3
ding-libs 0.6.1-3
dmraid 1.0.0.rc16.3-12
dosfstools 4.1-3
findutils 4.7.0-2
flex 2.6.4-3
gdbm 1.18.1-3
gzip 1.10-3
hdparm 9.58-3
ifenslave 1.1.0-11
inetutils 1.9.4-8
ipw2100-fw 1.3-10
ipw2200-fw 3.1-8
jfsutils 1.1.15-7
ldns 1.7.1-2
libaio 0.3.112-2
libgssglue 0.4-4
libmnl 1.0.4-3
libmpc 1.1.0-2
libnl 3.5.0-2
libssh2 1.9.0-2
lzo 2.10-3
m4 1.4.18-3
minizip 1:1.2.11-4
mkinitcpio-nfs-utils 0.3-7
net-tools 1.60.20181103git-2
nilfs-utils 2.2.8-2
npth 1.6-2
patch 2.7.6-8
pinentry 1.1.0-5
pptpclient 1.10.0-2
procinfo-ng 2.0.304-8
psmisc 23.3-2
pth 2.0.7-7
reiserfsprogs 3.6.27-3
rpcbind 1.2.5-3
run-parts 4.8.6.1-2
sysfsutils 2.1.0-11
tar 1.32-3
vi 1:070224-4
which 2.21-5
wireless_tools 30.pre9-3
zlib 1:1.2.11-4

These packages don't appear broken on reproducible.archlinux.org because this issue only affects packages that were built before the developer in question updated their archlinux-keyring package to a version that includes the relevant key extension(s), that were rebuilt after the expiry. Assuming developers update reasonably frequently, that reproducible.archlinux.org rebuilds packages shortly after they are published, and that it doesn't rebuild packages after it gets a successful repro, all of these packages would have been checked within the remaining validity of the signing keys.

A related issue is that we ignore any key revocations that weren't in effect at the time the package was published. If you're hoping to get a chain of trust stretching from newer packages back to older packages, then the compromise of a signing key for an old package is a big problem. There's a compromise path like:

1. Steal developer signing key
2. Sign a compromised version of gcc at some old version which introduces a flaw into some other package
3. Push compromised gcc to package archive
4. Build compromised package using compromised gcc
5. Publish compromised package
6. The bad gcc is never checked, because it looks like an out-of-date package, and the new package reproduces because reproducers will use the bad gcc package, even if the signing key is later revoked

Always using the newest archlinux-keyring package would solve both of these issues, though it's still not clear to me how to recover from a compromised developer signing key.

[Foxboron]

The keyring problem is complicated. We can't use a recent keyring and downgrade to the old one as we disable keys that belongs to packagers that has quit the team. Downgrading the package does not remove these keys. That implies older packages with completely valid signatures at the time of build would fail to validate. This is why I implemented the keyring cache.

Now we have the second problem, key expiry in the older keyring which I did not for see when rewriting this initially. I don't know how to solve it. We can't blanket enable older keys, and we don't really want to just assign trust to all keys and ignore the issue. Not sure how to solve this in a good way.

A related issue is that we ignore any key revocations that weren't in effect at the time the package was published. If you're hoping to get a chain of trust stretching from newer packages back to older packages, then the compromise of a signing key for an old package is a big problem. There's a compromise path like:

Stealing the signing key is not something we care about in the context of reproducible builds though, it's a side-effect of rebuilding the package. In your scenario (if gcc is reproducible) it wouldn't matter if the key is stolen as the package is not reproducible. This should kick off the needed security measures in a perfect world.

[Foxboron]

There was an attempt at utilizing libfaketime for the pacstrap portion, but it seems like the LD_PRELOAD stuff gets removed before pacman hit's the gpgme portion of the code.

[fergus-dall]

Not sure how to solve this in a good way.

According to https://www.gnupg.org/documentation/manuals/gnupg-2.0/GPG-Esoteric-Options.html, gpg has an option --faked-system-time to pretend the system time is something other then its real value. It claims to be "only useful for testing" though, so I'm not sure we should expect to be able to rely on it.

In your scenario (if gcc is reproducible) it wouldn't matter if the key is stolen as the package is not reproducible. This should kick off the needed security measures in a perfect world.

The issue I see is that it's possible to add a package to the build environment without that package itself being reproducibly built, either by replacing an existing (but not-current) package in the archive, or by inserting a new one with an unused version number. Rebuilderd only concerns itself with up-to-date packages, and delegates setting up the build environment to repro, which in turn relies on the developer signatures on archive packages. Therefore we still need to worry about whether those signatures can be trusted, and whether the keys that created them have been compromised.

To fix this, you would ideally want to be able to say that no package can be used to build any other package until it has been verified, except for some set of pre-trusted packages (which might just be whatever existed in the archive when you started doing reproducible builds). That would let you stop worrying about signatures on build environment packages, and therefore key validity, entirely. This requires rebuilderd to know a lot more about package dependencies then it currently does though, which I suspect makes for a substantial re-architecting, and doesn't help with validating single packages in isolation (I assume that's an intended use case?).

[Foxboron]

According to https://www.gnupg.org/documentation/manuals/gnupg-2.0/GPG-Esoteric-Options.html, gpg has an option --faked-system-time to pretend the system time is something other then its real value. It claims to be "only useful for testing" though, so I'm not sure we should expect to be able to rely on it.

We can't use it (probably) because pacman uses gpgme and I don't know how this is exposed.

The issue I see is that it's possible to add a package to the build environment without that package itself being reproducibly built, either by replacing an existing (but not-current) package in the archive, or by inserting a new one with an unused version number.

This isn't a problem we want to care about yet. An unused version number wouldn't be pulled as we are only pulling down packages defined in .BUILDINFO. So our goal is to only reproduce the package given the .BUILDINFO. Enforcing reproducible packages upwards in the chain isn't solveable at this stage, and it might come when we are further along the way.

This requires rebuilderd to know a lot more about package dependencies then it currently does though, which I suspect makes for a substantial re-architecting, and doesn't help with validating single packages in isolation (I assume that's an intended use case?).

I think you are three steps a head when thinking about this. The goal here is to ensure packages can be built reproducibly. Nothing else. Thinking about the trust issue or dependency issue is complicated and nothing that can be done yet because we can't have a complete chain of reproducible packages. I also think it's a misunderstanding about the current stage of rebuilderd+repro. This is experimental to find issues in tools, packages and infrastructure. We can't rely on this for any sort of security yet.

[SantiagoTorres]

It is possible to signal the --faked-system-time flag by doing something akin to: echo "faked-system-time $SOURCE_DATE_EPOCH" >> /etc/pacman.d/gnupg/gpg.conf. As gpgme is just popen-ing into gpg we should be good to go

[Foxboron]

@SantiagoTorres Brilliant! That didn't occur to me!