Packages can fail to build due to expired signing keys in old keyrings #87
I recently set up a rebuilderd instance running on archlinux core, and discovered 54 packages fail to build because one or more of the packages in their build environment fails the GPG signature check. All the packages failing validation have valid signatures from the perspective of the current archlinux-keyring package, but in the old archlinux-keyring packages used in the builds, the signing keys have expired.
Specifically, the following two keys:
Signed the following five packages:
Which are used along with pre-extension archlinux-keyring versions in the builds of the following packages:
These packages don't appear broken on reproducible.archlinux.org because this issue only affects packages that were built before the developer in question updated their archlinux-keyring package to a version that includes the relevant key extension(s), that were rebuilt after the expiry. Assuming developers update reasonably frequently, that reproducible.archlinux.org rebuilds packages shortly after they are published, and that it doesn't rebuild packages after it gets a successful repro, all of these packages would have been checked within the remaining validity of the signing keys.
A related issue is that we ignore any key revocations that weren't in effect at the time the package was published. If you're hoping to get a chain of trust stretching from newer packages back to older packages, then the compromise of a signing key for an old package is a big problem. There's a compromise path like:
Always using the newest archlinux-keyring package would solve both of these issues, though it's still not clear to me how to recover from a compromised developer signing key.
The text was updated successfully, but these errors were encountered:
The keyring problem is complicated. We can't use a recent keyring and downgrade to the old one as we disable keys that belongs to packagers that has quit the team. Downgrading the package does not remove these keys. That implies older packages with completely valid signatures at the time of build would fail to validate. This is why I implemented the keyring cache.
Now we have the second problem, key expiry in the older keyring which I did not for see when rewriting this initially. I don't know how to solve it. We can't blanket enable older keys, and we don't really want to just assign trust to all keys and ignore the issue. Not sure how to solve this in a good way.
Stealing the signing key is not something we care about in the context of reproducible builds though, it's a side-effect of rebuilding the package. In your scenario (if gcc is reproducible) it wouldn't matter if the key is stolen as the package is not reproducible. This should kick off the needed security measures in a perfect world.
According to https://www.gnupg.org/documentation/manuals/gnupg-2.0/GPG-Esoteric-Options.html, gpg has an option
The issue I see is that it's possible to add a package to the build environment without that package itself being reproducibly built, either by replacing an existing (but not-current) package in the archive, or by inserting a new one with an unused version number. Rebuilderd only concerns itself with up-to-date packages, and delegates setting up the build environment to repro, which in turn relies on the developer signatures on archive packages. Therefore we still need to worry about whether those signatures can be trusted, and whether the keys that created them have been compromised.
To fix this, you would ideally want to be able to say that no package can be used to build any other package until it has been verified, except for some set of pre-trusted packages (which might just be whatever existed in the archive when you started doing reproducible builds). That would let you stop worrying about signatures on build environment packages, and therefore key validity, entirely. This requires rebuilderd to know a lot more about package dependencies then it currently does though, which I suspect makes for a substantial re-architecting, and doesn't help with validating single packages in isolation (I assume that's an intended use case?).
We can't use it (probably) because
This isn't a problem we want to care about yet. An unused version number wouldn't be pulled as we are only pulling down packages defined in
I think you are three steps a head when thinking about this. The goal here is to ensure packages can be built reproducibly. Nothing else. Thinking about the trust issue or dependency issue is complicated and nothing that can be done yet because we can't have a complete chain of reproducible packages. I also think it's a misunderstanding about the current stage of rebuilderd+repro. This is experimental to find issues in tools, packages and infrastructure. We can't rely on this for any sort of security yet.
Old packages would previously fail to build if they depended on a package that was signed with an expired key, from the perspective of the keyring they were signed with. We can fix this by setting faked-system-time to control in gpg.conf to the package build time. This reveals a further issue that the keyring master key will be generated, and sign the archlinux master keys, at the time repro builds that keyring, which may be in the future relative to the build time of packages that depend on it. As gpg ignores signatures and keys which appear to be from the future, this will also break the build. We can fix this by also setting faked-system-time to the *keyring* package build time when setting up the keyring. Finally, to ease transition, we discard and rebuild any keyrings in the cache that weren't built in this way. This should allow recovery from this issue without manual intervention. Fixes archlinux#87