Provide a method for cryptographic verification of source code

[hlieberman]

With the demise of PGP signatures on PyPi, there is no longer an easy way to verify the upstream source prior to pulling it into Debian.

I'd like to figure out some solution to deal with this, even if it's some hacky method of having someone send me a signed email with a hash of the code.

[osirisinferi]

Wouldn't it be possible to add the .asc file(s) to the Github release assets? Can multiple signatures be aggregated into a single file? You'd not want to clutter the release assets too much of course.

[bmw]

Harlan, how do some of the packages for our dependencies like https://salsa.debian.org/python-team/packages/python-cryptography handle this? Do they use the signed tags from GitHub and if so would you be able to do the same for us?

If so, assuming that'd require you switch to pulling the source code from GitHub instead of PyPI, josepy should be easy, but this repo would require essentially downloading all the files in this monorepo and then cding to the directory with the project name and excluding everything else. After that I think it'd more or less work the same from your perspective though.

Is that feasible and how hard is it to do? If that's the route we take, we may be able to help you with the transition.

[hlieberman]

python-cryptography also used to use GPG signatures; right now, I believe Sandro simply chose to do nothing and trust in TLS to secure the connection. That's certainly an option, especially if there was a semi-out-of-band method for me to check hashes or something.

We could also look at switching to github releases. I've been hesitant to do that for a while now, mostly because of the absolute pain in the ass it was to try and get it to work when I last tried. Admittedly though, that was back in 2015, and python packaging has come a ways since then. I could try playing around with it again, if that's the direction you'd like to see it move in. It would mean re-creating a 1:1 link between the versions of all the different plugins again, which I know is something else that y'all have wanted.

Let me try and mess around a bit with the packaging and see how much suffering it would be to get pybuild to support building the monorepo.

[jaskij]

There were some issues with the stability of source archives in the past, so if you want to use GitHub releases, it would probably be better to generate a source archive as an artifact from CI. Reading further, you can also download a tarball of a specific commit.

[adferrand]

Indeed we could leverage the CI capacities and Github releases. My preference would go to improve the release pipeline to create a signed Github release, publish the sdists of each package as assets of this release + a txt file containing the checksums of each sdist.

(NB: the signed windows installer is already published with this approach)

[bmw]

I certainly haven't forgotten about this, but I've been busy with other stuff. I'll write a much more thoughtful response soon but I wanted to give a quick update here to let you all know that this was still on my radar.

[bmw]

After thinking about this a bit, some options I've considered are:

1. We just drop PGP signatures entirely. I hate this, but it seems like this is the approach taken by almost every other Linux distro and almost all of Certbot's Python dependencies in Debian. If the stack we're built on doesn't have this extra security, does us having it really buy us much and is it worth the effort?
2. Debian (and any other distribution mechanism) that wants to use our PGP signatures switches to pulling from source. This is trickier, but would allow people to just use our existing signature.
3. The approach Adrien described in Provide a method for cryptographic verification of source code #9740 (comment).
4. We add something like SHA256SUMS and SHA256SUMS.gpg files to our GitHub releases. This would contain the hashes of all of our packages pushed to PyPI. We could upload the packages to the GitHub release in addition to PyPI as well, but I think I'd rather skip this unless it's a big benefit to someone.

Harlan, I'd love to hear which of these options you prefer or if you have different ideas once you get a chance to play around with things and give it a think. I don't really have a good sense for what Debian policies/tooling makes easy or difficult here.

[hlieberman]

Hi Brad, In order of least work to most, it's probably 3, 4, 1, 2. There's some tooling already for signed SHA256sums files that I could leverage, as well as tools for signed sdist's. I definitely acknowledge that the juice may not be worth the squeeze here. Theoretically, Debian developers are supposed to be reviewing the changes on package updates manually, which could theoretically make it more difficult for someone to slip malware in, but... that's a very shallow form of security. For me, the biggest advantage comes from the users having to trust _me_ less. Between reproducible builds, source-only uploads, and the signature chain, the only way for me to influence certbot would be changes to the debian packaging itself -- which is far, far harder to hide something in than the source code. Certainly not impossible, but more difficult. Which of these would be the least suffering for y'all? Sincerely, -- Harlan Lieberman-Berg ~hlieberman

[bmw]

What's easier for you about 3 than 4? Is it the inclusion of the sdists in the GitHub release?

On reread, I'm also not exactly sure what Adrien had in mind in 3 about signing. Would we continue to sign each sdist, the checksum file, and/or something else entirely? I'm not aware of a way to sign a GitHub release in its entirety including additional assets added after the fact like the sdists. What signature(s) would work best for you Harlan?

1 is the least suffering technically, but it's probably the most painful ethically, socially, spiritually, etc. 🙃 I imagine we can come up with something like 3 or 4 that everyone is more or less happy with.

[adferrand]

To clarify my 3, what I have in mind is something like this: https://github.com/traefik/traefik/releases

So a signed GitHub release, with an additional asset taking the form a txt file containing the checksum of each other asset of the release.

This is basically 4 + the released PyPI sdists/wheels in GitHub directly. On that matter, from a packaging perspective, I think it becomes more interesting to keep using a single source, and so drop PyPI entirely in favor of GitHub releases to fetch the sdists. You get as a bonus to being able to check the release you pull (it is also signed) during the packaging process itself.

[bmw]

Thanks for clarifying. Would we also have something like checksums.txt.gpg like I suggested in 4? My understanding is that the only signature on the traefik release is on the git tag and you're still just relying on HTTPS to verify the authenticity of the other assets including the checksum file.

[adferrand]

Yes you are right and I took Traefik just as an example. The point is to rely on operations on the CI and publish additional assets on the GitHub release. From there we follow the most suitable approach for a OS packaging, which is most likely what is described in https://wiki.debian.org/Creating%20signed%20GitHub%20releases. Please note that the natural way with asc files is a signature per file, and other solutions are a little or more hacky (see https://stackoverflow.com/questions/37725969/several-pgp-signatures-for-one-file).

So at the end, we would get a signed GitHub release, with an sdist (+ wheels ?) asset + an asc file for each asset.

[bmw]

I wasn't trying to score debate points or something by nitpicking the traefik example. Sorry if it came across that way. I was just trying to understand what you were thinking about how signing could/should work for us.

I think I prefer the single signed checksum file approach to reduce the number of assets. It's an approach used by Debian themselves and Harlan says there's already some tooling he can leverage for it. I also think we should skip uploading wheels as GitHub assets for now for the same reason, but we can change either or both of these things if the need arises.

If anyone has a problem with this plan, especially Harlan (since this is mainly being done for his needs as the Debian package maintainer), please let me know! I doubt we'll have this for our next release (which I suspect isn't a huge deal as I think we'll have time for more releases before the next Debian or Ubuntu packaging freeze), but I do think that we should prioritize this work soon and I will make sure that happens.

[bmw]

Ubuntu 24.04 freeze is scheduled for February 29th[1]. If we want a version newer than 2.1.0 frozen in Ubuntu 24.04 for the next 5+ years, we should try to resolve this well before then so Harlan has time to update tooling on his end and do a Debian release.

[hlieberman]

I'd definitely like to get this done sooner than later, @bmw. Just got a ping on a downstream bug asking for an update, actually.

If we need to get to a better version soon, we could work something out as a one-off. I'd be happy to accept a GPG signed email with a verification of the hashes from you or another one of the devs, lacking a permanent solution.

[ohemorange]

@hlieberman here's a tarball of the release signatures for each package
2.9.0_sigs.tar.gz
sha256sum of 2.9.0_sigs.tar.gz is 01419066ef55e27b8036d5ed96eeac5c35e472d249705488475dc6cfd789e6bd, I can send you that hash through other channels as well.

Edit:
Just in case this needs to be redone, here are the steps to generate these:

mkdir 2.9.0_sigs
cp releases/le*/dist*/*.gz.asc 2.9.0_sigs
tar -czvf 2.9.0_sigs.tar.gz 2.9.0_sigs/
sha256sum 2.9.0_sigs.tar.gz 

[bmw]

starting with our next release, all source code tarballs will be included in our github release along with a SHA256SUMS and SHA256SUMS.asc file using our same signing keys. please let us know if you have any trouble with this harlan

[hlieberman]

Seems perfect to me! I'll open a Debian bug to move the watch files for everything over. Sincerely, -- Harlan Lieberman-Berg ~hlieberman