Support signing releases with a PGP key in PyPi deployments

[thedrow]

Twine supports doing just that using the --sign flag.
Could we please add support for it in dpl?

[webknjaz]

@thedrow to do this the PR author would need to solve issue of putting/encoding key blobs into travis config.

[jezdez]

So I'm not sure if putting a private GPG key into a repo is considered sensible even if it's encrypted. Just for the sake of argument though, Travis has a way to encrypt files: https://docs.travis-ci.com/user/encrypting-files/

Someone used that API to put a GPG keyring into a repo: https://stackoverflow.com/questions/45188811/how-to-gpg-sign-a-file-that-is-built-by-travis-ci/45201190#45201190

There a lots of moving parts that can't be easily implemented just by a patch in dpl though, I'm not convinced this is a great idea. On top of that, no Python installer right now AFAIK supports GPG signature verification (for the same reasons of having lots of moving parts), which means I'm not sure why you'd need that in the first place.

[webknjaz]

My point was that I'd not encourage deployment process, which would require users doing extra manual setup and/or encryption additionally to just configuring dpl.
I'd like to see it all embedded into single and easy-to-use config.

Encrypting files separately is a low-level API, which is okay if users use it when building their custom flows. On the other hand, dpl (and deploy: step in Travis) is a high-level API. I'd say it's harmless to mix those, even if it's documented.

[webknjaz]

@thedrow you may try to do a command injection via distributions: option 😆

[thedrow]

@jezdez The Celery project is required by the Debian packaging policy to sign all releases. See celery/kombu#773
The lack of this feature prevents us from providing automated releases for all Celery related projects.

[jezdez]

@thedrow I can't speak for Debian's packaging policy (do you have a link elaborating on the requirement to sign all releases?) but as a fellow Celery member I'm not aware of plans of the Celery team how to mitigate a breach of Travis-CI encryption keys which would require reissuing the release signing keys. Which is what makes me uneasy to sign packages automatically on a 3rd party system whose encryption keys is outside our control in the first place.

I'd be interested to have others from the Python packaging world that have a more up-to-date view on this (and more security background) than I chime in of course. @dstufft? @ncoghlan? @alex? @ewdurbin?

[alex]

As far as I know, debian is the only PyPI consumer that does anything with the PGP signatures.

They do nothing for any pip users (the majority of downloads AFAIK).

Personally I don't think we should invest any effort into the (completely broken) PGP infrastructure. If people want to spend time to make PyPI more resilient, they should contribute to warehouse and to deploying TUF.

[ewdurbin]

Regarding PyPI specific stuff

PyPI currently dutifully accepts and serves detached signatures uploaded along side files as specified in PEP 503.

If there is a GPG signature for a particular distribution file it MUST live alongside that file with the same name with a .asc appended to it. So if the file /packages/HolyGrail-1.0.tar.gz existed and had an associated signature, the signature would be located at /packages/HolyGrail-1.0.tar.gz.asc.

No other handling of GPG/PGP material is currently present or planned.

If consumers of a package have a sufficient GPG/PGP Web of Trust established which include the signing key... 🎉 awesome! If not... it really does no good.

I agree with @alex that implementing The Update Framework is a more reasonable long term approach. Draft PEPs 458 and 480 already exist and have explored most details as they relate specifically to PyPI.

With the current split brain state of PyPI running the old codebase and warehouse simultaneously, major changes like this are cumbersome, but as we approach warehouse's release they should be more viable.

Regarding GPG/PGP in general as it relates to dpl

I don't think I'd hand a Signing key off to a third party for automated signing. 🤷‍♀️

If signing releases before upload is a critical part of the workflow, we meet the ever present security vs. convenience situation. The convenience of CI performing releases must be abandoned for the security of having a maintainer perform the signature before upload.

[ncoghlan]

As a downstream, Fedora (et al) don't rely on GPG signatures to authenticate upstream releases - they only use them to secure the link between the distro build servers and distro end users.

I'm also surprised by the statement that signing upstream releases is a requirement for inclusion in Debian, as I definitely don't sign contextlib2 releases, yet https://packages.debian.org/jessie/python/python-contextlib2 still exists.

celery/kombu#773 also gets back to the fundamental problem with attempting to use GPG for anything that isn't the domain of dedicated package maintainers: there are enough new points of potential failure that aren't a result of a security compromise that you end up increasing the chance that folks won't receive a security update due to an error in key management.

It's far less fragile to rely on trust-on-first-release hash checking, potentially enhanced by publishing the expected artifact hashes through multiple channels (so compromise of one channel only gives the ability to conduct a freeze attack by inducing hash mismatches).

[stale]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically closing the issue. This is often because the request was already solved in some way and it just wasn't updated or it's no longer applicable. If that's not the case, please do feel free to either reopen this issue or open a new one. We'll gladly take a look again! You can read more here: https://blog.travis-ci.com/2018-03-09-closing-old-issues

[thedrow]

Since the answer is that this is not a good idea, I'm not sure what else there is to be done here.

[webknjaz]

@thedrow I've got an idea for GitHub integration for this, if I get to implement it I'll post back here.