-
Notifications
You must be signed in to change notification settings - Fork 944
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Roadmap] Implement PEP 740 #15871
Comments
I've gone ahead and filed a tracking issue in pip-tools as well: jazzband/pip-tools#2080. |
With the exciting changes suggested in pypa/gh-action-pypi-publish#230, it looks like we'll be able to leverage bits of composite actions and call the action even (if it makes sense to do so, as an alternative to using a Python lib within the container like the rest of stuff). |
Should this be done by the uploader, though? My initial feeling was that this should be done right after the job building the artifacts. Though, perhaps not by pypa/build (because of the privilege elevation concerns I often complain about). Are we going to treat the artifacts that the uploader gets as coming from trusted sources? Should we be emphasizing the security considerations and teaching the users more about the least privilege principle? |
Hmm -- I suppose it doesn't have to be, although for targeting publish provenance (and getting provenance generation exposed to the largest number of users) I think it's the most straightforward path. Long term, PEP 740 is designed to enable multiple attestations: the building step(s) can generate one, as can the publishing step. But I believe getting that into a workable state will require a decent bit more coordination, so I want to target an MVP of "the artifact is signed by the same identity that pushed it to the index" 😅 |
@woodruffw fair! I just wanted to make sure we're on the same page. |
@webknjaz Glad you're thinking about build provenance already, I'm interested in tackling build provenance for Python too 🚀 These initial attestations are similar to NPM's publish provenance so is best done by the uploader. If the attestation can show "this artifact was uploaded by this workflow which matches a Trusted Publisher configuration for the project" we can already build some interesting things like verified GitHub URLs, exposing which artifacts have Trusted Publishers configured to users, monitoring for changes in provenance, etc |
@sethmlarson fair enough! I remember similar ideas being brought up during the initial private beta of trusted publishing. It's nice to have some more or less specified goals recorded and synchronized, though, which is why I posted my thoughts :) Looking back, I'm glad that I encouraged implementing it in my action from the very beginning, months before it's gone GA so everybody could start using it right away with almost no changes. So I'm hoping that the Sigstore integration happens in a similar fashion, enabling any early adopters to start uploading signatures with zero changes to their workflows (if they already use trusted publishing, they have the OIDC access set up). |
I think your initial feeling is valid: since it's not guaranteed that the workflow using the publish action is also the workflow that built the artifacts, this isn't necessarily "build provenance". That said, having an attestation ('publish provenance') from the uploader in the absence of any other attestation being provided to it would still be very valuable here I think, especially for projects that aren't using Trusted Publishing yet. |
@woodruffw What behavior do we want when uploading attestations to a file already in PyPI? Currently, the behavior for duplicate files is either stop the upload process and return OK (if the new file is the same as the existing one), or fail (if the files don't match). |
For the MVP, I think we want to skip if the distribution file already exists, for consistency with the existing upload API behavior. Longer term, this probably falls under the "Upload 2.0 API" PEP -- we'll want a way to augment existing uploaded distributions with new attestations. But that's out of scope for now, IMO 🙂 |
FYI: there's some new developments within GitHub, it seems — pypa/gh-action-pypi-publish#236 (comment). |
I think this one is somewhat separate — it signs built artifacts (in that snippet) while the |
Something that @haydentherapper raised: Trusted Publishing supports private repos. This isn't a "disclosure" issue at the moment since we don't expose the Trusted Publisher metadata anywhere, but with PEP 740 we will. Some points:
Some engineering details:
|
Per NPM's docs, it looks like they require a public repo for provenance:
This doesn't quite mesh with PyPI's model since each distribution file has its own metadata, but in theory we could check them all for consistency. But that is potentially more brittle than just rejecting attestations if the accompanying OIDC JWT doesn't have the right visibility. |
From discussion in the Sigstore Slack: one route forward here is to reject attestations from private repositories, unless the project explicitly is explicitly configured (either on PyPI or the project metadata) to allow them. This eliminates the risk of a user inadvertently leaking a private repository name (or similar) through their Trusted Publisher configuration and subsequent publish attestations. If configured on PyPI, this could be something like a new checkbox under If configured via project metadata, things get a little hairier (since the metadata is duplicated once per distribution, and there's no free-form metadata at the moment). So doing it via PyPI is probably easier. |
@facutuesca and I had a conversation about the persistence side of this the other day; summarizing:
Some related thoughts:
|
This might add some complexity, since right now there is no way of modelling an "inactive" publisher: if a publisher exists in the DB, then it's active. Do we want to add a new column to the table, an |
Hmm, I hadn't thought of this. A new |
I was imagining a separate model for a "verified attestation" that's separate from the publisher, due to the publisher potentially changing over time. |
I think there are two separate pieces of state here:
(2) may also need a relationship to the "thing" that can verify it, which for now will always be an There might be a better way to model all of that, of course. But we'll definitely need both the verified attestation and the thing that verified it, i.e. the |
After further discussion with @woodruffw, we think that it might be better to verify a release's URLs once (at the time the release is created, during the first file's upload), and store the fact that the URL was verified in the release metadata. This implies:
In short, whether a release's metadata URLs are verified or not depends exclusively on the first file uploaded (and its corresponding attestation). The decision comes from the fact that PEP-740 allows uploading releases where only some of the distributions have attestations, so whether a release has "verified" URLs can be ambiguous if some of the files have publish attestations and others don't. Instead, we choose to use the first file uploaded (the one that creates the release) as the source of truth for verifying the release's URLs. Also, by verifying the URL once and storing the result, we skip having the |
@facutuesca @woodruffw Is there a risk in checking and updating the verified status of URLs for each artifact? It was noted somewhere that many projects use multiple build infrastructures to upload artifacts, it would be unfortunate to lose the "verified" status of metadata based on a race of build infra. |
I don't think there's a risk per se, but it's unfortunately not currently straightforward to do with Warehouse's current TL;DR: to check each artifact, I think we'd need a stronger consistency guarantee about the metadata recorded in the (The alternative here would be to consider the |
I don't really love this, as it requires us to make sure we check I think I'd prefer storing details on the event instead, which would allow us to persist the details even if the publisher is deleted.
I think we should probably do that eventually, but it's OK to maintain the status quo for now and only consider the first file published. |
PEP 740 is in a final but not yet approved state. This issue is intended to lay out the dependencies/subcomponents of its implementing, including things that can be done in a preliminary manner.
Index side
attestations
form field in the current legacy upload endpointattestations
and verify each against the uploading trusted publisher; fail the file upload if any attestations are invalidUploader/publish side
gh-action-pypi-publish
gh-action-pypi-publish
should usesigstore-python
to sign the attestation payload defined in PEP 740twine
: Proposal: preliminary support for PEP 740 pypa/twine#1094twine
supports uploading,gh-action-pypi-publish
should use that support to actually upload the attestation objects it generates aboveCC @di @webknjaz @facutuesca for visibility
The text was updated successfully, but these errors were encountered: