RFC for linking packages to their source and build #626
Conversation
|
One question for clarification: This RFCs repo has historically been mostly focused on the CLI. Is this RFC for the implementation of this functionality in the CLI or for implementation in the Registry? |
|
|
||
| Code on GitHub, GitLab etc can be browsed and audited, but packages in the registry are opaque and much harder to vet. If packages are built and published in the open these attacks become a whole lot harder. However, right now there’s no way of knowing where a package came from when you retrieve it from the registry. | ||
|
|
||
| We want to solve the problem of npm packages being disconnected from their source by linking the published package to the source code repository and build that it originated from. |
There was a problem hiding this comment.
given that build output should never be committed, and builds/publishes are typically done on local machines, how can this link be verified?
There was a problem hiding this comment.
This RFC will only work with packages published from trusted build infrastructure. It will not support local publish.
There was a problem hiding this comment.
What are the criteria for "trusted"? What are the associated costs in standing up such infrastructure, and what portions of the community would be excluded by those costs?
There was a problem hiding this comment.
@ljharb I believe the premise of this entire RFC is predicated on builds/publishes happening in the open/cloud (as the motivation aims at getting closer to being able to meet SLSA Level 3 standards - ref. https://github.com/npm/rfcs/blob/link-packages-to-source-and-build/accepted/0000-link-packages-to-source-and-build.md#non-falsifiable-provenance-using-a-trusted-builder & https://slsa.dev/spec/v0.1/levels#detailed-explanation)
Notably, "private/inner-source" is considered a "non-goal" (ref. https://github.com/npm/rfcs/blob/link-packages-to-source-and-build/accepted/0000-link-packages-to-source-and-build.md#non-goals) so any local builds/publishes would be out of scope afaik.
There was a problem hiding this comment.
This is all covered in the RFC
There was a problem hiding this comment.
If the manual approval allows me to input an actual second factor token, then that would be great! The things you linked don't seem to permit that yet tho.
I look forward to the RFC discussion.
There was a problem hiding this comment.
@ljharb I'm surfacing that request to the actions team.
In general I see staging / approvals as out of scope for npm as a whole, but up for challenging those assumptions.
There was a problem hiding this comment.
I would like to point out that one of the major use cases of SLSA provenance is to mitigate the threat of compromised (Npm) registry credentials. That's because the provenance provides a link to the source repo. So if an attacker were to compromise a registry credential and push a malicious package, the provenance would change from "source=github.com/genuine/source" to "source=github.com/attacker/source" or even "". The registry or another system that monitors it would immediately detect the compromise. In essence, this means that the system fails "safe" so long as:
- you have 2FA on your registry account, protecting privileged settings, etc
- you use an automation token to push packages.
An attacker who gets hold of an automation token can push packages but these would be immediately detected because of the change of source. In a nutshell, you don't need staged builds if you upload provenance from a trusted builder. (I'm assuming that merely pushing packages in an attempt to DoS a user's registry account is not a valuable attack).
Let me know if there are some nuances I'm missing.
(Note: you could improve on automation token using OIDC to further harden the system and improve user experience.)
There was a problem hiding this comment.
An attacker who gets hold of an automation token can push packages but these would be immediately detected because of the change of source. In a nutshell, you don't need staged builds if you upload provenance from a trusted builder.
There was a problem hiding this comment.
If you disaggregate the system pushing the package to the repo from the system building/signing the artifact, you will be able to protect against the upload token being compromised because you can validate the signed artifacts with published public keys.
Furthermore, sigstore (I believe mentioned in this RFC, I'm still reading through it) can provide some ability to detect a compromised signing key. Will comment more on this later after reading more.
|
@bnb while the repository has historically been focused on the CLI, and this RFC touches a broader surface area, we felt this work was too important to not work through it with the broader community community. |
|
@MylesBorins should we expect discussion time set aside in the npm CLI RFC meeting, a separate synchronous meeting, or no synchronous meeting to discuss? |
|
@bnb added the |
| ##### What goes on the public Rekor ledger? | ||
| Only public npm packages will be signed and published to the public [rekor.sigstore.dev](https://docs.sigstore.dev/rekor/overview) ledger by default. | ||
|
|
||
| Privately scoped packages, or packages from private repositories, will not be signed or published to the public ledger. This will be determined by interrogating the npm registry at publish time. Users will be able to override this similar to the `cosign --force` command by passing an argument to `npm publish`, e.g. `--force-build-provenance`. |
There was a problem hiding this comment.
Is there a potential dependency confusion attack here? Ie. can I get a private package to publish to rekor if I upload a package of that name to the public registry?
There was a problem hiding this comment.
Oh yeah you mean a case where you have some internal corp dep pulled from a private registry and then someone finds this name and creates a public version? Not sure we can do much to mitigate the creation of these. Do you see an added risk being able to create these?
An issue could be getting the wrong provenance attestations during verification if we just used the package name and version. We can mitigate this by querying for the attestations using a shamsum of the installed package's tarball.
There was a problem hiding this comment.
I think what @jchestershopify means is if I have package x internally and someone publishes x, the RFC says that npm will interrogate the registry for x and get the internal package signed accidentally because x exists on the public registry. This is especially heinous in this case, since the ledger is immutable.
There was a problem hiding this comment.
When you say internal package x you mean on some other registry right? I think in this case the signature would be for the public package x and associated via the shasum of the tarball, which won't match the internal package x. Maybe I'm missing something though?
There was a problem hiding this comment.
We're also planning on creating a "release attestation" on the npm registry that gets uploaded to the public Rekor ledger. This is a signed statement that the npm registry accepted and authorized the publish for public package x on top of the "build attestation" that gets created during publish in the CI/CD system, which links the source and build to the package.
|
|
||
| The [above](https://www.esentire.com/security-advisories/npm-library-supply-chain-attack) [attacks](https://www.esentire.com/security-advisories/coa-npm-supply-chain-attack) [are](https://www.esentire.com/security-advisories/rc-npm-supply-chain-attack) examples of this, and they frequently occur due to compromised npm credentials but can also happen due to compromised CI/CD or builds. | ||
|
|
||
| Code on GitHub, GitLab etc can be browsed and audited, but packages in the registry are opaque and much harder to vet. If packages are built and published in the open these attacks become a whole lot harder. However, right now there’s no way of knowing where a package came from when you retrieve it from the registry. |
There was a problem hiding this comment.
unpkg.com makes it much easier - it seems like maybe addressing this (making npm packages browseable on the web safely) would be a good first step?
I'm reasonably certain that the owner of unpkg.com would be happy to transfer the IP to npm if it meant it'd be an official solution (at the least, it's worth asking)
There was a problem hiding this comment.
We already have a beta implementation of exploring code in npm and are funding improvements to it for live code audit on npmjs.com in parallel to this work.
|
|
||
| ## Goals | ||
| - Establishes a verifiable link between a public npm package and the source repository and build it originated from. | ||
| - Does not expose any Personally identifiable information (PII) about maintainers, e.g. emails. |
There was a problem hiding this comment.
emails are already exposed for all publishers to npm - this seems like it should be a non-goal.
There was a problem hiding this comment.
This is about the data that ends up in the public ledger, separate from the registry
There was a problem hiding this comment.
Users may be using different email addresses for their OIDC provider vs the one they use for npm.
There was a problem hiding this comment.
emails are already exposed for all publishers to npm
edit: missed above discussion when posting this.
Yes this is true. We're not proposing changing what goes in the packument. The thing we've focused on in this RFC is the signed statement about the repo and build that will go on a public immutable signature ledger (Rekor). We didn't want a solution where this signed statement would include any kind of identifying information about maintainers.
There was a problem hiding this comment.
I have a friend who is an expert and advocate on this exact topic in favor of maintaining developer privacy. Would it make sense to invite them to review the proposal and provide feedback?
There was a problem hiding this comment.
@fkautz yes please
| - Verification should be performed without depending on any third-party systems other than the registry. | ||
| - Compatible with third-party npm clients, e.g. `yarn` and `pnpm`. | ||
| - Should allow third-party npm registries, e.g. GitHub Packages, Artifactory and Verdaccio to follow suit and implement similar interfaces. | ||
| - Should be maintained with >99.9% uptime so that developers are not blocked from publishing new packages. |
There was a problem hiding this comment.
3 9's seems pretty paltry; can we aim higher?
There was a problem hiding this comment.
Bear in mind this SLO affects publication only; it won't break builds if sigstore is offline. I expect most publishing authors will be fine with waiting a little bit.
There was a problem hiding this comment.
+1, this SLO should be for signing which must be online (for fetching an identity certificate and publishing to the log).
Note that the verification mechanism should be designed with offline verification in mind. For example, querying the log for checking for inclusion would be an online action, but this can be designed to be offline with a persisted proof of inclusion.
There was a problem hiding this comment.
Yes this is only during publish when talking to Sigstore infrastructure. Sigstore is run as free-to-use public good infrastructure hosted on GCP which has SLAs from 99.9% to 99.99% on used services AFAICT.
There's a lot to figure out committing to a SLOs on a shared service like this, with people on-call from different organisations. My hope would be that we can eventually get to four 9s uptime once this process is more mature.
There was a problem hiding this comment.
I reached out to the sigstore team, and they pointed me to this as their current official statement, as of June 29, 2022: https://blog.sigstore.dev/an-update-on-general-availability-5c5563d4e400
They are targeting four months from June 29th (interpreting as end of October) to ramp up operations, tooling, and publishing SLOs.
There was a problem hiding this comment.
Yep I believe the current target for the Sigstore GA is 99.9% uptime for the hosted versions of Fulcio and Rekor.
|
|
||
| ## Sigstore as signing solution | ||
| The [Sigstore](https://www.sigstore.dev/) project has been selected as the solution to signing npm packages (see detailed explanation below). It is currently the only working solution that supports our key requirements: | ||
| - Links packages hosted on the npm registry to the source and build they originated from (provenance information). |
There was a problem hiding this comment.
other than recording "an npm specifier" and "a github repo and SHA" together, how are these linked? published npm packages already include repo and SHA information in the packument, so how would sigstore's linking be different?
There was a problem hiding this comment.
This information would come as an OIDC claim from the publishing party and thus verifiable... e.g. the claim of a specific repo would come from the GitHub action itself, not meta data published to the registry from a personal machine
There was a problem hiding this comment.
Just want to be mindful/note that there's nothing in this RFC which mentions that source, CI or build envs will be or will become immutable; which puts their audibility into question (ex. I can build & publish my package with a trusted builder but then delete that action/workflow/repo - killing the link/trace)
There was a problem hiding this comment.
Does this mean you have to use GitHub as your source repo?
There was a problem hiding this comment.
We're proposing ways to make this information harder to falsify. Currently you can say the repo is anything and it's not validated, so an attacker can easily forge this information.
A part of the solution is using information from the OIDC id token that you can get from a supporting CI/CD system, this includes information about the repo and commit. The whole thing is then signed and can be verified against the CI/CD's public key for example.
There was a problem hiding this comment.
nothing in this RFC which mentions that source, CI or build envs will be or will become immutable
Yes this is true. Worth noting though that the entry on Rekor and the stored copy on npm will show an audit trail of where it was originally built from. This might still be useful for post-mortem analysis. Having a broken link could also end being a strong signal to the community that a package is no longer maintained and should not be used.
Does this mean you have to use GitHub as your source repo?
No, we'll support any provider supported by Sigstore's Fulcio service. Today this only includes GitHub Actions but it's vendor neutral and any provider that meets the following requirements will be supported:
- OIDC ID tokens identifying the current workflow/run/build.
- ID token with a custom audience (needs to be set to
sigstorefor Fulcio to accept it). - Claims about the code repository, commit, build and actor
There was a problem hiding this comment.
Just want to be mindful/note that there's nothing in this RFC which mentions that source, CI or build envs will be or will become immutable; which puts their audibility into question (ex. I can build & publish my package with a trusted builder but then delete that action/workflow/repo - killing the link/trace)
I'm not sure if npm will want to do this but whether links are still valid is something that can be checked during package/artifact verification.
There was a problem hiding this comment.
I'm not sure if
npmwill want to do this but whether links are still valid is something that can be checked during package/artifact verification.
Yeah maybe this could be optional check that can be performed during verification. We probably don't want to do this by default as it could add significant time to the verification if we're checking 1000s+ links for a typical install.
Another idea would be to send the link though a proxy service on npm, and if this service detects a 404 it updates some state on the package that shows a big red warning next to the link next time you view the package on npmjs.com for example. This might cause more headaches than it's worth though, if requests intermittently fail or if the source repo is down and suddenly a bunch of links are broken and need to be fixed when the service is back up.
| Sigstore infrastructure and tooling are currently not considered production ready and are run on a best-effort basis. The Sigstore project is working towards a [General Availability](https://blog.sigstore.dev/an-update-on-general-availability-5c5563d4e400) release later this year but we don't yet know what these guarantees will look like yet. | ||
|
|
||
| As such, there are several risks of adopting Sigstore for npm that are worth calling out: | ||
| - No buy-in from the broader open source npm community. |
There was a problem hiding this comment.
tbh this seems like a potential dealbreaker
There was a problem hiding this comment.
Any of these risks are a potential deal breaker
Our intent and hope is that these will not prove to be a problem but we want to call out that these are indeed risks that could result in this specific approach not working out.
There was a problem hiding this comment.
I don't disagree, but IMO this isn't because there wouldn't be but more because there's been near zero exposure.
| #### How should maintainers authenticate to sign packages? | ||
| [Sigstore](https://www.sigstore.dev/) issues short-lived “disposable keys” notarized against log-ins with OIDC identities. So what OIDC identity provider (IdP) do maintainers use? | ||
|
|
||
| This has been the subject of some controversy; for instance, in the [proposal to add Sigstore signing to RubyGems](https://github.com/rubygems/rfcs/pull/37), users worried about “vendorization” (making the repository reliant on third parties to function, e.g. GitHub, Google or Microsoft) and privacy (some maintainers are pseudonymous; Sigstore certificates include email addresses by default when signing using the [cosign](https://github.com/sigstore/cosign) CLI tool). |
There was a problem hiding this comment.
this should be considered a big concern for npm as well
There was a problem hiding this comment.
Would npm consider being an OIDC Provider for users who do not want to rely on a third party provider? It already has unique user names, manages user identities, and has 2FA support.
There was a problem hiding this comment.
It is an option, that was brought up in #626 (comment).
While it does provide a good user experience, the concern is that it doesn't isolate the identity provider from the package repository, so a compromise of a user account for one would compromise both.
Co-authored-by: Jordan Harband <ljharb@gmail.com>
|
|
||
| We want to solve the problem of npm packages being disconnected from their source by linking the published package to the source code repository and build that it originated from. | ||
|
|
||
| Once a package includes provenance information (where and how it was built) we can start showing this when browsing a package on the npm registry. You would be able to click through to the particular commit and build that published a given version: |
There was a problem hiding this comment.
Do you expect that this data will be able to be reflected in third-party registry caches like Verdaccio, jFrog Artifactory, and Sonatype Nexus, or is it being designed in a way that will be proprietary to the npm website? Additionally, will the API be somewhat reproducible for those third-party registries for non-public modules?
There was a problem hiding this comment.
Ah, this was answered in the bullets below.
There was a problem hiding this comment.
is it being designed in a way that will be proprietary to the npm website? Additionally, will the API be somewhat reproducible for those third-party registries for non-public modules?
We don't want it to be proprietary to the official npm registry, which is partly why we're excited about adopting Sigstore. We're planning to document what is required by third-party npm registries to support this functionality and make it as easy as possible to adopt.
There was a problem hiding this comment.
There are several of us in both the CNCF and OpenSSF who are willing to help spec this out or provide guidance. Please let us know if you would like assistance.
|
|
||
| <img width="376" alt="screenshot of npm package with provenance information" src="https://user-images.githubusercontent.com/20165/176637201-fdb02c11-c810-48e1-a203-ea4fb2008ad3.png"> | ||
|
|
||
| Developers consuming open source packages should get the benefit of this without any changes to their workflows. To begin with, the package integrity should be verified when running the `npm audit signatures` command and eventually transparently integrated into `npm install` and enabled by default. |
There was a problem hiding this comment.
It would be nice to be able to have the existing npm audit signatures functionality and the new functionality as independent features. Not saying that this shouldn't be added to npm audit signatures but more that we should also ensure that end-users can run the two sets of functionality independently.
There was a problem hiding this comment.
Indeed; the entire point of npm audit foo when we discussed npm audit signatures was that every foo would be independently configurable and invokable.
There was a problem hiding this comment.
existing
npm audit signaturesfunctionality and the new functionality as independent features
Thoughts on verifying all available signatures (registry and build) when running npm audit signatures but allowing you to filter the type with an argument like: npm audit signatures --type=registry|build (naming tbd)?
There was a problem hiding this comment.
There is going to be broader shared behavior between the various signature verification commands such as "what do you do if packages are not yet in cache" that lead me to feeling a single command makes far more sense
There was a problem hiding this comment.
FWIW I'm not at all against a single command but I have seen so many situations where people want just one task and don't care about other tasks in a bundled command.
It feels... very un-npm-y to me not to be able to opt-in to choosing more limited but precise sets of work that do the exact thing I want completed, especially when the functionality is fundamentally not related (checking what you received is what's on npm vs. what this RFC proposes).
There was a problem hiding this comment.
Also to answer @feelepxyz: that's a totally fine solution to me!
There was a problem hiding this comment.
npm audit is intended to be the single command - every subpart of it is intended to be granularly configurable and disableable by users, so that nobody is forced to do a check they don't want.
There was a problem hiding this comment.
If the consensus is to split it up into two separate functions, I would recommend splitting something like this:
npm audit signatures # runs everything related to signatures
npm audit signatures compare # only computes the hashes and compares
npm audit signatures integrity # validates the signature and ensures it is present and trusted in sigstoreThis way, npm audit remains a single command, and npm audit signatures does not result in a user misunderstanding what it does.
| Open source maintainers should be able to add build provenance information to their packages with near-zero initial and ongoing overhead. | ||
|
|
||
| ## Goals | ||
| - Establishes a verifiable link between a public npm package and the source repository and build it originated from. |
There was a problem hiding this comment.
It might be good to clarify at some point in this RFC if "source repository" must be GitHub or not. I'm 100% sure this will be a question that comes up, so it'd be good to be clear about that.
There was a problem hiding this comment.
There's no requirement on the source repo being a GitHub repo. The requirement is on the CI/CD system. So for example when CircleCI supports the Fulcio service (which verifies OIDC id tokens from the cicd system and mints short-lived signing certificates) you would be able to use any source repo supported by Circle.
Will clarify this
There was a problem hiding this comment.
Make sure to consider that the source git repo may change, e.g. tags are mutable. branches may be deleted and recreated with new content. Malicious content could be uploaded (by the dev, or through a compromised user credential), a build submitted, then the branch deleted and recreated in an attempt to hide the payload.
(May already be covered, still reading through the document).
There was a problem hiding this comment.
Malicious content could be uploaded (by the dev, or through a compromised user credential), a build submitted, then the branch deleted and recreated in an attempt to hide the payload.
TBH we don't yet have great answers around these issues. We could try and mitigate against some things on the npm side, e.g. by optionally verifying the repo still exists during verification but verifying the correct code still exists is probably not feasible.
It would be great if we could somehow create immutable checkpoints in the source code repo/CI system. One naive idea could be to package up a shallow clone of the git repo as a build artifact
| - Establishes a verifiable link between a public npm package and the source repository and build it originated from. | ||
| - Does not expose any Personally identifiable information (PII) about maintainers, e.g. emails. | ||
| - Avoids developer-managed keys (as there's no good way to offer trust to the community given the challenges of distributing public keys). This, along with opt-in signing, adds near-zero initial and ongoing overhead for open source maintainers. | ||
| - Maintainers can opt-in to including build provenance information (where the code lives and how it was built) when publishing using `npm publish` for public packages. | ||
| - Incentivizes maintainers to build in the open because of the strong guarantees that this offers. | ||
| - Verification happens transparently on `npm install` without the need to obtain or manage additional tools or keys. | ||
| - Verification should have a negligible performance impact on `npm install`. | ||
| - Verification should be performed without depending on any third-party systems other than the registry. | ||
| - Compatible with third-party npm clients, e.g. `yarn` and `pnpm`. | ||
| - Should allow third-party npm registries, e.g. GitHub Packages, Artifactory and Verdaccio to follow suit and implement similar interfaces. | ||
| - Should be maintained with >99.9% uptime so that developers are not blocked from publishing new packages. | ||
| - Should allow future extensions to support centrally managed signing authorities such as certificates managed by an enterprise and inner source within air-gapped enterprise environments. | ||
| - Buy-in from the broader open source community. |
There was a problem hiding this comment.
Almost all of these seem to be effectively external, non-maintainer benefits. Are there any more maintainer-focused goals? Right now, I see very little incentive for maintainers to be excited for this outside of ones like me who work at companies and get paid to do this kind of maintenance.
There was a problem hiding this comment.
This is good feedback
We want to allow authorising the publish using the same OIDC id token from the CI/CD system. So instead of using a long-lived access token to authorize the publish, you would log into your npm dashboard and set up which source repo is allowed to publish what package. The publish step then exchanges the OIDC id token for a short lived access token after the npm registry has checked the source repo from the id token matches the allowed one.
Once you've set up publishing in this way, you would not need to use long-lived access tokens and the risk of stolen credentials basically goes away if you also use 2FA on your npm account. The setup for this would be exactly the same as what we're proposing in this RFC so you essentially get it for free. I imagine adoption would go up a lot once we have both features in place and all major CI/CD providers are fully supported.
There was a problem hiding this comment.
+1 to more clearly articulating the benefits to maintainers. You may also wish to separate goals (e.g. improve security) from requirements (e.g. zero overhead).
To me, the main benefit for maintainers is:
- Maintainers gain increased protection against malicious upload by compromised credentials or (other) rogue maintainers.
| - Running the signed timestamp authority for Rekor as well as a Certificate Transparency monitor for Fulcio. This will help spread the trust outside the four walls of Sigstore. | ||
| - Maintaining the Sigstore trust root and having a root key holder as part of the group of Sigstore root key holders. | ||
|
|
||
| We also remain open to an alternative solution if Sigstore is not able to meet npm's requirements for uptime and support. |
There was a problem hiding this comment.
Are there any potential alternatives that have been identified if this is the case?
There was a problem hiding this comment.
Not concrete alternatives but the rough shape would probably look similar to the architecture in this RFC with the difference that npm/github would be running the services instead. Either fully fledged or a paired down version.
It would be sad if we got to this as it would probably mean that we either end up with some proprietary signature service just for npm/github or potentially multiple competing public good instances.
| The Cosign functionality will be embedded directly in npm CLI, so we're left with the public [fulcio.sigstore.dev](https://docs.sigstore.dev/fulcio/overview/) (CA) and [rekor.sigstore.dev](https://docs.sigstore.dev/rekor/overview/) (ledger) services. | ||
|
|
||
| ##### What does [fulcio.sigstore.dev](https://docs.sigstore.dev/fulcio/overview/) give us? | ||
| - Independent party to validate claims of the OIDC identity token which contains references back to the repo, workflow run and git SHA. |
There was a problem hiding this comment.
If this RFC is only about provenance generated through CI, this is good.
Do you want to support other identities signing provenance or artifacts such as email or SPIFFE IDs?
RFC for linking public npm packages to the source code repository and build it originated from.
View rendered version