Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: Process for signing charts (establishing provenance) #23

technosophos opened this issue Aug 6, 2016 · 2 comments

proposal: Process for signing charts (establishing provenance) #23

technosophos opened this issue Aug 6, 2016 · 2 comments


Copy link

@technosophos technosophos commented Aug 6, 2016

We want to be able to add provenance to all charts in this repo. The helm command has (admittedly limited) provenance tools to help with this. This proposal outlines how we could add provenance data to charts.

Fundamental Assumption: A chart with an accompanying .prov file is an assertion about the integrity of a file. Specifically:

  1. The chart is crytographicly verified (hashes match, content is unaltered)
  2. The chart has been reviewed and verified by the core team, and the signer has verified this
  3. The signer is a person who has the authority to speak for the "official" kubernetes/charts team

The process outlined here is designed to meet each of those three expectations.

When is a Chart Signed?

A chart is signed when it has passed all of its acceptance criteria, and it is ready to be pushed to the chart repository. So the chart should be (a) passing automated tests (CI), (b) approved via LGTM or whatever human process we use, and (c) accepted as ready to release (e.g. the one who opened the request believes it is now ready for release).

At that point, one of the "core maintainers" signs the chart and commits the result to the repo.

$ helm package --sign --key "my-key" mysql
$ git add mysql-1.2.3.tgz.prov
$ ... # rest of Git workflow

(alternately: we could have the signer load the signature directly into GCS. But then it will not be tracked in revision control)

At this point, the CI/CD toolchain can actually verify that it can rebuild the package and match the .prov file before it sends the chart to the chart repository. This guarantees that we don't accidentally release a chart that is half done.

So this process addresses assumptions 1 and 2: It provides cryptographic proof that a chart is unaltered, and it asserts that someone has verified that the process of approving the chart has been completed.

How do we manage cryptographic keypairs?

Every core maintainer of the project may have one or more signing keypairs (ideally just one).

Private keys: In this model, the signer is the only one who needs the private key. So none of the "official" infrastructure needs to store private keys.

Public Keys: The public keys used to sign must be made available by the maintainer. Possible ways to make these available include mundane ways like sending them via email, or other ways like storing them in (cf.

How do we say which public keys are trusted for the kubernetes/charts repository?

We need a way to assert something like this: "The following keys are official for the kubernetes/charts repo..."

One way to do this would be to put those public keys in a well documented place that the helm tool itself could look up. It might be necessary to also cryptographically sign that information. with a "root key" or use one of the other chain of trust mechanisms popular in PKI.

As a first step, this proposal suggests that a binary keyring file called pubkeys.pgp be added to the root of the repository on GCS, and that this keyring file (temporarily) be manually managed by the team lead for the kubernetes/charts project. (That'd presumably be @prydonious)

If this is deemed sufficient, we should file an issue at to include support for automatically fetching and using this file to verify charts in a given repo.

So it would work like this:

  1. User runs helm install charts/mysql-1.2.3
  2. Helm client updates index.yaml for kubernetes/charts
  3. Helm client fetches pubkeys.pgp for kubernetes/charts and caches it locally
  4. Helm client fetches mysql-1.2.3.tgz and mysql-1.2.3.tgz.prov
  5. Helm client cryptographically verifies mysql-1.2.3.tgz against the .prov file using ONLY the keys in the cached pubkeys.pgp file.
  6. Helm installs the chart

(OPTIONAL STEP 3.5) Helm client verifies that the key used to sign is in some other chain of trust (say, local keyring or

How do we manage compromised/rogue keys?

Say we successfully implement this plan, but at some point a key that was used to sign numerous charts becomes compromised. What do we do?

  1. The key is removed from the pubkeys.pgp file in the GCS storage. This will result in all charts signed with that key becoming untrusted. And that is good.
  2. We manually re-audit all charts created with that key, and a different key is used to sign those. That different key must already be in the pubkeys file

In addition, we could provide a canonical place to post revoked keys. (perhaps revoked.pgp)

Known Weaknesses

The following are known weaknesses in this scheme:

  • If the pubkeys.pgp file is compromised, we lose the entire chain of trust. This could be partially mitigated by signing the pubkeys.pgp file with a "root key" (discussed above), and having that root key managed on or another third-party trusted key repo.
  • It assumes that each individual signer will be responsible with his/her secret key files
  • A full-on domain hijack of the chart repo could completely circumvent this. This is partially mitigated by an SSL certificate on the domain.
Copy link

@philips philips commented Sep 12, 2016

Can this be moved over to the helm repo? The provenance docs live over there as well.

Copy link

@seanknox seanknox commented May 25, 2017

This was completed by the wizards of Helm in the Helm repo proper.

@seanknox seanknox closed this May 25, 2017
jordanjennings pushed a commit to jordanjennings/charts that referenced this issue Nov 27, 2018

Set up changes for MITM PT registration
hairmare pushed a commit to hairmare/charts that referenced this issue Jun 22, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants