proposal: Process for signing charts (establishing provenance)

[technosophos]

We want to be able to add provenance to all charts in this repo. The helm command has (admittedly limited) provenance tools to help with this. This proposal outlines how we could add provenance data to charts.

Fundamental Assumption: A chart with an accompanying .prov file is an assertion about the integrity of a file. Specifically:

1. The chart is crytographicly verified (hashes match, content is unaltered)
2. The chart has been reviewed and verified by the core team, and the signer has verified this
3. The signer is a person who has the authority to speak for the "official" kubernetes/charts team

The process outlined here is designed to meet each of those three expectations.

When is a Chart Signed?

A chart is signed when it has passed all of its acceptance criteria, and it is ready to be pushed to the chart repository. So the chart should be (a) passing automated tests (CI), (b) approved via LGTM or whatever human process we use, and (c) accepted as ready to release (e.g. the one who opened the request believes it is now ready for release).

At that point, one of the "core maintainers" signs the chart and commits the result to the repo.

$ helm package --sign --key "my-key" mysql
$ git add mysql-1.2.3.tgz.prov
$ ... # rest of Git workflow

(alternately: we could have the signer load the signature directly into GCS. But then it will not be tracked in revision control)

At this point, the CI/CD toolchain can actually verify that it can rebuild the package and match the .prov file before it sends the chart to the chart repository. This guarantees that we don't accidentally release a chart that is half done.

So this process addresses assumptions 1 and 2: It provides cryptographic proof that a chart is unaltered, and it asserts that someone has verified that the process of approving the chart has been completed.

How do we manage cryptographic keypairs?

Every core maintainer of the project may have one or more signing keypairs (ideally just one).

Private keys: In this model, the signer is the only one who needs the private key. So none of the "official" infrastructure needs to store private keys.

Public Keys: The public keys used to sign must be made available by the maintainer. Possible ways to make these available include mundane ways like sending them via email, or other ways like storing them in https://keybase.io (cf. https://keybase.io/technosophos).

How do we say which public keys are trusted for the kubernetes/charts repository?

We need a way to assert something like this: "The following keys are official for the kubernetes/charts repo..."

One way to do this would be to put those public keys in a well documented place that the helm tool itself could look up. It might be necessary to also cryptographically sign that information. with a "root key" or use one of the other chain of trust mechanisms popular in PKI.

As a first step, this proposal suggests that a binary keyring file called pubkeys.pgp be added to the root of the repository on GCS, and that this keyring file (temporarily) be manually managed by the team lead for the kubernetes/charts project. (That'd presumably be @prydonious)

If this is deemed sufficient, we should file an issue at https://github.com/kubernetes/helm to include support for automatically fetching and using this file to verify charts in a given repo.

So it would work like this:

1. User runs helm install charts/mysql-1.2.3
2. Helm client updates index.yaml for kubernetes/charts
3. Helm client fetches pubkeys.pgp for kubernetes/charts and caches it locally
4. Helm client fetches mysql-1.2.3.tgz and mysql-1.2.3.tgz.prov
5. Helm client cryptographically verifies mysql-1.2.3.tgz against the .prov file using ONLY the keys in the cached pubkeys.pgp file.
6. Helm installs the chart

(OPTIONAL STEP 3.5) Helm client verifies that the key used to sign is in some other chain of trust (say, local keyring or keybase.io).

How do we manage compromised/rogue keys?

Say we successfully implement this plan, but at some point a key that was used to sign numerous charts becomes compromised. What do we do?

1. The key is removed from the pubkeys.pgp file in the GCS storage. This will result in all charts signed with that key becoming untrusted. And that is good.
2. We manually re-audit all charts created with that key, and a different key is used to sign those. That different key must already be in the pubkeys file

In addition, we could provide a canonical place to post revoked keys. (perhaps revoked.pgp)

Known Weaknesses

The following are known weaknesses in this scheme:

• If the pubkeys.pgp file is compromised, we lose the entire chain of trust. This could be partially mitigated by signing the pubkeys.pgp file with a "root key" (discussed above), and having that root key managed on keybase.io or another third-party trusted key repo.
• It assumes that each individual signer will be responsible with his/her secret key files
• A full-on domain hijack of the chart repo could completely circumvent this. This is partially mitigated by an SSL certificate on the domain.

[philips]

Can this be moved over to the helm repo? The provenance docs live over there as well.

[seanknox]

This was completed by the wizards of Helm in the Helm repo proper.