Signing artifacts with cosign

[justaugustus]

What would you like to be added:

Signing artifacts with cosign

(I'm opening a placeholder to capture work. Let's capture thoughts here and we'll iterate over the issue description together.)

• Grant Release Manager Admins roles/cloudkms.signerVerifier: infra/gcp: Grant Release Manager Admins roles/cloudkms.signerVerifier k8s.io#2614
• Generate a key for testing: cosign: Add a public key for testing #2226

Why is this needed:

Should supersede sections of #914 once we flesh this out.

/assign @justaugustus @cpanato @puerco
cc: @kubernetes/sig-release-admins @kubernetes/release-engineering @dlorenc @lukehinds

[dlorenc]

Yay!

[justaugustus]

Some stream of consciousness...

• What should we sign (containers, generic artifacts, checksums, etc.)?
• Should we sign images used in CI?
• Oh no! We don't uniquely tag all of those things
• Is/should a signature [be] embedded as part of a manifest?
• Wait, should we sign images before or after promotion?
• If we sign before promotion, will anything be mutated during promotion?
• Shouldn't we get the image promoter to sign as part of the run?
• Should we sign artifacts multiple times? CI/GCB + Release Manager + image promoter
• What sort of annotations should we include as part of the signature?
• Should we always upload all image/artifact types to a transparency log?
• Should we sign individual architecture images, the image manifest, or both?

[jonjohnsonjr]

What should we sign (containers, generic artifacts, checksums, etc.)?

Everything!

Should we sign images used in CI?

Why not?

Oh no! We don't uniquely tag all of those things

No need!

Is/should a signature [be] embedded as part of a manifest?

Definitely not!

Wait, should we sign images before or after promotion?

Why not both?

If we sign before promotion, will anything be mutated during promotion?

Not the artifacts themselves!

Shouldn't we get the image promoter to sign as part of the run?

Sure, why not?

Should we sign artifacts multiple times? CI/GCB + Release Manager + image promoter

Sure!

What sort of annotations should we include as part of the signature?

Great question! Ideally, anything you want to quickly verify.

Should we always upload all image/artifact types to a transparency log?

At least for things you intend folks to consume, I would think yes.

Should we sign individual architecture images, the image manifest, or both?

I'd say just the manifest list, but it depends on if the downstream software understands manifest lists. If that's too hard to keep up, just sign both.

[puerco]

Wait, should we sign images before or after promotion?

I was thinking that, at least for the images that go out as part of the kubernetes releases, we should sign them in CI before promotion and then have the promoter verify the signature to accept them for promotion. And then add a final signature during promotion for verification downstream.

What should we sign (containers, generic artifacts, checksums, etc.)?

Another idea: what about if we start a new repo for .deb and .rpm packages signed with a new key and publish those in parallel to the current ones. Then we slowly make the transition to these as the official ones, signed with a key managed by the community?

[justaugustus]

Thanks for the quick response, @jonjohnsonjr!

Let's sign ALL THE THINGS!
GCB tags + substitutions are probably a good place to start for annotations.

we should sign them in CI before promotion

+1

and then have the promoter verify the signature to accept them for promotion. And then add a final signature during promotion for verification downstream.

Hearing a few things for image promoter:

• enable validation
• validate that the promoter can support non-image media types
• enable signing

Another idea: what about if we start a new repo for .deb and .rpm packages signed with a new key and publish those in parallel to the current ones. Then we slowly make the transition to these as the official ones, signed with a key managed by the community?

Are you referring to:

1. apt/yum repos, or
2. "image" repos

If #1, let's consider this out of scope, at least for this issue:

• Google does not plan on providing more apt/yum repos
• time and cost associated with rolling/maintaining our own separately from container images

If #2, I think you're touching on the "galaxy-brain stuff" I've mentioned on a few of the leads call!

As an example:

Homebrew now hosts all of their bottles on the Container registry. This was made possible because the Container registry supports OCI images. Homebrew can now distribute over a half a petabyte of binary packages to their users every month while surfacing them in GitHub search for improved discoverability. Shaun Jackman, a Homebrew maintainer, found the OCI support particularly useful and said, “GHCR’s implementation of ORAS/OCI provides a standard API to query, upload, and download binary artifacts, so that Homebrew could use off-the-shelf compatible tools, easing our migration of bottle storage to GHCR.”

ref: https://github.blog/2021-06-21-github-packages-container-registry-generally-available/

I'd like to modify the image promoter to be a generic artifact promoter that uses GCR/GAR as the backend.

If we do that, we can genericize all of the signing/validation workflows.

[saschagrunert]

And then add a final signature during promotion for verification downstream.

Isn't changing the signature after promotion confusing for end users? Why won't wee keep the original signature if the content of the image does not change during promotion?

[cpanato]

second what @puerco said

[lukehinds]

I am not sure if you're at the right stage of planning to this level, but you should definitely start recording entries into rekor as well.

[justaugustus]

Isn't changing the signature after promotion confusing for end users? Why won't wee keep the original signature if the content of the image does not change during promotion?

@saschagrunert -- How I see this potentially playing out (for release artifacts in particular):

• Stage: Artifacts are signed with GCB w/ a key held by @kubernetes/sig-release-admins / @kubernetes/release-engineering
• Release:
  • Artifacts are signed by the artifact/image promoter
  • krel validates signatures from RelEng + artifact/image promoter before completing the release
  • We publish multiple signatures for user validation

Optionally, somewhere in the middle, we could have the Release Manager in charge sign the staging images as part of krel promote-images and promote the:

• artifacts
• GCB staging signature
• Release Manager signature

I am not sure if you're at the right stage of planning to this level, but you should definitely start recording entries into rekor as well.

@lukehinds -- Absolutely!

[justaugustus]

cc: @kubernetes/sig-release @kubernetes/wg-k8s-infra

[sftim]

My question: does this change need a KEP?

[justaugustus]

My question: does this change need a KEP?

Discussion here: kubernetes/sig-release#1724

[shibumi]

A few thoughts on this:

Right now, there is a cosign test key for signing. It would be nice, if we could switch to keyless signing via workload identities.
I wrote a little blog article about how to sign BLOBs with cosign keyless: https://shibumi.dev/posts/first-look-into-cosign/

The next cosign version should be able to:

• create a keyless signature for any artifact
• store the signature on disk/github release log and/or in the rekor transparency log.

Current blockers:

• The public rekor/fulcio instances are still "test environments" and not meant for production use.
• The cosign feature for keyless signatures is still considered experimental.
• I don't know how the Github OIDC-Issuer / Workload Identity part will work in a big org like kubernetes.

I would suggest to wait until the sigstore stack is ready :)

It also might make sense to combine all of this with TUF or in-toto on the long run. First TUF, then in-toto.

[BenTheElder]

if we could switch to keyless signing via workload identities.

+1, one concern I have is making sure that the signatures are meaningful, having workload identity based signatures from the relatively locked down release process would provide some assurances.

I don't know how the Github OIDC-Issuer / Workload Identity part will work in a big org like kubernetes.

build / publish actually runs in Google Cloud Build currently anyhow, FWIW. (or CI builds happen in our own Kubernetes based CI, which has GKE workload identity available).

Definitely smells like a KEP to me :-)

Somewhat related: I think if we want to sign CI builds we need to better secure them first (move them out to GCB for starters).

[shibumi]

@BenTheElder Just for the record: https://github.com/google-github-actions/auth this might be interesting then, right?

[BenTheElder]

Er I don't think so for the main project 😅
The main Kubernetes project isn't using GitHub Actions.
We use:

• https://github.com/kubernetes/test-infra/tree/master/prow
• https://cloud.google.com/build

Some of the smaller repos use GitHub Actions for various things but not Kubernetes releases or CI builds.

[saschagrunert]

The initial draft of the sining KEP is proposed in kubernetes/enhancements#3061

[trishankatdatadog]

Would be a great pioneering use case for sigstore trust delegations! @asraa

[saschagrunert]

Tracking the first implementation in #2383

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[ameukam]

/remove-lifecycle stale
/milestone v1.24

[ameukam]

/priority important-longterm

[saschagrunert]

I think this can be closed and we can track the work in kubernetes/enhancements#3031