Fix unsigned patch releases

[puerco]

While cutting the February patch releases, the image promoter got rate limited by Fulcio, the sigstore certificate authority (see this long thread in slack for more context). This caused the signatures in the published images to be in an inconsistent state: some images are signed, some not, and some don't have their signatures replicated.

In order to fix the problem we need to check the signatures of images, ensure they are signed with the expected identity, and that they are correctly replicated. Then, based on that there are two actions to be taken:

1. Sign and replicate those missing signatures
2. Replicate signatures of any partially signed images

After manually fixing these, we can move the promoter subcommand to audit the signatures in the future.

Justification

The signatures on our images are the stamp of approval to show that the community approved them to be published to the production registries. Any signed image can be traced back to a PR in a manifest where the change was signed off by the relevant community members. We can always sign them after publishing by ensuring we are signing on the correct digests based on the manifest data.

Action Plan

• Manually map out which images were affected slack ref
• Add a command to the image promoter to find unsigned images, it should support date ranges and --dry-run to check what it would do
  kpromo sigcheck: Verify and fix missing image signatures kubernetes-sigs/promo-tools#745
  Signature Check: Check Expected Identity in Certificate kubernetes-sigs/promo-tools#767
• Expose the concurrency limits to control them from the promoter's job configuration
  cip: Expose signing concurrency as flags, default to 50 kubernetes-sigs/promo-tools#770
• Rate limit copy calls in the promoter to stay under the AR quotas
  Rate limit crane.Copy() operations kubernetes-sigs/promo-tools#771
• Release k-sigs/promo-tools@v3.5.1 kubernetes-sigs/promo-tools#752
• Create a one-time run to ensure running within prow has the correct view of things
• Do a test run with one of the images
• Run it on the rest images
• Update the promoter code to verify any unsigned images against the manifests to ensure we sign allowed artifacts
• Modify the job to constantly audit and verify the latest promoted images

/cc @cpanato @kubernetes/release-managers

[puerco]

/priority critical-urgent

[puerco]

The scope of this issue is now expanded to fix the March patches which got rate limited when calling the registry. This is a new problem and we now have to maneuver around the AR registry limits [slack ref].

[puerco]

/assign

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[xmudrii]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[xmudrii]

/retitle Fix unsigned patch releases
/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[xmudrii]

/lifecycle frozen

[anshikavashistha]

@puerco This project seems interesting to me. I really want to work on this project .Is there any prerequisite task that needs to be done ?
Please share the link of community channel or any slack channel.