feat: add cert rotator

[binbin-li]

Description

What this PR does / why we need it:

This is a following PR of #831. Add cert-rotator in Ratify to support rotating TLS related certificates automatically. The overall design doc can be found here: https://hackmd.io/@akashsinghal/BySB4VbHh

1. We adopt the cert-controller used in Gatekeeper. The cert-controller checks the validation of certificates every 12 hours and generates a new certificate that is valid for 10 years 90 days before the old certificate expires.
2. Upgrade GK versions from 3.10.0/3.11.0 to 3.11.0/3.12.0 as cert-controller only works for the externaldata.gatekeeper.sh/v1beta1.
3. Add access permissions to providers and secrets for Ratify as it needs to list/update them.
4. Add featureFlag CERT_ROTATION so that this feature is behind FF.
5. Add Ratify_NAME env var to Ratify as cert-rotator needs it for validation and cert generation.
6. Updated generate-tls-certs.sh so that the generated certs can pass the cert-rotator validation.
7. Add blocker for tlsCertWatcher so that cert-rotator will not start until certWatcher is ready.

Which issue(s) this PR fixes (optional, using fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when the PR gets merged):

Fixes #834

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Helm Chart Change (any edit/addition/update that is necessary for changes merged to the main branch)
• This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Please also list any relevant details for your test configuration

Adding a new e2e test which will be provided an expiring cert in deployment, which will trigger a cert rotatation immediately.

Checklist:

• Does the affected code have corresponding tests?
• Are the changes documented, not just with inline documentation, but also with conceptual documentation such as an overview of a new feature, or task-based documentation like a tutorial? Consider if this change should be announced on your project blog.
• Does this introduce breaking changes that would require an announcement or bumping the major version?
• Do all new files have appropriate license header?

Post Merge Requirements

• MAINTAINERS: manually trigger the "Publish Package" workflow after merging any PR that indicates Helm Chart Change

[codecov]

Codecov Report

Patch coverage: 18.18% and project coverage change: -0.48 ⚠️

Comparison is base (c13f848) 55.03% compared to head (7cab6f8) 54.55%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #869      +/-   ##
==========================================
- Coverage   55.03%   54.55%   -0.48%     
==========================================
  Files          80       81       +1     
  Lines        4739     4797      +58     
==========================================
+ Hits         2608     2617       +9     
- Misses       1848     1896      +48     
- Partials      283      284       +1     

Impacted Files	Coverage Δ
cmd/ratify/cmd/serve.go	43.75% <0.00%> (-0.94%)	⬇️
httpserver/server.go	6.12% <0.00%> (-0.13%)	⬇️
pkg/featureflag/featureflag.go	100.00% <ø> (ø)
pkg/manager/manager.go	3.59% <0.00%> (-1.25%)	⬇️
pkg/utils/podinfo.go	100.00% <100.00%> (ø)

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[susanshi]

As discussed offline, lets validate if Ratify throws error on startup when customer provides a expired cert.

[susanshi]

PR looks great Binbin, could you provide more details on logs and test coverage?

Have we covered cases for:

• rotation customer provided cert
• rotation for ratify auto generated cert
• Are there any customer provided cert that we don't support rotation that we should document?
• Once a cert is rotated by the rotator, what does ratify log look like?
• Have we validated ratify 's configuration matrix with cert rotater disabled?
  thanks!

[binbin-li]

PR looks great Binbin, could you provide more details on logs and test coverage?

Have we covered cases for:

• rotation customer provided cert
• rotation for ratify auto generated cert
• Are there any customer provided cert that we don't support rotation that we should document?
• Once a cert is rotated by the rotator, what does ratify log look like?
• Have we validated ratify 's configuration matrix with cert rotater disabled?
  thanks!

1. For customer provided/ratify generated certs, cert-rotator will behave in the same way. If they're invalid, rotator will refresh invalid certs, and generate a log like:
   
   And if it's valid, it will generate a log like:
   

2. Actually if customer provided any incompatible/invalid certs, cert-rotator will just auto-rotate it. But I can add some docs on the cert generated by cert-rotator so that customers can follow.

3. I'll add a basic notation test with rotator disabled.

[binbin-li]

I tried to use helm upgrade which works without providing --set featureFlags.RATIFY_CERT_ROTATION=true flag. But it always failed to deploy Ratify pod with it. I checked logs which shows the pod is already up but helm regards it as not ready. What's more interesting is that the same helm upgrade command works on my local machine.

[akashsinghal]

oh that's interesting. Maybe there's another resource other than the pod that is failing to be upgraded due some cpu/mem limitation? Either way I'm personally ok with your solution.

[akashsinghal]

lgtm