apiext: replace expired webhook certificates

[superfrink]

Description

Currently, emissary-apiext will create a new webhook key and certificate when one does not exist and does not replace the certificate when it is expired. The change in this PR automatically replaces the key and certificate when it is expired.

Related Issues

Fixes #4442

Testing

Manual testing by scaling the emissary-apiext pods to zero and back up again. First with an expired certificate and second with a not-expired certificate.

$ kubectl -n emissary-system get secret emissary-ingress-webhook-ca -o yaml | grep tls.crt | awk '{print $2}' | base64 --decode | openssl x509 -noout -text | grep Not
            Not Before: Dec 23 00:57:19 2023 GMT
            Not After : Dec 23 00:59:19 2023 GMT

$ date
Fri 22 Dec 2023 17:59:37 MST

$ kubectl -n emissary-system scale --replicas=0 deployment emissary-apiext
deployment.apps/emissary-apiext scaled

$ kubectl -n emissary-system scale --replicas=1 deployment emissary-apiext
deployment.apps/emissary-apiext scaled

$ kubectl -n emissary-system get secret emissary-ingress-webhook-ca -o yaml | grep tls.crt | awk '{print $2}' | base64 --decode | openssl x509 -noout -text | grep Not
            Not Before: Dec 23 00:59:58 2023 GMT
            Not After : Dec 23 01:01:58 2023 GMT

$ date
Fri 22 Dec 2023 18:00:08 MST

$ kubectl -n emissary-system scale --replicas=0 deployment emissary-apiext
deployment.apps/emissary-apiext scaled

$ kubectl -n emissary-system scale --replicas=1 deployment emissary-apiext
deployment.apps/emissary-apiext scaled

$ kubectl -n emissary-system get secret emissary-ingress-webhook-ca -o yaml | grep tls.crt | awk '{print $2}' | base64 --decode | openssl x509 -noout -text | grep Not
            Not Before: Dec 23 00:59:58 2023 GMT
            Not After : Dec 23 01:01:58 2023 GMT

The new log message:

time="2023-12-23 00:59:53.5883" level=warning msg="Will try to replace cert not valid before 2023-12-23 00:57:19 +0000 UTC and after 2023-12-23 00:59:19 +0000 UTC" func=github.com/emissary-ingress/emissary/v3/pkg/apiext/internal.EnsureCA file="/go/pkg/apiext/internal/ca.go:69" CMD=apiext PID=1

The log message was not present when scaling up the pods and the certificate expiry date was still valid.

Checklist

• Does my change need to be backported to a previous release?

  • What backport versions were discussed with the Maintainers in the Issue?

• I made sure to update CHANGELOG.md.

  Remember, the CHANGELOG needs to mention:

  • Any new features
  • Any changes to our included version of Envoy
  • Any non-backward-compatible changes
  • Any deprecations

• This is unlikely to impact how Ambassador performs at scale.

  Remember, things that might have an impact at scale include:

  • Any significant changes in memory use that might require adjusting the memory limits
  • Any significant changes in CPU use that might require adjusting the CPU limits
  • Anything that might change how many replicas users should use
  • Changes that impact data-plane latency/scalability

• My change is adequately tested.

  Remember when considering testing:

  • Your change needs to be specifically covered by tests.
    • Tests need to cover all the states where your change is relevant: for example, if you add a behavior that can be enabled or disabled, you'll need tests that cover the enabled case and tests that cover the disabled case. It's not sufficient just to test with the behavior enabled.
  • You also need to make sure that the entire area being changed has adequate test coverage.
    • If existing tests don't actually cover the entire area being changed, add tests.
    • This applies even for aspects of the area that you're not changing – check the test coverage, and improve it if needed!
  • We should lean on the bulk of code being covered by unit tests, but...
  • ... an end-to-end test should cover the integration points

• I updated DEVELOPING.md with any any special dev tricks I had to use to work on this code efficiently.

• The changes in this PR have been reviewed for security concerns and adherence to security best practices.

[LanceEa]

@superfrink - Thanks for the PR. When the CA Cert gets renewed it also needs to patch the CRD's which previously was only being done on startup or new CRD's applied to cluster.

I had actually been working on a larger fix to this and just landed it which ensures the CA Cert is renewed and that the CABundle is properly injected in the CRD's when it changes. It also opens up the ability manage it externally via CertManager (which has been requested in the past).

#5494

Please feel free to try that out and let me know if you see any issues!