Istio sidecar-injector mutatingwebhookconfiguration certifcate does not auto-renew

[Stono]

Bug description
The certificate in the mutatingwebhookconfiguration of the sidecar-injector caBundle expired, and did not auto renew.

Internal error occurred: failed calling webhook "sidecar-injector.istio.io": Post https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s: x509: certificate has expired or is not yet valid

Following on from #14516 - when we were trying to get services back online, pods were unable to start due to the certificate on the injector being invalid.

This compounded an already bad day.

Affected product area (please put an X in all that apply)

[ ] Configuration Infrastructure
[x] Docs
[x] Installation
[x] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[x] Security
[ ] Test and Release
[x] User Experience

Expected behaviour
I expect the sidecar injector to manage it's caBundle appropriately.

Steps to reproduce the bug
set caBundle in the mutatingwebhookconfiguration of the sidecar-injector to be out of date

Version (include the output of istioctl version --remote and kubectl version)
All

How was Istio installed?
Helm

[duderino]

Setting to P0 since this exacerbated the recovery from #14516

[myidpt]

Working on it. Using Kube CA to generate certs for the webhook solves this issue. But it will come out in 1.3.

In 1.2, we will change the root cert TTL to 10 years or beyond for now, and we will make a transition support plan and may cherrypick a root cert rotation PR.

[Stono]

caBundle gets auto generated now (eg if I set caBundle:'' in the secret and restart the injector it regenerates correctly.

It seems like a small piece of work to add a simple check that uses that same code to refresh the secret if it's due soon?

[myidpt]

@Stono I think you are right. I'll see how to do the check automatically.

[myidpt]

@Stono
Some update about our findings.
@howardjohn and I have tested and checked the code (John found it although I didn't get a chance to locate it) that the webhooks will update the mutating/validatingwebhookconfiguration after they restart, with the new root cert. We have tested that after the root transition, the update can happen with the upgrade to 1.1.8, which restarts the webhooks to overwrite the old configurations.

[Stono]

Hey!
Thanks very much for testing and confirming that @myidpt @howardjohn.
Really not sure what happened in my situation as I did have to edit the caBundle. Perhaps it was because I didn't time it right (eg citadel hadn't managed to update the ca secret for the injector at the point I tried).

I think this issue should still be used to capture the work required to not need to restart the injector when the CA bundle is updated though; it should watch the mounted certs and just handle it (like pilot-agent).

[myidpt]

Sorry for the delay. I have just validated that the rotated root certificate will trigger an update on the caBundle for both validating and mutating webhook configurations, without the need to restart galley or the sidecar-injector.
Closing this issue.