Validating webhook does not pick up new certificate when renewed by certificate manager

[mshanmu]

https://github.com/flant/shell-operator/blob/main/pkg/webhook/server/server.go#L26

See above link, it looks like the webhook server loads the certificate file only once when the server starts and does not have any mechanism to pick up the new certificate and keep using the old (now invalidated) cert. Once it gets restarted the server picks up the correct certificate and things works again.

You could try using a certwatcher to fix this issue. For an example, you can check this PR kubeflow/kubeflow#6581

Expected behavior (what you expected to happen):
Works well even when certificate is changed

Actual behavior (what actually happened):
Saw the below error,
"failed to call webhook: post "":
x509: certificate has expired or not yet valid

Steps to reproduce:

1. Install cert-manager in k8s
2. Use self signed certificate with 5 days validity
3. After 5 days, notice the " x509: certificate has expired or not yet valid" failure

Environment:

• Shell-operator version: v1.0.12
• Kubernetes version: v1.24.2
• Installation type (kubectl apply, helm chart, etc.): helm chart

Anything else we should know?:

Additional information for debugging (if necessary):

Hook script

Logs

[nabokihms]

@mshanmu I confirm that this issue is valid. It will take us some time to fix, because it is not fully aligned with our current roadmap. Yet we are always willing to accept contributions!

[mshanmu]

Thanks @nabokihms !! Will try to send in a PR for this issue.