Webhook Can Enter Infinite Crashloop

[ian-mi]

Describe the bug
The eventing-webhook will continually crash with errors such as

{"level":"error","ts":"2020-09-29T21:44:49.120Z","logger":"eventing-webhook.sinkbindings.webhook.sources.knative.dev","caller":"controller/controller.go:528","msg":"Reconcile error","commit":"74b9bed","knative.dev/pod":"eventing-webhook-847f8cdc6d-cbftm","error":"secret "eventing-webhook-certs" is missing "ca-cert.pem" key","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20200922164940-4bf40ad82aab/controller/controller.go:528\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20200922164940-4bf40ad82aab/controller/controller.go:514\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20200922164940-4bf40ad82aab/controller/controller.go:451"}

http: TLS handshake error from 10.24.2.1:36304: server key missing

The eventing-webhook-certs secret remains unpopulated and does not appear to get reconciled by the webhook.

Expected behavior
The webhook certificate reconciler should populate the secret on startup.

To Reproduce
This occurs some small percentage of the time when applying the release yaml.

Knative release version
0.17.2

[ian-mi]

This scenario appears to occur when the webhook must obtain a previously held leader lease but the eventing-webhook-certs has not yet been reconciled. In this case the webhook will shutdown before the leader lease can be acquired and the cert can be reconciled.