Setup Readiness and Liveness probe

[phisco]

What happened?

Crossplane currently doesn't have any liveness probe configured, with webhooks enabled this could lead to some calls being rejected at Crossplane startup, which is undesirable.

This is not really a big issue, as those calls would still fail on first installation, but could improve the upgrade process, as the new pod would only be considered ready once it's actually ready to serve traffic, properly leveraging Kubernetes rollout.

It would be nice if we could add probes to Crossplane's pod, see the conversation here for some hints at how to do that.

How can we reproduce it?

It's a timing issue, just spinning up Crossplane while applying any resource having a validating webhook, so either a Composition or CompositeResourceDefinition, could lead to some failure:

failed calling webhook "compositions.apiextensions.crossplane.io": failed to call webhook: Post "https://crossplane-webhooks.crossplane-system.svc:9443/validate-apiextensions-crossplane-io-v1-composition?timeout=10s": dial tcp 10.96.47.154:9443: connect: connection refused

What environment did it happen in?

Crossplane version: latest master, but most probably happening since 1.11

[negz]

FWIW I've hit the above as a flaky E2E test a few times now.

[phisco]

Fixed by #4748