🐛 Revert: move health probes to runnable

[sbueringer]

This PR reverts part of: #2275

On main we have the following sequence in controllerManager.Start():

• ...
• add health probe server
• ...
• Start webhook runnables
• Start cache runnables
• Start other runnables (including health probe server)

Usually this works except under the following circumstances (concrete example):

• The controller implements a reconciler for a CRD called MachineDeployment
• MachineDeployment has multiple apiVersions and thus the controller also implements a conversion webhook
• There are MachineDeployments in the older apiVersion stored in etcd

Now we have the following problem:

• Start cache runnables fails because caches are not getting ready
• Caches are not getting ready because MachineDeployment list does not work
• MachineDeployment list does not work because the conversion webhook does not work ("Post "https://capi-webhook-service.capi-system.svc:443/convert?timeout=30s\": dial tcp 10.142.135.97:443: connect: connection refused")
• conversion webhook does not work because the service/pod does not get ready
• Pod does not get ready because health probes are not started
• health probes are not started because cache does not get ready

tl;dr cyclic dependency between start caches and start runnables.

This PR solves this by immediately starting the health probes again.

[sbueringer]

/assign @vincepri @alvaroaleman

/hold
I'll verify with Cluster API if the deadlock is gone before merge

[sbueringer]

/hold cancel

Ran the Cluster API test with the CR version from this PR locally and everything works as expected again.

[sbueringer]

(cc just fyi @zqzten, we're reverting this part for now to ensure CR main works again, we can work on a new implementation independent of that)

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sbueringer, vincepri

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [sbueringer,vincepri]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[zqzten]

I think we can introduce a new "basics" runnable group which includes the previous internal servers and start it before any other runnable groups, this seems to solve the dead lock issue here. wdyt @sbueringer

[sbueringer]

I think we can introduce a new "basics" runnable group which includes the previous internal servers and start it before any other runnable groups, this seems to solve the dead lock issue here. wdyt @sbueringer

Yup, something like this would work.