ControllerManager/DefaultController error handling

[olegsavin]

Describe the bug
ControllerManager/DefaultController do not throw/retry if any of controllers fails preFlightCheck.
DefaultController silently exits run() func, ControllerManager doesn't do anything about it, doesn't re-try or throw also.
There is no way for outside consumer to figure out if all controllers have been started fine or not and retry if not.

Client Version
18

Kubernetes Version
1.26

Java Version
Java 11

To Reproduce
Hard repro, API server REST call timeout caused this.

Expected behavior
Client/consumer of ControllerManager should be able to specify what happens in case not all controllers start successfully - best effort or hard fail ControllerManager.run(). Expose running status of controllers. Exception in preFlightCheck should be propagated to client or client is able to provide custom handler - for individual controller and combined all failures of controllers registered in ControllerManager,

[brendandburns]

We'd be happy to take PRs to improve the behavior here.

[kvmw]

We ran into the same issue, because one of our controllers was registered to list/watch a CRD type which was not there yet. The CRD was provided by another deployment and it just happened that our deployment was faster than the other one and the controller informer was trying to list the CRD type which was not there yet.

A simple way to reproduce this issue is by making sure that readyFuncs always return false.
If readyFuncs does not return true in 30 seconds (default timeout), pre-flight check fails and controller exits run() function without any error or failure signal.

Exposing the status of the controllers (including ControllerManager) would be sufficient to solve the issue. The status can be used in probe endpoints to expose the issue and cause pod restart if needed.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.