[feat] Add prometheus config reloader livenessProbe & readinessProbe

[dongjiang1989]

Description

#4730
#3587
#5294

Type of change

What type of changes does your code introduce to the Prometheus operator? Put an x in the box that apply.

• CHANGE (fix or feature that would cause existing functionality to not work as expected)
• FEATURE (non-breaking change which adds functionality)
• BUGFIX (non-breaking change which fixes an issue)
• ENHANCEMENT (non-breaking change which improves existing functionality)
• NONE (if none of the other choices apply. Example, tooling, build system, CI, docs, etc.)

Changelog entry

Please put a one-line changelog entry below. This will be copied to the changelog file during the release process.

prometheus-config-reloader containers add livenessProbe & readinessProbe

[simonpasquier]

Out of curiosity, do you any concrete use case for adding liveness/readiness probes to the config reloader? In other words, did you face situations where probes would have helped?

One problem I can see with the current approach is that the next operator's version requires to use a config reloader image with the new health endpoint. This would make upgrades more complicated than needed IMHO.
To smooth the upgrade, we can introduce a new operator's flag to enable liveness/readiness probes for the config reloader (e.g. --enable-config-reloader-probes).

[dongjiang1989]

Out of curiosity, do you any concrete use case for adding liveness/readiness probes to the config reloader? In other words, did you face situations where probes would have helped?

One problem I can see with the current approach is that the next operator's version requires to use a config reloader image with the new health endpoint. This would make upgrades more complicated than needed IMHO. To smooth the upgrade, we can introduce a new operator's flag to enable liveness/readiness probes for the config reloader (e.g. --enable-config-reloader-probes).

THKS for U review! The Open Policy Agent is installed in the online kubernetes cluster. All containers in the pod must have livenessProbe or ReadinessProbe.

like：

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredProbes
metadata:
  name: must-have-probes
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
  parameters:
    probes: ["readinessProbe", "livenessProbe"]
    probeTypes: ["tcpSocket", "httpGet", "exec"]

add --enable-config-reloader-probes flag done

[simonpasquier]

The Open Policy Agent is installed in the online kubernetes cluster. All containers in the pod must have livenessProbe or ReadinessProbe.

Ack, I think that being an opt-in feature is the best approach because from past experience, liveness probes can create accidental problems (e.g. probe timeouts triggering container restarts).

[dongjiang1989]

The Open Policy Agent is installed in the online kubernetes cluster. All containers in the pod must have livenessProbe or ReadinessProbe.

Ack, I think that being an opt-in feature is the best approach because from past experience, liveness probes can create accidental problems (e.g. probe timeouts triggering container restarts).

I agree! It is very necessary to add --enable-config-reloader-probes flag. default value is false

[sthaha]

Thank you @dongjiang1989 , Looks good to me. Will defer to @simonpasquier 's to review and approve

[dongjiang1989]

@simonpasquier Please review and approve

[JoaoBraveCoding]

from my side lgtm