Values in CRDs for liveness/readiness/startup probes not reflecting after Helm upgrade

[citiatish]

We are trying to update the default failureThreshold, initialDelaySeconds and timeoutSeconds for the liveness, readiness, and startup probes in our CloudNativePG setup.

Here is the relevant section in crds.yaml we've updated:
probes:
description: |-
The configuration of the probes to be injected
in the PostgreSQL Pods.
properties:
liveness:
description: The liveness probe configuration
properties:
failureThreshold:
description: |-
Minimum consecutive failures for the probe to be considered failed after having succeeded.
Defaults to 3. Minimum value is 1.
format: int32
type: integer
default: 6

However, after upgrading the Helm chart, the new values are not reflected in the deployed Pods. The probes still show the old/default value of failureThreshold: 3.

Example pod details:
Liveness: http-get https://:8000/healthz delay=0s timeout=5s period=10s #success=1 #failure=3
Readiness: http-get https://:8000/readyz delay=0s timeout=5s period=10s #success=1 #failure=3
Startup: http-get https://:8000/healthz delay=0s timeout=5s period=10s #success=1 #failure=360

We’d like to understand:
• Is updating crds.yaml the correct approach to change default probe behavior?
• If yes, is there an additional step needed for these changes to take effect?
• Or is there an override that should be applied elsewhere (e.g. in a custom resource or chart values)?

[dosubot]

Hello @citiatish! 👋 I'm Dosu, here to assist you while you wait for a human maintainer. I can help you troubleshoot bugs, answer questions, and guide you on contributing. Let's tackle your issue together!

To ensure that changes to the default probe configurations are reflected in the deployed Pods after a Helm upgrade, updating the crds.yaml alone is not sufficient. Here are the steps and considerations:

1. Updating CRDs: Modifying the crds.yaml file updates the Custom Resource Definitions, but these changes need to be applied to the running instances. This typically involves a rolling update of the cluster to apply the new configurations. Ensure that the CRDs are properly applied to the Kubernetes cluster.

2. Restarting Pods: Changes to probe configurations require a restart of the Pods to take effect. When you upgrade the operator, a one-time rolling update of the cluster should be executed, which will apply the changes to the probes ^[1].

3. Helm Chart Values: To override default probe configurations, you should update the values.yaml file in the Helm chart. This is the recommended approach for setting custom values for probes. You can specify the desired values for failureThreshold, initialDelaySeconds, and other probe parameters in the values.yaml file under the ProbesConfiguration section ^[2].

4. Custom Resource Overrides: If you need specific configurations for different instances, consider using custom resources to override the default values specified in the Helm chart.

Ensure that after making these changes, you perform a Helm upgrade and verify that the Pods are restarted to reflect the new configurations.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[citiatish]

@dosu

We are not able to find ProbesConfiguration section in the values.yaml file of cloudnative-PG helm chart.

https://github.com/cloudnative-pg/charts/blob/main/charts/cloudnative-pg/values.yaml.

But we can see probe in the webhook section. Is it the same where have to define probing or what does it do under webhook?

[dosubot]

The ProbesConfiguration section is not present in the values.yaml file of the CloudNativePG Helm chart. Instead, probe configurations are defined under the webhook section. These probes are specifically for the webhook service, not for the PostgreSQL Pods themselves ^[1].

The webhook section includes configurations for liveness, readiness, and startup probes to ensure the webhook service is running, responsive, and ready to accept traffic. These settings do not affect the probe configurations for PostgreSQL Pods ^[1].

To customize probe configurations for PostgreSQL Pods, you would typically need to define them in the custom resource or directly in the Helm chart values if supported. If the Helm chart does not provide a direct way to configure these probes, you might need to look into other methods, such as modifying the custom resource definitions or using Kubernetes manifests to apply specific configurations.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[rbozan]

@citiatish Did you figure this out?

Personally I solved it by having a separate script executing this:

kubectl patch cluster cnpg-database-cluster --type='merge' -p='
{
  "spec": {
    "probes": {
      "readiness": {
        "timeoutSeconds": 30,
        "periodSeconds": 10,
        "failureThreshold": 3
      },
      "liveness": {
        "timeoutSeconds": 30,
        "periodSeconds": 10,
        "failureThreshold": 3
      },
      "startup": {
        "timeoutSeconds": 30,
        "periodSeconds": 10,
        "failureThreshold": 60
      }
    }
  }
}'