core: increase liveness probe timeout to 2s

[subhamkrai]

we have noticed multiple failures because of the probe
failing, most of the time it's due to fewer resources.
But increasing timeout fixes that, so increasing the
probe timeout to 2s from default 1s so that it will
give more time to probe before failing.

Signed-off-by: subhamkrai srai@redhat.com

Description of your changes:

Which issue is resolved by this Pull Request:
Resolves #

Checklist:

• Commit Message Formatting: Commit titles and messages follow guidelines in the developer guide).
• Skip Tests for Docs: If this is only a documentation change, add the label skip-ci on the PR.
• Reviewed the developer guide on Submitting a Pull Request
• Pending release notes updated with breaking and/or notable changes for the next minor release.
• Documentation has been updated, if necessary.
• Unit tests have been added, if necessary.
• Integration tests have been added, if necessary.

[travisn]

If the user overrides the liveness probe settings, this timeout can still be overridden, right?

[subhamkrai]

If the user overrides the liveness probe settings, this timeout can still be overridden, right?

I'll test and update but I think we should be able to override.

[subhamkrai]

If the user overrides the liveness probe settings, this timeout can still be overridden, right?

I'll test and update but I think we should be able to override.

I changed the mon deployment with timeout 5 and it was modified but when I restarted the operator, the timeout value changed to 2, and the respective ceph pod(in my case mon pod) restart and had timeout 2

[subhamkrai]

I think with the above observation, it will be problematic because somehow the operator restarts then it will bring the timeout to 2 and the respective ceph pod will restart but it will be in CLBO state due to liveness probe error. thoughts @travisn

[travisn]

If the user overrides the liveness probe settings, this timeout can still be overridden, right?

I'll test and update but I think we should be able to override.

I changed the mon deployment with timeout 5 and it was modified but when I restarted the operator, the timeout value changed to 2, and the respective ceph pod(in my case mon pod) restart and had timeout 2

This certainly sounds unexpected. You're sure all you did was restart the operator? How about trying a few variations to narrow down the repro:

1. Create a new cluster with the default timeout, then update the cluster CR with a new timeout
2. Restart the operator
3. Create a new cluster with the modified timeout
4. Restart the operator

At each step, see if the timeout is expected. It sounds like you already ran the first two steps. How about the other steps?

[subhamkrai]

If the user overrides the liveness probe settings, this timeout can still be overridden, right?

I'll test and update but I think we should be able to override.

I changed the mon deployment with timeout 5 and it was modified but when I restarted the operator, the timeout value changed to 2, and the respective ceph pod(in my case mon pod) restart and had timeout 2

This certainly sounds unexpected. You're sure all you did was restart the operator?
I created a cluster with 2s and modified deployment of mon to timeout 5s, restarted operator pod and a timeout was back to 2s.
How about trying a few variations to narrow down the repro:

1. Create a new cluster with the default timeout, then update the cluster CR with a new timeout
   default timeout 2s(not k8s default 1s)?
2. Restart the operator
3. Create a new cluster with the modified timeout
4. Restart the operator

At each step, see if the timeout is expected. It sounds like you already ran the first two steps. How about the other steps?

question: after step 2, you said creating a new cluster means deleting the cluster.yaml file and creating again?

[travisn]

question: after step 2, you said creating a new cluster means deleting the cluster.yaml file and creating again?

Per offline discussion, sounds like our confusion was about updating the cluster CR instead of editing the deployment manually. Sounds like the behavior was expected since the deployment was modified manually.

[BlaineEXE]

Seems good to me after the last discussion right?

[travisn]

Seems good to me after the last discussion right?

Agreed, will go ahead with it.