core: increase liveness probe timeout to 5s

[eljabsheh]

stability issues have been observed with 1s.
socket latency is expected whenever CPUs are
under minor pressure. Increasing value to
5s should cover most small-medium scale envs.

Resolves BZ: 2126566

Signed-off-by: Randy J. Martinez randy@cephtips.com

Description of your changes:

Which issue is resolved by this Pull Request:
Resolves #

Checklist:

• Commit Message Formatting: Commit titles and messages follow guidelines in the developer guide).
• Skip Tests for Docs: If this is only a documentation change, add the label skip-ci on the PR.
• Reviewed the developer guide on Submitting a Pull Request
• Pending release notes updated with breaking and/or notable changes for the next minor release.
• Documentation has been updated, if necessary.
• Unit tests have been added, if necessary.
• Integration tests have been added, if necessary.

[subhamkrai]

LGTM!

one small question:
default is 1s and making it 5s is a bit longer but if it makes it more stable then good. have you seen 5s giving more stability? @randymtz

[subhamkrai]

LGTM!

one small question: default is 1s and making it 5s is a bit longer but if it makes it more stable then good. have you seen 5s giving more stability? @randymtz

never minds. Seems like this was discussed in yesterday call.

[travisn]

As discussed, 5s could give more stability and less aggressive to restart the OSDs that are resource-starved, although it is still arbitrary.
@randymtz Do you have a cluster where you could observe that 2s was causing the OSDs to be restarted, but 5s was more stable? Perhaps we should also increase the failureThreshold which defaults to 3.

[eljabsheh]

As discussed, 5s could give more stability and less aggressive to restart the OSDs that are resource-starved, although it is still arbitrary. @randymtz Do you have a cluster where you could observe that 2s was causing the OSDs to be restarted, but 5s was more stable? Perhaps we should also increase the failureThreshold which defaults to 3.

update: Awaiting feedback in BZ from linked-case to gauge 5s effectiveness. Once received, we'll be able to confirm if failureThreshold requires update as well.

note: Some OSDs were reporting CLBO post livenessProbe update. After investigating, the env did not have PR#10250 value update. I provided patch and am awaiting feedback for that as well.

[travisn]

The BZ has the feedback that 5s is working better, let's go ahead with this.