Bug 1913099: Handle keepalived hangs in liveness probe #2319
Conversation
Currently, if keepalived stops responding for any reason but the process is still there, we will never notice because the previous version of /tmp/keepalived.data will still be present in /tmp. This change writes the fault string to keepalived.data before doing the kill -s SIGUSR1 so that we assume failure unless keepalived overwrites the file as expected. A loop with a sleep is added to the check to give keepalived time to respond to the signal. If it takes longer than 5 seconds to write a non-fault status to the file then we will assume it is dead.
|
@openshift-cherrypick-robot: Bugzilla bug 1907671 has been cloned as Bugzilla bug 1913099. Retitling PR to link against new bug. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
openshift-ci-robot
changed the title
openshift-ci-robot
added
bugzilla/severity-high
|
@openshift-cherrypick-robot: This pull request references Bugzilla bug 1913099, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker. 6 validation(s) were run on this bug
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
@cybertron @celebdor ptal, was there a reason not to backport this to 4.6? |
|
/lgtm This should be fine to backport. |
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: cybertron, openshift-cherrypick-robot, sinnykumari The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci-robot
added
the
approved
|
/retest Please review the full test history for this PR and help us cut down flakes. |
12 similar comments
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
Why doesn't a skip prevent retests? Perhaps an override will work better for these tests that haven't passed in nearly a month. |
|
@sdodson: Overrode contexts on behalf of sdodson: ci/prow/okd-e2e-aws, ci/prow/okd-e2e-gcp-op, ci/prow/okd-e2e-upgrade In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
14 similar comments
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
russellb
added
the
cherry-pick-approved
|
/retest Please review the full test history for this PR and help us cut down flakes. |
russellb
removed
the
cherry-pick-approved
|
(patch manager) 4.6 blocker identified, so pulling unmerged patches from the queue |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
1 similar comment
|
/retest Please review the full test history for this PR and help us cut down flakes. |
crawford
changed the title
crawford
added
the
cherry-pick-approved
|
/retest Please review the full test history for this PR and help us cut down flakes. |
1 similar comment
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
@openshift-cherrypick-robot: All pull requests linked via external trackers have merged: Bugzilla bug 1913099 has been moved to the MODIFIED state. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
This is an automated cherry-pick of #1972
/assign sdodson