Bug 1999585: [release-4.8] add alert for machine with long deletion phase #908
Conversation
This change adds the MachineNotYetDeleted alert for a machine with phase "Deleting" for longer than 360 minutes, at "warning" severity. Also updates the alerts documentation with new information. The MachineWithNoRunningPhase alert has also been modified to exclude machines with the "Deleting" phase to ensure that we do not double fire an alert for this condition. ref: https://issues.redhat.com/browse/OCPCLOUD-921
…ineWithNoRunningPhase b13dbe9 (add alert for machine with long deletion phase, 2021-02-22, openshift#807) added "|Deleting". But to move from "not equal" to "regexp does not match" we need to match that with !~.
|
@vrutkovs: No Bugzilla bug is referenced in the title of this pull request. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
vrutkovs
changed the title
|
@vrutkovs: An error was encountered querying GitHub for users with public email (zhsun@redhat.com) for bug 1999585 on the Bugzilla server at https://bugzilla.redhat.com. No known errors were detected, please see the full error message for details. Full error message.
non-200 OK status code: 403 Forbidden body: "{\n \"documentation_url\": \"https://docs.github.com/en/free-pro-team@latest/rest/overview/resources-in-the-rest-api#secondary-rate-limits\",\n \"message\": \"You have exceeded a secondary rate limit. Please wait a few minutes before you try again.\"\n}\n"
Please contact an administrator to resolve this issue, then request a bug refresh with In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: JoelSpeed The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
the
approved
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
2 similar comments
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/bugzilla refresh |
openshift-ci
bot
added
bugzilla/severity-high
|
@vrutkovs: This pull request references Bugzilla bug 1999585, which is valid. 6 validation(s) were run on this bug
Requesting review from QA contact: In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
8 similar comments
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
14 similar comments
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
mfojtik
removed
the
cherry-pick-approved
|
patchmanager: this looks to be permanently broken and retesting >150 times. @vrutkovs can you please investigate? |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
1 similar comment
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
we discovered an edge case regression with this alert, in situations where the machine-api-operator is moved or rescheduled the time series associated with the long deletion phase gets reset. the reset can cause our tests to fail, we have added #913 to address the issue, perhaps we need to form the cherry-pick as the sum of both PRs? |
Maybe it would be best to keep them separated, so that we wouldn't mix bug cherry-pick rules? Bringing this commit it may make us seek a new cherrypick request. @mfojtik |
|
[patch-manager] Approving cherry pick with score 0.5 - previously perma-failing test now passed so re-approving |
hardys
added
the
cherry-pick-approved
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
4 similar comments
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
@vrutkovs: All pull requests linked via external trackers have merged: Bugzilla bug 1999585 has been moved to the MODIFIED state. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Cherry-pick of #807 and #878, as CCX has noticed an uptick in MachineWithoutValidNode alerts on vsphere clusters recently