Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[release-4.14] OCPBUGS-31428: CEO aliveness check should only detect deadlocks #1231

Merged
merged 1 commit into from Mar 28, 2024
Merged

[release-4.14] OCPBUGS-31428: CEO aliveness check should only detect deadlocks #1231

merged 1 commit into from Mar 28, 2024

Conversation

openshift-cherrypick-robot
Copy link

This is an automated cherry-pick of #1225

/assign tjungblu

@tjungblu
OCPBUGS-30873: CEO aliveness check should only detect deadlocks
4517a47 
Currently we only detect whether a controller has been running
continuously into errors. Whereas we wanted to detect real deadlock
situations. This change defuses the aliveness check to only declare real
locking situations as problematic.

Additionally, to not create insane amounts of log traffic, this change
will throttle the stack dumping to once every 15 minutes. Previously it
would trigger almost immediately every health probe invocation and
create multi-megabyte log spam.

Signed-off-by: Thomas Jungblut <tjungblu@redhat.com>
@openshift-cherrypick-robot openshift-cherrypick-robot mentioned this pull request Mar 27, 2024
Merged
@openshift-ci-robot
Copy link

@openshift-cherrypick-robot: Jira Issue OCPBUGS-30915 has been cloned as Jira Issue OCPBUGS-31428. Will retitle bug to link to clone.
/retitle [release-4.14] OCPBUGS-31428: CEO aliveness check should only detect deadlocks

In response to this:

This is an automated cherry-pick of #1225

/assign tjungblu

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot changed the title [release-4.14] OCPBUGS-30915: CEO aliveness check should only detect deadlocks [release-4.14] OCPBUGS-31428: CEO aliveness check should only detect deadlocks Mar 27, 2024
@openshift-ci-robot openshift-ci-robot added jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Mar 27, 2024
@openshift-ci-robot
Copy link

@openshift-cherrypick-robot: This pull request references Jira Issue OCPBUGS-31428, which is invalid:

  • expected dependent Jira Issue OCPBUGS-30915 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is MODIFIED instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

This is an automated cherry-pick of #1225

/assign tjungblu

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Mar 27, 2024

@openshift-cherrypick-robot: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws-etcd-recovery 4517a47 link false /test e2e-aws-etcd-recovery
ci/prow/e2e-gcp-qe-no-capabilities 4517a47 link false /test e2e-gcp-qe-no-capabilities

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@tjungblu
Copy link
Contributor

/jira refresh

@openshift-ci-robot openshift-ci-robot added the jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. label Mar 27, 2024
@openshift-ci-robot
Copy link

@tjungblu: This pull request references Jira Issue OCPBUGS-31428, which is valid. The bug has been moved to the POST state.

6 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.14.z) matches configured target version for branch (4.14.z)
  • bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
  • dependent bug Jira Issue OCPBUGS-30915 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-30915 targets the "4.15.z" version, which is one of the valid target versions: 4.15.0, 4.15.z
  • bug has dependents

Requesting review from QA contact:
/cc @geliu2016

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot removed the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label Mar 27, 2024
@openshift-ci openshift-ci bot requested a review from geliu2016 March 27, 2024 11:28
@dusk125
Copy link
Contributor

dusk125 commented Mar 27, 2024

/lgtm
/label backport-risk-assessed

@openshift-ci openshift-ci bot added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label Mar 27, 2024
@openshift-ci openshift-ci bot added lgtm Indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Mar 27, 2024
geliu2016
geliu2016 approved these changes Mar 28, 2024
View reviewed changes
Copy link

@geliu2016 geliu2016 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/label cherry-pick-approved

@openshift-ci openshift-ci bot added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Mar 28, 2024
@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Mar 28, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dusk125, geliu2016, openshift-cherrypick-robot

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@geliu2016
Copy link

/label qe-approved

@openshift-ci openshift-ci bot added the qe-approved Signifies that QE has signed off on this PR label Mar 28, 2024
@openshift-ci-robot
Copy link

@openshift-cherrypick-robot: This pull request references Jira Issue OCPBUGS-31428, which is valid.

6 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.14.z) matches configured target version for branch (4.14.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • dependent bug Jira Issue OCPBUGS-30915 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-30915 targets the "4.15.z" version, which is one of the valid target versions: 4.15.0, 4.15.z
  • bug has dependents

Requesting review from QA contact:
/cc @geliu2016

In response to this:

This is an automated cherry-pick of #1225

/assign tjungblu

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot requested a review from geliu2016 March 28, 2024 02:02
@openshift-merge-bot openshift-merge-bot bot merged commit 74f5363 into openshift:release-4.14 Mar 28, 2024
16 of 18 checks passed
@openshift-ci-robot
Copy link

@openshift-cherrypick-robot: Jira Issue OCPBUGS-31428: All pull requests linked via external trackers have merged:

Jira Issue OCPBUGS-31428 has been moved to the MODIFIED state.

In response to this:

This is an automated cherry-pick of #1225

/assign tjungblu

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-bot
Copy link
Contributor

[ART PR BUILD NOTIFIER]

This PR has been included in build cluster-etcd-operator-container-v4.14.0-202403272311.p0.g74f5363.assembly.stream.el8 for distgit cluster-etcd-operator.
All builds following this will include this PR.

@openshift-merge-robot
Copy link
Contributor

Fix included in accepted release 4.14.0-0.nightly-2024-03-28-071905

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Elbehery Elbehery Awaiting requested review from Elbehery

@hasbro17 hasbro17 Awaiting requested review from hasbro17

@geliu2016 geliu2016 Awaiting requested review from geliu2016

Assignees

@dusk125 dusk125

@tjungblu tjungblu

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. qe-approved Signifies that QE has signed off on this PR
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
7 participants
@openshift-cherrypick-robot @openshift-ci-robot @tjungblu @dusk125 @geliu2016 @openshift-bot @openshift-merge-robot