Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug 1974830: Update KubeDeploymentReplicasMismatch alert #1253

Merged
merged 1 commit into from Jun 30, 2021
Merged

Bug 1974830: Update KubeDeploymentReplicasMismatch alert #1253

merged 1 commit into from Jun 30, 2021

Conversation

philipgough
Copy link
Contributor

  • I added CHANGELOG entry for this change.
  • No user facing changes, so no entry in CHANGELOG was needed.

@openshift-ci openshift-ci bot requested review from arajkumar and jan--f June 28, 2021 12:12
@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 28, 2021
@philipgough philipgough changed the title Bz 1974830 Bug 1974830: Update KubeDeploymentReplicasMismatch alert Jun 28, 2021
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jun 28, 2021

@philipgough: This pull request references Bugzilla bug 1974830, which is invalid:

  • expected the bug to target the "4.9.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1974830: Update KubeDeploymentReplicasMismatch alert

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci openshift-ci bot added bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Jun 28, 2021
@paulfantom
Copy link
Contributor

/lgtm
/hold

Putting on hold to more allow testing in CI.

@openshift-ci openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 28, 2021
@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jun 28, 2021
@openshift-ci openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Jun 28, 2021
jan--f
jan--f approved these changes Jun 28, 2021
View reviewed changes
Copy link
Contributor

@jan--f jan--f left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm 👍

@philipgough
Copy link
Contributor Author

/bugzilla refresh

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jun 28, 2021

@philipgough: This pull request references Bugzilla bug 1974830, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.9.0) matches configured target release for branch (4.9.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @juzhao

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci openshift-ci bot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Jun 28, 2021
@openshift-ci openshift-ci bot requested a review from juzhao June 28, 2021 13:44
@philipgough
Update expression for 'control plane ready' metric to return binary v…
234f259 
…alue

The previous version of this expression returned either 1 or null which may have
caused issues with how Prometheus evaluated the expression and hence fired
the related alerts when it should not have.

alert:KubeDeploymentReplicasMismatch: Update alert to account for surges

This change updates the comparison operator in order to account for potential surges.
paulfantom
paulfantom reviewed Jun 29, 2021
View reviewed changes
jsonnet/rules.libsonnet
kube_deployment_spec_replicas{namespace=~"(openshift-.*|kube-.*|default|logging)",job="kube-state-metrics"}
!=
>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: let's make sure this goes upstream :)

@simonpasquier
Copy link
Contributor

/lgtm

After discussing offline with @pgough we understand why the former PR #1245 didn't work. cluster:control_plane:all_nodes_ready was returning 1 only when all control plane nodes were ready, otherwise it wouldn't return any data (e.g. Prometheus would insert stale markers). And calling scalar() on an instant vector which doesn't have exactly one element (e.g. empty) returns NaN. Thus the full expression ... * scalar(cluster:control_plane:all_nodes_ready)) != 0 returned NaN too which is enough to trigger the alert.

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jun 29, 2021
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jun 29, 2021

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jan--f, paulfantom, PhilipGough, simonpasquier

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [PhilipGough,jan--f,paulfantom,simonpasquier]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@paulfantom
Copy link
Contributor

/unhold

@openshift-ci openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 30, 2021
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

1 similar comment
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jun 30, 2021

@philipgough: The following test failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
ci/prow/e2e-aws-single-node 234f259 link /test e2e-aws-single-node

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-merge-robot openshift-merge-robot merged commit c25af1a into openshift:master Jun 30, 2021
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jun 30, 2021

@philipgough: All pull requests linked via external trackers have merged:

Bugzilla bug 1974830 has been moved to the MODIFIED state.

In response to this:

Bug 1974830: Update KubeDeploymentReplicasMismatch alert

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@wking wking mentioned this pull request Jul 1, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@simonpasquier simonpasquier simonpasquier left review comments

@paulfantom paulfantom paulfantom left review comments

@jan--f jan--f jan--f approved these changes

@arajkumar arajkumar Awaiting requested review from arajkumar

@juzhao juzhao Awaiting requested review from juzhao

Assignees

@simonpasquier simonpasquier

@paulfantom paulfantom

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@philipgough @paulfantom @simonpasquier @openshift-bot @jan--f @openshift-merge-robot