Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug 1932624: jsonnet/rules,pkg/operator: use gauge based reconcilation state metrics and alerting #1078

Merged
merged 2 commits into from Mar 27, 2021
Merged

Bug 1932624: jsonnet/rules,pkg/operator: use gauge based reconcilation state metrics and alerting #1078

merged 2 commits into from Mar 27, 2021

Conversation

s-urbaniak
Copy link
Contributor

/cc @openshift/openshift-team-monitoring

@simonpasquier does that reflect what you had in mind?

  • I added CHANGELOG entry for this change.
  • No user facing changes, so no entry in CHANGELOG was needed.

@openshift-ci-robot openshift-ci-robot requested a review from a team March 11, 2021 13:41
@openshift-ci-robot
Copy link
Contributor

@s-urbaniak: This pull request references Bugzilla bug 1932624, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.8.0) matches configured target release for branch (4.8.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @juzhao

In response to this:

Bug 1932624: jsonnet/rules,pkg/operator: use gauge based reconcilation state metrics and alerting

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. labels Mar 11, 2021
@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 11, 2021
@s-urbaniak
Copy link
Contributor Author

@simonpasquier @paulfantom ptal

@s-urbaniak
Copy link
Contributor Author

/test e2e-agnostic-upgrade

@s-urbaniak
Copy link
Contributor Author

e2e test seems to be unrelated 🤔

lilic
lilic reviewed Mar 17, 2021
View reviewed changes
Copy link
Contributor

@lilic lilic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

Seems like all comments were addressed. 🎉
/hold feel free to remove hold when ready to merge

@openshift-ci-robot openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 17, 2021
@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Mar 17, 2021
@s-urbaniak
Copy link
Contributor Author

/test e2e-agnostic-upgrade
/hold cancel

@openshift-ci-robot openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 22, 2021
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

8 similar comments
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@s-urbaniak
Copy link
Contributor Author

/hold

@openshift-ci-robot openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 23, 2021
@s-urbaniak
Copy link
Contributor Author

The upgrade failures seem to be legit:

alert ClusterMonitoringOperatorReconciliationErrors fired for 210 seconds with labels: {container="kube-rbac-proxy", endpoint="https", instance="10.128.0.79:8443", job="cluster-monitoring-operator", namespace="openshift-monitoring", pod="cluster-monitoring-operator-677f86d474-hwv4d", service="cluster-monitoring-operator", severity="warning"}

paulfantom
paulfantom reviewed Mar 23, 2021
View reviewed changes
jsonnet/rules.libsonnet Outdated Show resolved Hide resolved
@s-urbaniak
jsonnet/rules,pkg/operator: use gauge based reconcilation state metri…
a6a8c67 
…cs and alerting.

This improves the current metric by replacing it with a gauge based one.
The referencing alert and its description are changed.
@openshift-ci-robot openshift-ci-robot removed the lgtm Indicates that a PR is ready to be merged. label Mar 24, 2021
@s-urbaniak
Copy link
Contributor Author

ok, let's see what e2e has to say 🤞

@simonpasquier
Copy link
Contributor

/lgtm

@simonpasquier
Copy link
Contributor

/retest

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Mar 25, 2021
@openshift-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lilic, s-urbaniak, simonpasquier

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [lilic,s-urbaniak,simonpasquier]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@s-urbaniak
Copy link
Contributor Author

/retest

@s-urbaniak
Copy link
Contributor Author

/hold cancel
/retest

@openshift-ci-robot openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 26, 2021
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

3 similar comments
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Mar 27, 2021 •
edited

@s-urbaniak: The following test failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
ci/prow/e2e-aws-single-node 84da526 link /test e2e-aws-single-node

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-merge-robot openshift-merge-robot merged commit 26125c1 into openshift:master Mar 27, 2021
@openshift-ci-robot
Copy link
Contributor

@s-urbaniak: All pull requests linked via external trackers have merged:

Bugzilla bug 1932624 has been moved to the MODIFIED state.

In response to this:

Bug 1932624: jsonnet/rules,pkg/operator: use gauge based reconcilation state metrics and alerting

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@s-urbaniak s-urbaniak deleted the bz-1932624 branch March 31, 2021 09:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@simonpasquier simonpasquier simonpasquier left review comments

@paulfantom paulfantom paulfantom left review comments

@lilic lilic lilic left review comments

@juzhao juzhao Awaiting requested review from juzhao

Assignees

@simonpasquier simonpasquier

@lilic lilic

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
7 participants
@s-urbaniak @openshift-ci-robot @openshift-bot @simonpasquier @paulfantom @lilic @openshift-merge-robot