Skip to content

Conversation

@yithian
Copy link
Contributor

@yithian yithian commented Dec 23, 2020

Description

Includes a warning-level alert for when the elasticsearch-operator CSV's phase is not "success". ie: when csv_success{name =~ "elasticsearch-operator.*"} == 0.

We (OpenShift SRE-P) will configure alertmanager to suppress our alert-level alerts for ElasticsearchClusterNotHealthy while this alert is active.

/cc @alanconway
/assign @ewolinetz

Links

@yithian
Copy link
Contributor Author

yithian commented Dec 23, 2020

obsoletes #585

@yithian
Copy link
Contributor Author

yithian commented Dec 23, 2020

/hold

until after change freeze

@openshift-ci-robot openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 23, 2020

# --------- ElasticsearchCSVNotSuccessful ---------
- eval_time: 10m
alertname: ElasticsearchOperatorCSVNotSuccessful
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would "ElasticSearchNotReconciled" be more descriptive from a user perspective?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think its descriptive enough, we also would be pointing to documentation from the message of the alert (in the future)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for what it's worth, i don't have any attachment to ElasticsearchOperatorCSVNotSuccessful as an alertname. if we want to change it, i'm happy to do so

@ewolinetz
Copy link
Contributor

ewolinetz commented Jan 4, 2021

/cc @blockloop @lukas-vlcek

@ewolinetz
Copy link
Contributor

/approve

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 4, 2021
"message": "Elasticsearch Operator CSV has not reconciled succesfully."
"summary": "Elasticsearch Operator CSV Not Successful"
"expr": |
csv_succeeded{name =~ "elasticsearch-operator.*"} == 0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this expression simply check the status of the csv to denote if it was successful?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

iirc, csv_succeeded == 0 is the failed phase

@ewolinetz
Copy link
Contributor

/lgtm
I'll create a 4.7 and link it here so that it can be merged in

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jan 6, 2021
@openshift-ci-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ewolinetz, yithian

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ewolinetz ewolinetz changed the title configure a warning level alert for when the CSV phase isn't success Bug 1913464: configure a warning level alert for when the CSV phase isn't success Jan 6, 2021
@openshift-ci-robot openshift-ci-robot added the bugzilla/severity-unspecified Referenced Bugzilla bug's severity is unspecified for the PR. label Jan 6, 2021
@openshift-ci-robot
Copy link

@yithian: This pull request references Bugzilla bug 1913464, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.7.0) matches configured target release for branch (4.7.0)
  • bug is in the state NEW, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

Bug 1913464: configure a warning level alert for when the CSV phase isn't success

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label Jan 6, 2021
@ewolinetz
Copy link
Contributor

/hold cancel

@openshift-ci-robot openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 7, 2021
@ewolinetz
Copy link
Contributor

/bugzilla refresh

@openshift-ci-robot openshift-ci-robot added bugzilla/severity-low Referenced Bugzilla bug's severity is low for the branch this PR is targeting. and removed bugzilla/severity-unspecified Referenced Bugzilla bug's severity is unspecified for the PR. labels Jan 7, 2021
@openshift-ci-robot
Copy link

@ewolinetz: This pull request references Bugzilla bug 1913464, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.7.0) matches configured target release for branch (4.7.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

2 similar comments
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-merge-robot openshift-merge-robot merged commit 81ea004 into openshift:master Jan 8, 2021
@openshift-ci-robot
Copy link

@yithian: All pull requests linked via external trackers have merged:

Bugzilla bug 1913464 has been moved to the MODIFIED state.

In response to this:

Bug 1913464: configure a warning level alert for when the CSV phase isn't success

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@yithian yithian deleted the OSD-5685_warning_alert_csv_succeeded branch January 11, 2021 19:25
@RiRa12621
Copy link

@ewolinetz can we backport this to 4.5?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/severity-low Referenced Bugzilla bug's severity is low for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants