Bug 1908958: Change ElasticsearchClusterNotHealthy to 7 minutes #600

sedroche · 2020-12-17T03:44:02Z

Description

ElasticsearchClusterNotHealthy is one of the noisier alerts on OSD. After reviewing this alert we found that this alert "self resolves" most commonly after 5 minutes.
Setting this to 7 minutes should relieve this.

/cc @alanconway
/assign @ewolinetz

Links

JIRA: https://issues.redhat.com/browse/OSD-5889
Bugzilla:
https://bugzilla.redhat.com/show_bug.cgi?id=1908958 4.7
https://bugzilla.redhat.com/show_bug.cgi?id=1908959 4.6

sedroche · 2020-12-17T07:00:35Z

/retest

alanconway

Do we have an issue to figure out why this alert is coming up so often? Increasing the timeout might be a reasonable work-around, but it sounds like we should evaluate whether the alert itself should be removed or modified, or whether it's indicating something we need to fix.

jcantrill · 2020-12-17T15:27:31Z

@sedroche you will need to create a bz for 4.7 and clone for 4.6 if the intention is to pull this change into these releases. It is the only way to satisfy the workflow

openshift-ci-robot · 2020-12-18T03:52:38Z

@sedroche: This pull request references Bugzilla bug 1908958, which is invalid:

expected the bug to target the "4.7.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1908958: Change ElasticsearchClusterNotHealthy to 7 minutes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sedroche · 2020-12-18T03:53:07Z

/bugzilla refresh

openshift-ci-robot · 2020-12-18T03:53:09Z

@sedroche: This pull request references Bugzilla bug 1908958, which is invalid:

expected the bug to target the "4.7.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sedroche · 2020-12-18T03:55:32Z

/bugzilla refresh

openshift-ci-robot · 2020-12-18T03:55:38Z

@sedroche: This pull request references Bugzilla bug 1908958, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.7.0) matches configured target release for branch (4.7.0)
bug is in the state NEW, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sedroche · 2020-12-18T03:58:51Z

@sedroche you will need to create a bz for 4.7 and clone for 4.6 if the intention is to pull this change into these releases. It is the only way to satisfy the workflow

@jcantrill I have created a bz for 4.7 and 4.6
https://bugzilla.redhat.com/show_bug.cgi?id=1908958
https://bugzilla.redhat.com/show_bug.cgi?id=1908959

openshift-ci-robot · 2020-12-18T03:59:17Z

@sedroche: This pull request references Bugzilla bug 1908958, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.7.0) matches configured target release for branch (4.7.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

Bug 1908958: Change ElasticsearchClusterNotHealthy to 7 minutes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2020-12-18T04:06:23Z

@sedroche: This pull request references Bugzilla bug 1908958, which is invalid:

expected the bug to target the "4.7.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1908958: Change ElasticsearchClusterNotHealthy to 7 minutes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sedroche · 2020-12-18T04:08:05Z

/bugzilla refresh

openshift-ci-robot · 2020-12-18T04:08:10Z

@sedroche: This pull request references Bugzilla bug 1908958, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.7.0) matches configured target release for branch (4.7.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sedroche · 2020-12-18T04:08:56Z

/test cluster-logging-operator-e2e

sedroche · 2020-12-18T04:10:08Z

/bugzilla refresh

openshift-ci-robot · 2020-12-18T04:10:12Z

@sedroche: This pull request references Bugzilla bug 1908958, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.7.0) matches configured target release for branch (4.7.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-merge-robot · 2020-12-18T07:09:40Z

@sedroche: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/cluster-logging-operator-e2e	`fbb197f`	link	`/test cluster-logging-operator-e2e`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

ewolinetz · 2021-01-04T19:27:20Z

/cc @blockloop @lukas-vlcek

ewolinetz · 2021-01-07T17:21:44Z

@sedroche this looks to need a rebase

sedroche · 2021-01-07T22:45:55Z

/retest

sedroche · 2021-01-08T04:22:08Z

/retest

sedroche · 2021-01-08T06:46:13Z

@ewolinetz rebased

ewolinetz · 2021-01-08T20:09:02Z

/lgtm

openshift-ci-robot · 2021-01-08T20:09:27Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alanconway, blockloop, ewolinetz, sedroche

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [alanconway,ewolinetz]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2021-01-08T22:51:17Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-01-09T01:39:33Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-01-09T01:52:32Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-ci-robot · 2021-01-09T03:56:37Z

@sedroche: All pull requests linked via external trackers have merged:

openshift/elasticsearch-operator#600

Bugzilla bug 1908958 has been moved to the MODIFIED state.

In response to this:

Bug 1908958: Change ElasticsearchClusterNotHealthy to 7 minutes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jcantrill · 2021-01-14T18:49:39Z

/cherrypick release-4.6

openshift-cherrypick-robot · 2021-01-14T18:50:06Z

@jcantrill: #600 failed to apply on top of branch "release-4.6":

Applying: Change ElasticsearchClusterNotHealthy to 7 minutes
Using index info to reconstruct a base tree...
M	files/prometheus_alerts.yml
Falling back to patching base and 3-way merge...
Auto-merging files/prometheus_alerts.yml
CONFLICT (content): Merge conflict in files/prometheus_alerts.yml
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 Change ElasticsearchClusterNotHealthy to 7 minutes
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherrypick release-4.6

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot assigned ewolinetz Dec 17, 2020

openshift-ci-robot requested a review from alanconway December 17, 2020 03:44

alanconway approved these changes Dec 17, 2020

View reviewed changes

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 17, 2020

sedroche changed the title ~~Change ElasticsearchClusterNotHealthy to 7 minutes~~ Bug 1908958: Change ElasticsearchClusterNotHealthy to 7 minutes Dec 18, 2020

openshift-ci-robot added the bugzilla/severity-unspecified Referenced Bugzilla bug's severity is unspecified for the PR. label Dec 18, 2020

openshift-ci-robot added the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Dec 18, 2020

openshift-ci-robot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Dec 18, 2020

openshift-ci-robot added bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. and removed bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. labels Dec 18, 2020

openshift-ci-robot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Dec 18, 2020

openshift-ci-robot added bugzilla/severity-low Referenced Bugzilla bug's severity is low for the branch this PR is targeting. and removed bugzilla/severity-unspecified Referenced Bugzilla bug's severity is unspecified for the PR. labels Dec 18, 2020

openshift-ci-robot requested review from blockloop and lukas-vlcek January 4, 2021 19:27

blockloop approved these changes Jan 7, 2021

View reviewed changes

sedroche added 2 commits January 8, 2021 08:39

Change ElasticsearchClusterNotHealthy to 7 minutes

7ca1656

Fix test

40a68d8

sedroche force-pushed the refine-ElasticsearchClusterNotHealthy branch from fbb197f to 40a68d8 Compare January 7, 2021 22:42

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jan 8, 2021

openshift-merge-robot merged commit a7e01c2 into openshift:master Jan 9, 2021

jcantrill mentioned this pull request Jan 14, 2021

Bug 1908959: Change ElasticsearchClusterNotHealthy to 7 minutes #622

Merged

Bug 1908958: Change ElasticsearchClusterNotHealthy to 7 minutes #600

Bug 1908958: Change ElasticsearchClusterNotHealthy to 7 minutes #600

Uh oh!

Conversation

sedroche commented Dec 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Links

Uh oh!

sedroche commented Dec 17, 2020

Uh oh!

alanconway left a comment

Choose a reason for hiding this comment

Uh oh!

jcantrill commented Dec 17, 2020

Uh oh!

openshift-ci-robot commented Dec 18, 2020

Uh oh!

sedroche commented Dec 18, 2020

Uh oh!

openshift-ci-robot commented Dec 18, 2020

Uh oh!

sedroche commented Dec 18, 2020

Uh oh!

openshift-ci-robot commented Dec 18, 2020

Uh oh!

sedroche commented Dec 18, 2020

Uh oh!

openshift-ci-robot commented Dec 18, 2020

Uh oh!

openshift-ci-robot commented Dec 18, 2020

Uh oh!

sedroche commented Dec 18, 2020

Uh oh!

openshift-ci-robot commented Dec 18, 2020

Uh oh!

sedroche commented Dec 18, 2020

Uh oh!

sedroche commented Dec 18, 2020

Uh oh!

openshift-ci-robot commented Dec 18, 2020

Uh oh!

openshift-merge-robot commented Dec 18, 2020

Uh oh!

ewolinetz commented Jan 4, 2021

Uh oh!

ewolinetz commented Jan 7, 2021

Uh oh!

sedroche commented Jan 7, 2021

Uh oh!

sedroche commented Jan 8, 2021

Uh oh!

sedroche commented Jan 8, 2021

Uh oh!

ewolinetz commented Jan 8, 2021

Uh oh!

openshift-ci-robot commented Jan 8, 2021

Uh oh!

openshift-bot commented Jan 8, 2021

Uh oh!

openshift-bot commented Jan 9, 2021

Uh oh!

openshift-bot commented Jan 9, 2021

Uh oh!

openshift-ci-robot commented Jan 9, 2021

Uh oh!

jcantrill commented Jan 14, 2021

Uh oh!

openshift-cherrypick-robot commented Jan 14, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

sedroche commented Dec 17, 2020 •

edited

Loading