Add alert for no routes configured in alertmanager #585

paulfantom · 2019-12-09T17:14:49Z

Follow up to #554

Requested in #554 (comment)
Requested in https://coreos.slack.com/archives/C0VMT03S5/p1575909697091600

Fixed recording rule name as requested in #554 (comment)

/cc @s-urbaniak @smarterclayton

jsonnet/rules.jsonnet

s-urbaniak · 2019-12-09T17:37:13Z

/lgtm

openshift-bot · 2019-12-09T18:00:01Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-12-09T18:12:59Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-12-09T19:17:54Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-12-09T19:30:56Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-12-09T20:36:08Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-12-09T22:33:22Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-12-10T00:55:42Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-12-10T01:21:54Z

/retest

Please review the full test history for this PR and help us cut down flakes.

smarterclayton · 2019-12-10T01:33:01Z

/hold

I expected the e2e test (only 2 firing alerts) to fail.

simonpasquier · 2019-12-10T08:08:05Z

jsonnet/rules.jsonnet

+            expr: 'cluster:alertmanager_routing_enabled:max > 0',
+            alert: 'AlertmanagerReceiversNotConfigured',
+            annotations: {
+              message: 'Alerts are not configured to be sent to a notification system. Configuring an alert route will allow you to be notified when important failures occur.',


"Configuring an alert route" -> "Configuring a notification receiver"?

I think you need at least one route configured to send alerts and while configuring route you need to configure any receiver either way.

The confusion for me is that there's always at least one route in the routing tree (the top-level route) but this isn't really important. As @lilic commented, it might be good to provide guidance. What about?

Alerts are not configured to be sent to a notification system, meaning that you may not be notified in a timely fashion when important failures occur. Check the OpenShift documentation to learn how to configure notifications with Alertmanager.

Ideally we would point to a playbook, but the above msg sgtm. 👍

paulfantom · 2019-12-10T09:02:09Z

/retest

smarterclayton · 2019-12-12T18:23:56Z

Can you rebase?

paulfantom · 2019-12-16T08:51:14Z

rebased
/unhold

paulfantom · 2019-12-16T08:52:30Z

/hold cancel

e2e shouldn't fail since openshift/origin#24276 is merged

paulfantom · 2019-12-19T15:27:30Z

Rebased, checked if alert severity is taken into consideration when rendering alert in openshift console (doesn't matter), changed alert message.

lilic

/approve

this lgtm, letting @smarterclayton have a final look

s-urbaniak · 2020-01-06T10:19:09Z

ping @smarterclayton for a final review pass.

lilic · 2020-01-13T12:33:04Z

@paulfantom lets just rebase and merge this, if Clayton wants something changed we can do that afterwards as well, just so we don't forget to ship this. SGTY?

…cal alert when no routes are configured

paulfantom · 2020-01-14T22:47:23Z

Rebased. Let's merge it asap as it seems people are under impression that this already landed (https://coreos.slack.com/archives/C0VMT03S5/p1579040352101100). We can later tweak messages as those are less important than telemetry data this provides.

/cc @lilic @s-urbaniak

smarterclayton · 2020-01-15T00:57:13Z

Warning is fine, console can deliberately elevate this with more priority

/cherry-pick release-4.3

^ since all the other pieces are in

openshift-cherrypick-robot · 2020-01-15T00:57:14Z

@smarterclayton: once the present PR merges, I will cherry-pick it on top of release-4.3 in a new PR and assign it to you.

In response to this:

Warning is fine, console can deliberately elevate this with more priority

/cherry-pick release-4.3

^ since all the other pieces are in

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

lilic

/lgtm

openshift-ci-robot · 2020-01-15T06:01:49Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: LiliC, paulfantom, s-urbaniak

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [LiliC,paulfantom,s-urbaniak]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2020-01-15T06:42:27Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-01-15T06:55:30Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-01-15T07:09:02Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-cherrypick-robot · 2020-01-15T08:19:53Z

@smarterclayton: #585 failed to apply on top of branch "release-4.3":

error: Failed to merge in the changes.
Using index info to reconstruct a base tree...
M	jsonnet/rules.jsonnet
Falling back to patching base and 3-way merge...
Auto-merging jsonnet/rules.jsonnet
CONFLICT (content): Merge conflict in jsonnet/rules.jsonnet
Patch failed at 0001 jsonnet: fix alertmanager recording rule name; add alertmanager critical alert when no routes are configured

In response to this:

Warning is fine, console can deliberately elevate this with more priority

/cherry-pick release-4.3

^ since all the other pieces are in

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

smarterclayton · 2020-01-23T17:51:59Z

Hrm, I’m not seeing this be reported at all:

sort_desc(count by (alertname) (count by (_id,alertname) (alerts{alertstate="firing",alertname="AlertmanagerReceiversNotConfigured"})))

This should be firing on a default cluster, is it not?

openshift-ci-robot requested review from s-urbaniak and smarterclayton December 9, 2019 17:14

openshift-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Dec 9, 2019

smarterclayton reviewed Dec 9, 2019

View reviewed changes

jsonnet/rules.jsonnet Outdated Show resolved Hide resolved

paulfantom force-pushed the route_metric_alert branch from d2ef574 to b518d04 Compare December 9, 2019 17:21

openshift-ci-robot assigned s-urbaniak Dec 9, 2019

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Dec 9, 2019

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 10, 2019

simonpasquier reviewed Dec 10, 2019

View reviewed changes

This was referenced Dec 10, 2019

test/extended/prometheus: exclude AlertmanagerReceiversNotConfigured from checking openshift/origin#24276

Merged

Bug 1797634: test/extended/prometheus: test if AlertmanagerReceiversNotConfigured is firing openshift/origin#24277

Merged

openshift-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 12, 2019

paulfantom force-pushed the route_metric_alert branch from b518d04 to 8fa59bc Compare December 16, 2019 08:51

openshift-ci-robot removed the lgtm Indicates that a PR is ready to be merged. label Dec 16, 2019

openshift-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 16, 2019

openshift-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Dec 19, 2019

paulfantom force-pushed the route_metric_alert branch from 5ebeef0 to 798e265 Compare December 19, 2019 15:57

openshift-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Dec 19, 2019

lilic reviewed Dec 19, 2019

View reviewed changes

openshift-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 13, 2020

paulfantom added 2 commits January 14, 2020 23:44

jsonnet: fix alertmanager recording rule name; add alertmanager criti…

458a082

…cal alert when no routes are configured

assets,pkg/manifests: regenerate

a1fcadf

paulfantom force-pushed the route_metric_alert branch from 798e265 to a1fcadf Compare January 14, 2020 22:45

openshift-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 14, 2020

openshift-ci-robot requested a review from lilic January 14, 2020 22:47

lilic approved these changes Jan 15, 2020

View reviewed changes

openshift-ci-robot assigned lilic Jan 15, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jan 15, 2020

openshift-merge-robot merged commit fb0882f into openshift:master Jan 15, 2020

paulfantom mentioned this pull request Jan 15, 2020

Bug 1791200: Add alert for no routes configured in alertmanager #613

Merged

paulfantom deleted the route_metric_alert branch January 15, 2020 08:39

paulfantom mentioned this pull request Jan 15, 2020

Bug 1791211: Add alert for no routes configured in alertmanager #614

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add alert for no routes configured in alertmanager #585

Add alert for no routes configured in alertmanager #585

paulfantom commented Dec 9, 2019 •

edited

s-urbaniak commented Dec 9, 2019

openshift-bot commented Dec 9, 2019

openshift-bot commented Dec 9, 2019

openshift-bot commented Dec 9, 2019

openshift-bot commented Dec 9, 2019

openshift-bot commented Dec 9, 2019

openshift-bot commented Dec 9, 2019

openshift-bot commented Dec 10, 2019

openshift-bot commented Dec 10, 2019

smarterclayton commented Dec 10, 2019

simonpasquier Dec 10, 2019

paulfantom Dec 10, 2019

simonpasquier Dec 18, 2019

lilic Dec 19, 2019

paulfantom commented Dec 10, 2019

smarterclayton commented Dec 12, 2019

paulfantom commented Dec 16, 2019 •

edited

paulfantom commented Dec 16, 2019

paulfantom commented Dec 19, 2019

lilic left a comment

s-urbaniak commented Jan 6, 2020

lilic commented Jan 13, 2020

paulfantom commented Jan 14, 2020

smarterclayton commented Jan 15, 2020

openshift-cherrypick-robot commented Jan 15, 2020

lilic left a comment

openshift-ci-robot commented Jan 15, 2020

openshift-bot commented Jan 15, 2020

openshift-bot commented Jan 15, 2020

openshift-bot commented Jan 15, 2020

openshift-cherrypick-robot commented Jan 15, 2020

smarterclayton commented Jan 23, 2020

Add alert for no routes configured in alertmanager #585

Add alert for no routes configured in alertmanager #585

Conversation

paulfantom commented Dec 9, 2019 • edited

s-urbaniak commented Dec 9, 2019

openshift-bot commented Dec 9, 2019

openshift-bot commented Dec 9, 2019

openshift-bot commented Dec 9, 2019

openshift-bot commented Dec 9, 2019

openshift-bot commented Dec 9, 2019

openshift-bot commented Dec 9, 2019

openshift-bot commented Dec 10, 2019

openshift-bot commented Dec 10, 2019

smarterclayton commented Dec 10, 2019

simonpasquier Dec 10, 2019

Choose a reason for hiding this comment

paulfantom Dec 10, 2019

Choose a reason for hiding this comment

simonpasquier Dec 18, 2019

Choose a reason for hiding this comment

lilic Dec 19, 2019

Choose a reason for hiding this comment

paulfantom commented Dec 10, 2019

smarterclayton commented Dec 12, 2019

paulfantom commented Dec 16, 2019 • edited

paulfantom commented Dec 16, 2019

paulfantom commented Dec 19, 2019

lilic left a comment

Choose a reason for hiding this comment

s-urbaniak commented Jan 6, 2020

lilic commented Jan 13, 2020

paulfantom commented Jan 14, 2020

smarterclayton commented Jan 15, 2020

openshift-cherrypick-robot commented Jan 15, 2020

lilic left a comment

Choose a reason for hiding this comment

openshift-ci-robot commented Jan 15, 2020

openshift-bot commented Jan 15, 2020

openshift-bot commented Jan 15, 2020

openshift-bot commented Jan 15, 2020

openshift-cherrypick-robot commented Jan 15, 2020

smarterclayton commented Jan 23, 2020

paulfantom commented Dec 9, 2019 •

edited

paulfantom commented Dec 16, 2019 •

edited