Alertmanager configured route metric #554
Conversation
openshift-ci-robot
requested review from
s-urbaniak,
simonpasquier and
smarterclayton
openshift-ci-robot
added
approved
|
This looks sufficient to test the outcome for me. Can we add a warning alert with a 1h interval that suggests the user configure notifications? Not having notification sinks configured to me seems like it crosses the warning threshold in that without notification sinks alert manager is at best a dashboard function. If there's controversy on that, we can take it as a separate PR, otherwise I'd suggest doing it here. @brancz @s-urbaniak @lilic opinions? |
openshift-ci-robot
added
the
needs-rebase
|
@smarterclayton I guess this is a good suggestion. While we have a watchdog alert by default, it still doesn't really kick in until you have fully configured alertmanager. A concrete alert showing up in console highlighting the fact of an unconfigured monitoring stack is probably a good addition, wdyt @lilic @brancz. |
|
Plus we can use the presence of that alert firing to attach the UX for
"hey, should configure a notification sink" vs using some other store for
that value. If you don't want it, silence it. Either way you learn how to
use alerts.
…On Tue, Nov 26, 2019 at 4:59 AM OpenShift CI Robot ***@***.***> wrote:
[APPROVALNOTIFIER] This PR is *APPROVED*
This pull-request has been approved by: *paulfantom
<#554#>*, *simonpasquier
<#554 (review)>*
The full list of commands accepted by this bot can be found here
<https://go.k8s.io/bot-commands?repo=openshift%2Fcluster-monitoring-operator>
.
The pull request process is described here
<https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process>
Needs approval from an approver in each of these files:
- OWNERS
<https://github.com/openshift/cluster-monitoring-operator/blob/master/OWNERS>
[paulfantom,simonpasquier]
Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#554?email_source=notifications&email_token=AAI37J7SPW7ROPLAULI7KQDQVTXPDA5CNFSM4JNO7L42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFFN3JA#issuecomment-558554532>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAI37JZYEXGQRJV4IQ7TNTLQVTXPDANCNFSM4JNO7L4Q>
.
|
|
@paulfantom ping, this needs a rebase |
…gured/sends notifications
paulfantom
force-pushed
the
am_configured_route
branch
from
e365bd2
to
b4da6d9
Compare
|
rebased |
openshift-ci-robot
removed
the
needs-rebase
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: paulfantom, s-urbaniak, simonpasquier The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
| @@ -356,6 +356,8 @@ spec: | |||
| label_node_hyperthread_enabled, label_node_openshift_io_os_id,label_node_role_kubernetes_io_master,label_node_role_kubernetes_io_infra) | |||
| ) by (label_kubernetes_io_arch, label_node_hyperthread_enabled, label_node_openshift_io_os_id,label_node_role_kubernetes_io_master,label_node_role_kubernetes_io_infra) | |||
| record: node_role_os_version_machine:cpu_capacity_sockets:sum | |||
| - expr: clamp_max(sum(alertmanager_notifications_total),1) | |||
| record: alertmanager_routing_enabled | |||
There was a problem hiding this comment.
This doesn’t follow naming best practices.
There was a problem hiding this comment.
Do you think something like cluster:alertmanager_routing_enabled:max would be better?
|
When the alert is added can you link it to here? /cherry-pick release-4.3 |
|
@smarterclayton: failed to push cherry-picked changes in GitHub: pushing failed, output: "remote: Not Found\nfatal: repository 'https://openshift-cherrypick-robot:CENSORED@github.com/openshift-cherrypick-robot/openshift/cluster-monitoring-operator/' not found\n", error: exit status 128 In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Idea is to have a metric which can be gathered by telemetry and tell us if clients use alertmanager and send notifications/alerts at all.
This will return simple binary value.
/cc @smarterclayton @simonpasquier @s-urbaniak