Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add recording rules for Alertmanager dashboard, #387

Merged
merged 4 commits into from
Sep 22, 2021

Conversation

stevesg
Copy link
Contributor

@stevesg stevesg commented Sep 2, 2021

What this PR does:
With large numbers of tenants the queries for these panels can become quite
slow as the metrics exposed are per-tenant.

Checklist

  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@stevesg stevesg marked this pull request as ready for review September 7, 2021 14:06
@stevesg stevesg requested a review from a team as a code owner September 7, 2021 14:06
@stevesg stevesg force-pushed the am-notifications-rules branch 2 times, most recently from c33b542 to 5cb38bc Compare September 7, 2021 14:12
@stevesg stevesg changed the title Add recording rules for Alertmanager notification rate panels. Add recording rules for Alertmanager dashboard, Sep 8, 2021
Copy link
Contributor

@gotjosh gotjosh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - but have one question.

@@ -11,11 +11,11 @@ local utils = import 'mixin-utils/utils.libsonnet';
})
.addPanel(
$.panel('Total Alerts') +
$.statPanel('sum(cortex_alertmanager_alerts{%s})' % $.jobMatcher('alertmanager'), format='short')
$.statPanel('sum(cluster_job_%s:cortex_alertmanager_alerts:sum{%s})' % [$._config.per_instance_label, $.jobMatcher('alertmanager')], format='short')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the recording rule is a sum, do we need to sum here again?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question, I also wondered about this, but in general I decided to leave all the sum()s as-is because the dashboard allows selecting multiple cluster/namespaces.

Copy link
Contributor

@gotjosh gotjosh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Collaborator

@pracucci pracucci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

With large numbers of tenants the queries for some panels on thos dashboard can
become quite slow as the metrics exposed are per-tenant.
@stevesg stevesg merged commit 0c84691 into grafana:main Sep 22, 2021
simonswine pushed a commit to grafana/mimir that referenced this pull request Oct 18, 2021
…ations-rules

Add recording rules for Alertmanager dashboard,
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants