Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scope PrometheusRule to a specific TempoStack instance #485

Closed
andreasgerstmayr opened this issue Jun 27, 2023 · 0 comments · Fixed by #514
Closed

Scope PrometheusRule to a specific TempoStack instance #485

andreasgerstmayr opened this issue Jun 27, 2023 · 0 comments · Fixed by #514
Assignees

Comments

@andreasgerstmayr
Copy link
Collaborator

When enabling .spec.observability.metrics.createPrometheusRules multiple alerts are generated, for example:

      - alert: TempoDistributorUnhealthy
        annotations:
          message: There are {{ printf "%f" $value }} unhealthy distributor(s).
          runbook_url: https://github.com/grafana/tempo/tree/main/operations/tempo-mixin/runbook.md#TempoDistributorUnhealthy
        expr: |
          max by (cluster, namespace) (tempo_ring_members{state="Unhealthy", name="distributor", namespace=~".*"}) > 0
        for: 15m
        labels:
          severity: warning

These alerts match Tempo instance of the entire cluster, but they should only match Tempo instances of the TempoStack where createPrometheusRules was enabled.

Suggested solution: add cluster="$tempostack_name", namespace="$namespace" to every alert.
Note: Also update the recording rules.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants