You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When enabling .spec.observability.metrics.createPrometheusRules multiple alerts are generated, for example:
- alert: TempoDistributorUnhealthy
annotations:
message: There are {{ printf "%f" $value }} unhealthy distributor(s).
runbook_url: https://github.com/grafana/tempo/tree/main/operations/tempo-mixin/runbook.md#TempoDistributorUnhealthy
expr: |
max by (cluster, namespace) (tempo_ring_members{state="Unhealthy", name="distributor", namespace=~".*"}) > 0
for: 15m
labels:
severity: warning
These alerts match Tempo instance of the entire cluster, but they should only match Tempo instances of the TempoStack where createPrometheusRules was enabled.
Suggested solution: add cluster="$tempostack_name", namespace="$namespace" to every alert.
Note: Also update the recording rules.
The text was updated successfully, but these errors were encountered:
When enabling
.spec.observability.metrics.createPrometheusRules
multiple alerts are generated, for example:These alerts match Tempo instance of the entire cluster, but they should only match Tempo instances of the TempoStack where
createPrometheusRules
was enabled.Suggested solution: add
cluster="$tempostack_name", namespace="$namespace"
to every alert.Note: Also update the recording rules.
The text was updated successfully, but these errors were encountered: