Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

External labels not applied to alerts #7327

Open
lennartack opened this issue May 2, 2024 · 4 comments
Open

External labels not applied to alerts #7327

lennartack opened this issue May 2, 2024 · 4 comments

Comments

@lennartack
Copy link

lennartack commented May 2, 2024

I have configured Prometheus with external_labels. If I understand correctly the Thanos sidecar should add these labels to metrics and rules when they are proxied to the querier. This works on my setup for metrics. However, it seems external labels are not applied to alerts. The /api/v1/alerts endpoint on the Thanos querier gives the alerts without external labels.

@GiedriusS
Copy link
Member

GiedriusS commented May 3, 2024

Thanos Query also deduplicates alerts so I think that's why you don't see them. Are they visible if you removed replica labels from Query?

@lennartack
Copy link
Author

@GiedriusS The only replica label I have configured is replica. The external label I'm looking for is called cluster. I don't think it is deduplication as the cluster label is applied to metrics, just not to alert instances.

@lennartack
Copy link
Author

lennartack commented May 3, 2024

I can also see the labels in the alert definitions at /api/v1/rules?type=alert. But not the instances at /api/v1/alerts. The latter is quite important for us for viewing alert instances in Grafana per cluster.

@viktordaniel
Copy link

viktordaniel commented Oct 1, 2024

👍 +1 on it, we have same problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants