Implement alert silence link

[pdf]

Judging by the placeholder links, I'm going to assume that the plan is to move silencing into prometheus-proper at some point (as opposed to alertmanager, for example). This would indeed be beneficial, because then the silenced state could appear in the time-series data, and be used for reporting (eg - I don't want my wall-board to light up if an alert has been silenced).

This issue is just to track implementation so that we can have a notification when the feature is committed.

[brian-brazil]

Silences are already implemented in the alertmanager, and there's no plans to move them to prometheus as multiple prometheus servers (some of which may be identical) talk to an alertmanager and you don't want to have to create silences multiple times.

The silence link is meant to be a link to the alertmanager to save you a little bit of time. As-is we should either remove it due to the confusion it causes, or make it actually work.

[matthiasr]

I just spoke with @fabxc about these links yesterday; I would expect them to take me to Alertmanager.

Having the silences live in Prometheus would make it impossible to silence across multiple Prometheus or interact with all of them. It would also be a problem with Prometheus' high availability model (have two). Alertmanager doesn't have one yet, but in general I expect syncing alert state to have very different expectations from the time series pieces, and be better contained in AM.

As for metrics showing the alert state, AM could just expose these (maybe it already does?) and if you want them in Prometheus just scrape it.

[fabxc]

We plan to make the redirection to alertmanager work. It just came up
yesterday with @matthiasr.

We certainly don't want to have two places to do silencing. Aside from
implementation, this would be a major source of confusion and errors during
operation.

On Thu, Aug 6, 2015, 9:55 AM Brian Brazil notifications@github.com wrote:

Silences are already implemented in the alertmanager, and there's no plans
to move them to prometheus as multiple prometheus servers (some of which
may be identical) talk to an alertmanager and you don't want to have to
create silences multiple times.

The silence link is meant to be a link to the alertmanager to save you a
little bit of time. As-is we should either remove it due to the confusion
it causes, or make it actually work.

—
Reply to this email directly or view it on GitHub
#966 (comment)
.

[pdf]

Indeed I hadn't considered the scaling issue.

So, with alertmanager remaining the authoritative source for silences I guess we need alertmanager to re-export received alerts, after appending the silenced state as a new label, so prometheus can scrape them back into a new series. I'll open an issue over there, though I know it's due for a rewrite.

[brian-brazil]

This may be a bit complicated due to us now supporting multiple alertmanagers.