Graphs in Alerts

[aaranmcguire]

Hi,

Apologies if this has already been mentioned, or should be in prometheus/alertmanager. I wondered if its posable to embed graphs within the alerts (emails,slack,etc). I've seen this done in Bosun and it makes it really nice when getting alerts as you can visualise the issue (e.g. CPU has just spiked, Disk space is low).

If it were to be implemented I'd want a way to have a different query (Like Boson) so I could get a longer time period.

Is this something thats posable? is someone already working on it? (or has it been turned down?)

I'm new to this project so don't think I have enough experience to implement it but I'd be willing to try if this concept was accepted.

[grobie]

This sounds like an issue you should open in the alertmanager repo instead. The only thing which is available right now is the generator URL. If you click that the prometheus graph page for the alert condition will open.

[brian-brazil]

Alerts are just a set of strings, if you want to do this I'd suggest plumbing through an img, iframe or A tag that'll be displayed by which ever notification method you're using. Another option is to include the expression(s) you want graphed as an annotation and plumb that through to something that can expand it.

From a general architecture/sanity standpoint sending actual images with Prometheus is unlikely to ever happen, and similarly I'd expect Alertmanager will not gain the features to display or expand them as that gets into messy reliability, security and UI stuff.

[aaranmcguire]

@brian-brazil Thats a shame, Although a URL may work it would of been nice to do something like this:

[brian-brazil]

The way I see it, a single alert definition could easily create 10K alerts. At 10KB each that's 100MB. That's a lot of data to be both regularly sending to the alertmanager, and even sending from the Prometheus as it'd take up a gigabit NIC for a second. This is not a safe feature to offer in a naive fashion.

[aaranmcguire]

@brian-brazil That makes sense.. I'll close this off then.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.