Duplicated alerts

[CrazyNash]

What did you do?
A prometheus cluster with three instances connected to alert manager cluster with three instances

What did you expect to see?
each alert fired only once

What did you see instead? Under which circumstances?

each alert fired three times

Environment

• System information:

  insert output of uname -srm here

• Alertmanager version:

  0.15-rc.3

• Prometheus version:

  the latest

The three prometheus instances connected to same db, so each alert will be sent to all three alert managers from all three prom instances. The alert from same prom is correctly duplicated to one, but not all of them from all prom. The original 3*3=9 alerts should be duplicated to 1 but 3 now

[hoffie]

Can you provide some more details?

• Has this setup worked before (e.g. with previous alertmanager versions)?
• Can you share the relevant prometheus configs (configured alertmanagers, alert rules) and alertmanager configs (command line, config file)? It would be even better if it was possible to provide some minimal configuration which triggers the problem so that others can easily reproduce it.

[CrazyNash]

@hoffie This is the first time it's set up like this.

I don't think detailed config info is necessary here, it's just a very common question, does alert manager support this? It seems alert manager only dedup alerts fom same source url, is it ever designed to support deduping alerts from multiple sources(three prometheus instances here)? if yes then how to set up? can't find any doc in the area.

[hoffie]

Yes, alertmanager is supposed to support this.
Your description sounds like different external labels in your Prometheus instances may be the problem. The following blog post may give the right pointers in such cases:
https://www.robustperception.io/high-availability-prometheus-alerting-and-notification/

If you need further help, I suggest posting to the mailing list or joining the IRC channel, as the Github issues are supposed to be used for bug reports and feature requests only. Even in this case, it will be helpful if you can share the relevant config portions so that others can help pinpointing the specific issue in your setup.

[CrazyNash]

@hoffie Thanks, it's very helpful, seems like relable is must have in the set up, I will try it out. By the way it's necessary/good to show this example in user guide.

[CrazyNash]

It works, thanks