Throttle resends of alerts

[brian-brazil]

We currently send alerts at every eval interval. With a 1s eval interval, this would produce a lot of traffic to the AM.

I propose we continue to send new alerts instantly, but throttle the resends to say 60s by default.

[brancz]

Generally I think that's ok, we just need to make it explicit in the documentation and make sure that the resolve timeout in the Alertmanager needs to be higher than this value.

[brian-brazil]

The resolve timeout is just a default, we can adjust the end time as required inside Prometheus.

[brancz]

Yes I realize that I just meant we need to document properly what the relationship between the two is.

[stuartnelson3]

The current ingest path in AM for alerts has pretty high lock contention, throttling resends would be 👍

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.