Alertmanager peerReconnectTimeout config

[jquick]

We are using Grafana in HA with alerting in kubernetes. This enables the alertmanager gossip network. Alertmanger has a default of:

--cluster.reconnect-timeout value: length of time to attempt to reconnect to a lost peer (default: "6h0m0s")

Ref: https://github.com/prometheus/alertmanager/blob/main/cmd/alertmanager/main.go#L230

This works fine except due to the turn over of pods the gossip network will try to hit any expired/terminated pods for 6 hours every 10 seconds. I didn't see an easy way to edit/change this but it would be nice if this were configurable.

[jquick]

going to open this on the main project