new flag: ruler.outage-tolerance

[owen-d]

What

Exposes the underlying Prometheus manager option, OutageTolerance. This is the lookback period for which the metric store can be queried for past alert evaluations. It's used when each rule group is initially run. In Cortex, this has implications not only during restarts, but also when the ring membership changes. Without this configuration, previous rule state cannot be restored, meaning that after re-sharding/restarts, we must wait the entire ForDuration until an alert may be considered firing.

edit: defaulted to prometheus defaults for these flags and included a total of 3 configs at Jacob's suggestion:

# Max time to tolerate outage for restoring "for" state of alert.
# CLI flag: -ruler.for-outage-tolerance
[for_outage_tolerance: <duration> | default = 1h]

# Minimum duration between alert and restored "for" state. This is maintained
# only for alerts with configured "for" time greater than grace period.
# CLI flag: -ruler.for-grace-period
[for_grace_period: <duration> | default = 10m]

# Minimum amount of time to wait before resending an alert to Alertmanager.
# CLI flag: -ruler.resend-delay
[resend_delay: <duration> | default = 1m]

[jtlisi]

LGTM

There are some other options we should also consider exposing here:

https://github.com/prometheus/prometheus/blob/b788986717e1597452ca25e5219510bb787165c7/cmd/prometheus/main.go#L234-L238

[pracucci]

Uups! The flag is ruler.resend-delay. @owen-d Would you mind opening a PR to fix the CHANGELOG entry, please?

[bboreham]

Also put it in the correct place, like we ask in the checklist.