Persist keep firing for state across restarts

[mustafain117]

This PR resolves issue #13957

In this PR:

• alerts using the keep_firing_for duration will be stored to persistence storage after each evaluation.
• alerts are restored from storage before the first evaluation for the group after server restarts.
• implemented the persistence store using badgerDB (open to feedback on choice of implementation)
  • store entries have a configurable TTL, runs GC to cleanup old entries
• users of Prometheus as a library can provide their own store implementation when starting the rule manager
• Feature flag to enable this feature

Testing:

Apart from unit tests, functionally tested with upto 1000 alerts:

First annotation represents feature being enabled, second annotation is for server restart after which alerts state is restored. Firing alerts are no longer firing when the feature is disabled and server restarts, whereas they keep firing when server restarts and feature is enabled. Minimal spike in eval latency when restoring state.

[roidelapluie]

Why can't we use the TSDB like we do for the alerts timestamps?

[mustafain117]

Why can't we use the TSDB like we do for the alerts timestamps?

To restore the alert we need more than just the KeepFiringSince timestamp. Restored alert needs to have the same value, labels and annotations, therefore, restoring just the timestamp is not sufficient. When Prometheus restarts, the expression is already resolved so we can't evaluate the rule again to restore the value, labels and annotations (approach used to restore 'for' state)

While the alert labels can be stored in TSDB as part of the sample, was not sure how annotations can be stored in the same sample (perhaps as labels with custom prefix to differentiate between the alert labels and annotations). Additionally, would also need to store the expression value as a separate metric.