vmalert: detect alerting rules which don't match any series at all

[hagen1778]

Is your feature request related to a problem? Please describe

Alerting rules, in most of the cases are defined with some condition in expression. For example:

• increase(http_requests_errors_total) > 0
• changes(app_uptime_seconds) > 2

If the expression returns something in response, the alert will be triggered. If no response is received, then the alert remains inactive - it is assumed conditions in expression aren't met.

However, there is no way to tell whether it is comparison condition wasn't met or it is series selector returned nothing. It is very easy to make a typo in http_reqeusts_errors_total and such an expression will always return nothing.

Describe the solution you'd like

It would be great if vmalert could differentiate between expressions that do not match any series at all and expressions that do not meet conditions. With recently enhanced response from VM HTTP API it becomes possible - #3925

What can be done:

1. vmalert can start parsing new field seriesFetched in response for alerting rules. vmalert shouldn't complain if the field is missing to remain compliant with Prometheus or older versions of VM.
2. The value from this field can be used for the following enhancements:
   2.1. expose seriesFetched as a metric per alerting rule, in the same fashion as vmalert_alerting_rules_last_evaluation_samples. Such a metric can then be used for alerting purposes. For example, vmalert_alerting_rules_last_evaluation_series_fetched == 0 will identify rules which do not match any series.
   2.2. expose seriesFetched in vmalert's UI next to alerting rule. Rules with seriesFetched==0 can be marked as yellow/warn rules which need to be taken care of. UI should contain an explanation why the rule is marked so and what can be done.
3. Changes should be consistent with alert's state object and object representation via API.
4. The improvement is relevant for alerting rules mostly. Recording rules are expected to always produce samples in response, so "bad" recording rules can be easily identified via vmalert_recording_rules_last_evaluation_samples metric.

Describe alternatives you've considered

Tools like https://github.com/cloudflare/pint already do such a validation by running alerting rules without conditions. However, such checks are done only periodically: before applying the rule or when making changes to the rule. If the metric changes in a runtime - the problem will remain undetected.
Using https://github.com/cloudflare/pint is also an extra step to do. It would be great if such a feature was built-in in the ruler.

Additional information

No response

[Haleygo]

@hagen1778 I would love to take this task if you want :)

Will add Stats when parsePrometheusResponse and store it in latest ruleStateEntry, is it ok with you?

[valyala]

The feature has been implemented in the commit 5b8450f . See these docs for details. This feature will be included in the next release. In the mean time it is possible to build vmalert from this commit according to these instructions and verify whether the feature works as intended.

[valyala]

FYI, vmalert exposes vmalert_alerting_rules_last_evaluation_series_fetched metric for probably incorrect alerting queries, which match zero time series, starting from v1.91.0. See these docs for details.

Closing the feature request as done.