Unit testing for alert rules are not working as expected

[kentsanggds]

Bug Report

What did you do?

I'm trying to set up the unit testing for alert rules according to these docs - https://github.com/prometheus/prometheus/blob/master/docs/configuration/unit_testing_rules.md#testyml

I had to fix the yaml by using - alert_rule_test rather than alert_rule_test to get it running

What did you expect to see?

The example alert rule test to work and report success

What did you see instead? Under which circumstances?

Running promtool test rules alerts.yml doesn't work, it fails.

Environment

Mac OS

• System information:

Darwin 17.7.0 x86_64

• Promtool version:

promtool, version 2.4.2 (branch: master, revision: 6932030)
build user: xxx
build date: 20181002-14:56:44
go version: go1.10.3

[gouthamve]

/cc @codesome

[codesome]

@kentsanggds Thanks for this, looks like docs are not clear.

I had to fix the yaml by using - alert_rule_test rather than alert_rule_test to get it running

In the docs I missed the indentation (it should be lined with input_series). Can you try with this (same thing with proper indentation): https://gist.github.com/codesome/f4426d28244ee9fac128031ed2f9af70?

Running promtool test rules alerts.yml doesn't work, it fails.

I will update the docs with clear instructions. alerts.yml is inputed in test.yml. So the command should be promtool test rules test.yml (with alerts.yml in the same directory).

[kentsanggds]

Thanks, I tried that out and that fixed it.
Strange that the yaml appeared to be acceptable, perhaps there should be additional checks for input series?

Also is it possible to simulate missing metrics in the input_series?

[codesome]

Strange that the yaml appeared to be acceptable, perhaps there should be additional checks for input series?

I am not clear what are we referring to.

Also is it possible to simulate missing metrics in the input_series?

I did not quite understand this, can you give an example?

[kentsanggds]

I managed to get what I thought was a working yaml file by adding a - to the alert_rule_test, so the test ran but didn't have any input_series, would be nice if there are no series data for the test to be flagged up.

Regarding missing metrics it would be nice if we could simulate when at particular data points the metric was missing, perhaps something like this -

100 101 100 - - - 120 130

where dashes - represent a missing metric as putting in a 0 would not count as a missing metric.

This would represent situations where the target metric was unavailable for scraping or the target was removed by accident for example.

[codesome]

No input_series would indicate the situation where none of the exporters are up. So I would like to keep it the way it is now.

And regarding having missing metrics, it is already supported. You use _ for that. You can refer this example from code to know more.

[kentsanggds]

Thanks, ok I understand the reason for the input_series.

With the missing metrics, I'm not sure it is working as expected as it doesn't trigger the alert, can you tell me how I would run the test so that I can modify the existing test to try things out.

[gouthamve]

@kentsanggds Please compile promtool from master and run ./promtool test rules test.yml

[kentsanggds]

I was actually referring to the tests here - https://github.com/prometheus/prometheus/blob/9bca041285e226410809b64ce2d178bc4ccabbcf/promql/parse_test.go, in the end I sorted it out and managed to get the alerting rule tests working as I would like, but it would be nice also to be able to run those tests to try things out without having to set up the tests.

One thing which I find odd is that if it fails due to the expected not matching the actual response, it will report the failure a number of times rather than just the 1 test that I'm adding, which is slightly confusing.

Also is it possible to make the tests more verbose to indicate which tests have passed as at the moment when it passes you just get SUCCESS.

[codesome]

Can you share your test files so that I can dig in more?

[kentsanggds]

Sure, I am working on a branch in github, have a look here for the test file -

https://github.com/alphagov/prometheus-aws-configuration-beta/blob/add-rules-tests/terraform/projects/app-ecs-services/config/alerts/test-observe-alerts.yml