Flatline alerts generating false negative alerts

[eeH9ahso]

Hi there. Similar to issue #1424, I have an issue where elastalert is generating false negative flatline alerts.

I have a rule that looks for an absence of logs coming in from the ca-central-* AWS availability zone:

name: CA Availability Zone log ingestion flatline
type: flatline
index: .ds-logs-import-default-*
use_count_query: true
is_enabled: true

threshold: 1
timeframe:
  minutes: 5
filter:
- query:
    query_string:
      query: "cloud.availability_zone: ca-central-*"

alert:
  - "slack"
alert_subject: "0 logs found for log source: CA Availability Zone"

slack_webhook_url: "https://hooks.slack.com/services/..."

Here are the debug logs:

2024-05-23 16:31:35,883    DEBUG apscheduler.scheduler Looking for jobs to run
2024-05-23 16:31:35,884    DEBUG apscheduler.scheduler Next wakeup is due at 2024-05-23 16:32:05.376417+00:00 (in 29.491891 seconds)
2024-05-23 16:31:35,884     INFO apscheduler.executors.default Running job "Rule: CA Availability Zone log ingestion flatline (trigger: interval[0:01:00], next run at: 2024-05-23 16:32:37 UTC)" (scheduled at 2024-05-23 16:31:35.883239+00:00)
2024-05-23 16:31:35,886    DEBUG urllib3.connectionpool Resetting dropped connection: mycluster.com
2024-05-23 16:31:35,983    DEBUG urllib3.connectionpool https://mycluster.com:9200 "POST /.ds-logs-import-default-*/_count?ignore_unavailable=true HTTP/1.1" 200 73
2024-05-23 16:31:35,984     INFO        elasticsearch POST https://mycluster.com:9200/.ds-logs-import-default-*/_count?ignore_unavailable=true [status:200 request:0.099s]
2024-05-23 16:31:35,984    DEBUG        elasticsearch > {"query":{"bool":{"filter":{"bool":{"must":[{"range":{"@timestamp":{"gt":"2024-05-23T16:30:34.703427Z","lte":"2024-05-23T16:31:34.703427Z"}}},{"query_string":{"query":"cloud.availability_zone: ca-central-*"}}]}}}}}
2024-05-23 16:31:35,984    DEBUG        elasticsearch < {"count":0,"_shards":{"total":12,"successful":12,"skipped":0,"failed":0}}
2024-05-23 16:31:35,984     INFO           elastalert Queried rule CA Availability Zone log ingestion flatline from 2024-05-23 16:30 UTC to 2024-05-23 16:31 UTC: 0 hits
2024-05-23 16:31:35,994    DEBUG urllib3.connectionpool https://mycluster.com:9200 "POST /.ds-logs-import-default-*/_count?ignore_unavailable=true HTTP/1.1" 200 73
2024-05-23 16:31:35,995     INFO        elasticsearch POST https://mycluster.com:9200/.ds-logs-import-default-*/_count?ignore_unavailable=true [status:200 request:0.010s]
2024-05-23 16:31:35,995    DEBUG        elasticsearch > {"query":{"bool":{"filter":{"bool":{"must":[{"range":{"@timestamp":{"gt":"2024-05-23T16:31:34.703427Z","lte":"2024-05-23T16:31:35.884925Z"}}},{"query_string":{"query":"cloud.availability_zone: ca-central-*"}}]}}}}}
2024-05-23 16:31:35,995    DEBUG        elasticsearch < {"count":0,"_shards":{"total":12,"successful":12,"skipped":0,"failed":0}}
2024-05-23 16:31:35,996     INFO           elastalert Queried rule CA Availability Zone log ingestion flatline from 2024-05-23 16:31 UTC to 2024-05-23 16:31 UTC: 0 hits
2024-05-23 16:31:36,008    DEBUG urllib3.connectionpool https://mycluster.com:9200 "POST /elastalert_status_silence/_search?_source_includes=until%2Cexponent&size=1 HTTP/1.1" 200 None
2024-05-23 16:31:36,009     INFO        elasticsearch POST https://mycluster.com:9200/elastalert_status_silence/_search?_source_includes=until%2Cexponent&size=1 [status:200 request:0.012s]
2024-05-23 16:31:36,009    DEBUG        elasticsearch > {"query":{"term":{"rule_name":"CA Availability Zone log ingestion flatline._silence"}},"sort":{"until":{"order":"desc"}}}
2024-05-23 16:31:36,009    DEBUG        elasticsearch < {"took":7,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":0,"relation":"eq"},"max_score":null,"hits":[]}}
2024-05-23 16:31:36,017    DEBUG urllib3.connectionpool https://mycluster.com:9200 "POST /elastalert_status_silence/_search?_source_includes=until%2Cexponent&size=1 HTTP/1.1" 200 None
2024-05-23 16:31:36,017     INFO        elasticsearch POST https://mycluster.com:9200/elastalert_status_silence/_search?_source_includes=until%2Cexponent&size=1 [status:200 request:0.008s]
2024-05-23 16:31:36,018    DEBUG        elasticsearch > {"query":{"term":{"rule_name":"CA Availability Zone log ingestion flatline.all"}},"sort":{"until":{"order":"desc"}}}
2024-05-23 16:31:36,018    DEBUG        elasticsearch < {"took":1,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":1030,"relation":"eq"},"max_score":null,"hits":[{"_index":"elastalert_status_silence","_id":"an1Kpo8Boey5o6I_Pzak","_score":
null,"_source":{"exponent":0,"until":"2024-05-23T16:31:34.914323Z"},"sort":[1716481894914]}]}}
2024-05-23 16:31:36,038    DEBUG urllib3.connectionpool https://mycluster.com:9200 "POST /elastalert_status_silence/_doc HTTP/1.1" 201 181
2024-05-23 16:31:36,039     INFO        elasticsearch POST https://mycluster.com:9200/elastalert_status_silence/_doc [status:201 request:0.020s]
2024-05-23 16:31:36,039    DEBUG        elasticsearch > {"exponent":0,"rule_name":"CA Availability Zone log ingestion flatline.all","@timestamp":"2024-05-23T16:31:36.018907Z","until":"2024-05-23T16:32:36.018901Z"}
2024-05-23 16:31:36,039    DEBUG        elasticsearch < {"_index":"elastalert_status_silence","_id":"UZNLpo8BWuTh9cgcLt5V","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":1143,"_primary_term":12}
2024-05-23 16:31:36,044    DEBUG urllib3.connectionpool Starting new HTTPS connection (1): hooks.slack.com:443
2024-05-23 16:31:36,131    DEBUG urllib3.connectionpool https://hooks.slack.com:443 "POST /services/... HTTP/1.1" 200 2
2024-05-23 16:31:36,132     INFO           elastalert Alert 'CA Availability Zone log ingestion flatline' sent to Slack

From Slack:

But you can see in the screenshot below, logs have consistently been coming in:

What am I doing wrong here?

Thank you very much!

[jertel]

It looks like ElastAlert 2 is functioning correctly, but the query to Elasticsearch isn't returning anything. Perhaps you have ElastAlert 2 pointed at the wrong ES cluster? Or there's a mistake in your datastream configuration, where it's not pulling from that index? Perhaps try setting the rule's index property to bypass the datastream and query the backing index directly.