count() should result in 0 if no timeseries found

[sgeisbacher]

Proposal

Use case. Why is this important?
for dynamic metrics, like node_exporters node_filesystem* where for each mountpoint that metrics increase, i use count() in alert-rules like here:

count(node_filesystem_avail_bytes{instance="nas.example.com",mountpoint=~"/srv/nas/(backups|files)"}) < 2

to get notified when one of them is not mounted anymore.
it works perfectly if one is missing as count() then returns 1 and the rule fires. but it does not fire if both are missing because than count() returns no data
the workaround is to additionally check with absent() but it's on the one hand annoying to double-check on each rule and on the other hand count should be able to "count" zero timeseries too:

count(node_filesystem_avail_bytes{instance="nas.example.com",mountpoint=~"/srv/nas/(backups|files)"}) < 2
 or absent(node_filesystem_avail_bytes{instance="nas.example.com",mountpoint=~"/srv/nas/(backups|files)"})

Bug Report

What did you do?
used count(...) in alert rules to get notified on missing metrics
What did you expect to see?
count(...) should return 0 if there is no result
What did you see instead? Under which circumstances?
count(...) returns no data if there are no metrics and so alert-rule does not fire (rule: count(...) < 2)

• System information:
  Linux 4.14.70-v7+ armv7l
• Prometheus version:
  prometheus 2.4.2
• Alertmanager version:
  alertmanager, version 0.15.2
• Alertmanager configuration file:

  - alert: important_nas_mounts_missing
    expr: count(node_filesystem_avail_bytes{instance="nas.example.com",mountpoint=~"/srv/nas/(backups|files)"}) < 2
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: IMPORTANT NAS MOUNTS MISSING
      description: 'backups or files mount on nas.example.com missing'

[brian-brazil]

See https://www.robustperception.io/why-can-countx-5-not-return-0. You want bool.

…

On Sun 9 Dec 2018, 12:57 Stefan Geisbacher ***@***.*** wrote: Proposal *Use case. Why is this important?* for dynamic metrics, like *node_exporter*s *node_filesystem** where for each mountpoint that metrics increase, i use count() in alert-rules like here: count(node_filesystem_avail_bytes{instance="nas.example.com",mountpoint=~"/srv/nas/(backups|files)"}) < 2 to get notified when one of them is not mounted anymore. it works perfectly if one is missing as count() then returns 1 and the rule fires. but it does not fire if both are missing because than count() returns *no-data* the workaround is to additionally check with *absent()* but it's on the one hand annoying to double-check on each rule and on the other hand count should be able to "count" zero timeseries too. Bug Report *What did you do?* used count(...) in alert rules to get notified on missing metrics *What did you expect to see?* count(...) should return 0 if there is no result *What did you see instead? Under which circumstances?* count(...) returns *no-data* if there are no metrics and so alert-rule does not fire (rule: count(...) < 2) - System information: Linux 4.14.70-v7+ armv7l - Prometheus version: prometheus 2.4.2 - Alertmanager version: alertmanager, version 0.15.2 - Alertmanager configuration file: - alert: important_nas_mounts_missing expr: count(node_filesystem_avail_bytes{instance="nas.example.com",mountpoint=~"/srv/nas/(backups|files)"}) < 2 for: 5m labels: severity: critical annotations: summary: IMPORTANT NAS MOUNTS MISSING description: 'backups or files mount on nas.example.com missing' — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#4982>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGyTdhxU4Z5CRvsSrnewHmK2K61ww9JPks5u3PrGgaJpZM4ZKFm2> .

[sgeisbacher]

oh ok, thx but to be honest thats not very intuitive and every new count()-user will fall in this trap again i guess 😅
nevertheless i'm comparing result of count() not inside like in your example. i'm not filtering by operator but by regex on a label.
so i think bool does not work here. is there some other workarround?

[brian-brazil]

https://www.robustperception.io/absent-alerting-for-scraped-metrics should point you in the right direction. In relation to your proposal, what labels would count return in this case?

…

On Sun 9 Dec 2018, 13:30 Stefan Geisbacher ***@***.*** wrote: oh ok, thx but to be honest thats not very intuitive and every new count()-user will fall in this trap again i guess 😅 nevertheless i'm comparing result of count() not inside like in your example. i'm not filtering by operator but by regex on a label. so i think bool does not work here. is there some other workarround? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#4982 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGyTdu47Gf_UPIsHZURvicDJi54Qk_7Oks5u3QJ2gaJpZM4ZKFm2> .