Metrics within prometheus is not real time

[ArtemChekunov]

I made a bunch with consul, consul_exporter and prometheus.
After that i made alert rule:

ALERT ConsulClientCount
  IF sum(consul_health_node_status{check="serfHealth",node=~".*-services-.*"}) < 100
  FOR 30s
  ANNOTATIONS {description="Consul Client Count value={{$value}})", summary="Consul Client Count LESS 100"}

but in case when consul client is down the alert is worked out only after ~5m

  scrape_interval:     5s 
  evaluation_interval: 5s.

P.S. consul_exporter returns data correctly

[brian-brazil]

It sounds like you're running into a staleness issue, as old timeseries that are no longer exported are still considered for 5 minutes. I'd suggest alerting on something that represents direct user experience, rather than trying to catch every possible failure mode such as a drop in clients.

[ArtemChekunov]

thanx

-query.staleness-delta 5s is solved my issue

[brian-brazil]

I'd strongly advise against dropping -query.staleness-delta to that low a value, it'll cause things to break in fun ways.

[fuzzyami]

@brian-brazil what would you consider a safe minimum? 1m?

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.