bad counter increase query result

[NoumanSaleem]

What did you do?
Ran a query to determine the increase in a counter for the last hour.

What did you expect to see?
The result to equal the counter (now) - counter (offset 1hr)

What did you see instead? Under which circumstances?
A value greater than the counter's current value.

Environment

• System information:
  Linux 3.13.0-93-generic x86_64
• Prometheus version:
  prometheus, version 1.1.1 (branch: master, revision: 24db241)
  build user: root@90d3f69e2d67
  build date: 20160907-09:42:10
  go version: go1.6.3
• Prometheus configuration file:

global:
  scrape_interval:     1m
  evaluation_interval: 1m

scrape_configs:
  - job_name: 'service'
    consul_sd_configs:
    - server: 172.17.42.1:8500
    relabel_configs:
      - source_labels: [__meta_consul_tags]
        regex: ',((api|data)-service),'
        action: keep
      - source_labels: [__meta_consul_service_address, __meta_consul_service_port]
        separator: ':'
        target_label: __address__
      - source_labels: [__meta_consul_service]
        target_label: name
      - source_labels: [__meta_consul_tags]
        target_label: tags

[brian-brazil]

I suspect you have had some counter resets due to restarts. This is only of the many edge cases that increase() and friends have to deal with.

[NoumanSaleem]

@brian-brazil thanks Brian. To be clear, we're talking about restarts on the application exposing the counter and not Prometheus itself, correct?

[brian-brazil]

Yes

[NoumanSaleem]

@brian-brazil does prometheus expose any info to help debug? Maybe a metric for resets?

[brian-brazil]

There's the resets() function, though in general for this sort of issue it's best to look at the raw data.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.