rate() is inaccurate for new label

[remram44]

Bug Report

What did you do?

I have a metric that shows calls to my app's "export" function. I use labels to signify what is exported, and in which format. I then plot this using sum(rate(export_total[1m])) by (what, extension)

What did you expect to see?

When someone calls the export function 5 times with a specific what/format pair, I see a spike in the graph to Y=5. If later it calls it again 3 times, I see a spike to Y=3.

What did you see instead? Under which circumstances?

Only the second spike appears.

This is because rate() doesn't treat the missing label as a value 0, and when it appears, rate assume that the current value is the baseline.

The workaround is to initialize all my counters with all possible label combinations to 0 (or any value) at application start, before users can increase them. This is annoying and difficult when those values are defined in a third-party library rather than my own app.

This behavior might be needed if an app already has non-zero counters when you start to scrape it, or when taking the range() of non-Counter metrics (such as Gauges) which don't necessarily start at 0. However it seems to me that a documented workaround is required to not have spikes silently disappear from graphs.

Related: prometheus/client_golang#190
Related: https://www.robustperception.io/existential-issues-with-metrics (notes that this can lead to alerts not firing)

Environment

• System information:

  Linux 4.9.0-8-amd64 x86_64

• Prometheus version:

  prometheus, version 2.8.0 (branch: HEAD, revision: 5936949)
  build user: root@4c4d5c29b71f
  build date: 20190312-07:46:58
  go version: go1.11.5
  (from Docker Hub)

• Prometheus configuration file:

global:
  scrape_interval: 15s
scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]
  - job_name: taguette
    static_configs:
      - targets: ["91.121.86.15:8900"]

[brian-brazil]

Dupe of #1673, there's nothing we can do here I'm afraid.