abnormal values with irate .

[tuxknight]

What did you do?

In a HA prometheus cluster, two prometheus instances had same remote_read and remote_write. But it breaks when using irate function. It works fine if I stop one prometheus.

What did you expect to see?
irate should return correct value.

What did you see instead? Under which circumstances?
irate expression: irate({__name__=~"^node_cpu(_seconds_total|)$",job="nodes",mode="idle", instance_ip="xxx",cpu="1"}[1m])
value expression: {__name__=~"^node_cpu(_seconds_total|)$",job="nodes",mode="idle", instance_ip="xxx",cpu="1"}

time	irate result	value
2019-04-15-10:11:15	6529644809.999999	6529644.81
2019-04-15-10:14:30	0.7496666666741173	6529797.44
2019-04-15-10:10:15	0.6843789586059327	6529603.42
2019-04-15-10:13:30	0.7489999999913076	6529752.59
2019-04-15-10:09:15	0.7348381288979846	6529560.58
2019-04-15-10:12:30	0.7489999999913076	6529708.76
2019-04-15-10:11:30	0.6106666666766007	6529663.13
2019-04-15-10:14:45	0.7496666666741173	6529797.44
2019-04-15-10:10:30	6529623630	6529623.63
2019-04-15-10:13:45	0.7489999999913076	6529752.59
2019-04-15-10:09:30	0	6529582.89
2019-04-15-10:12:45	0.7489999999913076	6529708.76
2019-04-15-10:11:45	0.6106666666766007	6529663.13
2019-04-15-10:15:00	0	6529815.08
2019-04-15-10:10:45	6529623630	6529623.63
2019-04-15-10:14:00	0.7453333333445092	6529774.95
2019-04-15-10:09:45	0	6529582.89
2019-04-15-10:13:00	0.7120000000111759	6529730.12
2019-04-15-10:12:00	0.772000000004967	6529686.29
2019-04-15-10:15:15	0	6529815.08
2019-04-15-10:11:00	6529644809.999999	6529644.81
2019-04-15-10:14:15	0.7453333333445092	6529774.95
2019-04-15-10:10:00	0.6843789586059327	6529603.42
2019-04-15-10:13:15	0.7120000000111759	6529730.12
2019-04-15-10:09:00	0.7348381288979846	6529560.58
2019-04-15-10:12:15	0.772000000004967	6529686.29

Environment

• System information:

  Linux 3.10.0-327.el7.x86_64 x86_64

• Prometheus version:

prometheus, version 2.7.1 (branch: HEAD, revision: 62e591f)
build user: root@f9f82868fc43
build date: 20190131-11:16:59
go version: go1.11.5

• Alertmanager version:

alertmanager, version 0.15.2 (branch: HEAD, revision: d19fae3bae451940b8470abb680cfdd59bfa7cfa)
build user: root@3101e5b68a55
build date: 20180814-10:53:39
go version: go1.10.3

• Prometheus configuration file:

global:
  evaluation_interval: 15s
  scrape_interval: 30s
  scrape_timeout: 30s


remote_write:
  - queue_config:
      capacity: 100000
      max_samples_per_send: 1000
      max_shards: 10000
    url: http://xxxxx:8086/api/v1/prom/write?db=prometheus


remote_read:
  - read_recent: true
    url: http://xxxxx:8086/api/v1/prom/read?db=prometheus


rule_files:
  - /data/application/prometheus/rules/*.rules

alerting:
  alertmanagers:
  - path_prefix: /am
    scheme: http
    static_configs:
    - targets:
      - xxxxx:9093
      - yyyyy:9093


scrape_configs:
  - file_sd_configs:
    - files:
      - /data/application/prometheus/file_sd/nodes.json
      refresh_interval: 2m
    job_name: node
    metrics_path: /metrics
    relabel_configs:
    - regex: (.+):(.+)
      replacement: ${1}
      source_labels:
      - __address__
      target_label: instance_ip

[simonpasquier]

Thanks for your report. It looks as if this is actually a question about usage and not development.

To make your question, and all replies, easier to find, we suggest you move this over to our user mailing list, which you can also search. If you prefer more interactive help, join or our IRC channel, #prometheus on irc.freenode.net. Please be aware that our IRC channel has no logs, is not searchable, and that people might not answer quickly if they are busy or asleep. If in doubt, you should choose the mailing list.

Once your questions have been answered, please add a short line pointing to relevant replies in case anyone stumbles here via a search engine in the future.