Unusual peak in recording rule for rate

[arunkumarpro1]

Bug Report

What did you do?
Used a recording rule as shown below:

  rules:
  - record: Rate_Interface_statistics
    expr: rate (Interface_statistics{map_key='rx_packets'}[20s])
    labels:
      TimeInterval: 20s
      map_key: rx_packets

What did you expect to see?

The metric in the recording rule is nearly constant around a particular value (around 15 k). Expected to see graph around a constant value in the same way as I see when I query the underlying expression(rate (Interface_statistics{map_key='rx_packets'}[20s])).

What did you see instead? Under which circumstances?

Instead, I see an unusual peak (around 164 M - PS: This peak is always around 164M. I am not sure if there is any significance for this value) at a particular time for the recording rule. But, I can see the constant graph when I query the expression directly.

Result from Recording rule

Result from Actual Expression

Environment

• System information:

  Linux 4.4.0-134-generic x86_64

• Prometheus version:

  prometheus, version 2.3.2 (branch: , revision: ) build user: build date: go version: go1.10.2

• Prometheus configuration file:

global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 5s
  external_labels:
    monitor: codelab-monitor
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - 127.0.0.1:9091
    scheme: http
    timeout: 10s
rule_files:
- /var/run/prometheus/alerts/*.yml
scrape_configs:
- job_name: prometheus
  scrape_interval: 5s
  scrape_timeout: 5s
  metrics_path: /metrics
  scheme: http
  static_configs:
  - targets:
    - 127.0.0.1:9091

[arunkumarpro1]

Any help with this issue?

[simonpasquier]

It can be a counter value that reset briefly to 0. To get more insights you would need to share the raw samples around the blip.

In general such questions are better answered on our user mailing list, which you can also search.