Average value not calculated right?

[slavede]

What did you do?
Tried to get average value

What did you expect to see?
Correctly calculated value

What did you see instead? Under which circumstances?
Wrong value

Environment

• System information:

Linux 2.6.32-358.el6.x86_64 x86_64

• Prometheus version:

  build user: root@3e392b8b8b44
  build date: 20160916-11:36:30
  go version: go1.6.3

• Alertmanager version:

  insert output of alertmanager -version here (if relevant to the issue)

• Prometheus configuration file:

# my global config
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
  evaluation_interval: 15s # By default, scrape targets every 15 seconds.
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: 'codelab-monitor'

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first.rules"
  # - "second.rules"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ['localhost:9091']

• Alertmanager configuration file:

• Logs:

insert Prometheus and Alertmanager logs relevant to the issue here

Similar question here:
http://stackoverflow.com/questions/39831998/how-does-prometheus-db-calculate-average-value

[brian-brazil]

Can you indicate how the value is wrong, what value you expected and what the input data is?

[slavede]

So, as you can see from the screen shot, last value is 60, the value before that it is 67. How is the average 66.4166 (based on the response screenshot)? I expected to be last value or the average of last two values (not still sure how Prometheus is calculating it)

[juliusv]

@slavede What you see in the graph are 60s-spaced interpolated query evaluation points. Not the raw samples in the database. The range vector that you input into avg_over_time() will extract all raw samples under the 1m time window and average over them. Probably you are collecting data at a higher frequency than your graph resolution is set to. It would make more sense to compare the following two instant queries:

• current_temp[1m] - this will give you the raw datapoints collected in the last 1m
• avg_over_time(current_temp[1m]) - this will average over the raw datapoints collected in the last 1m

Of course, then @brian-brazil might still say that one shouldn't average over temperatures at all ;)

[slavede]

Cool, thanks, nice explanation.

Then, if I do:

/api/v1/query_range?query=avg_over_time(current_temp[5m])&start=1475483802.739&end=1475498202.739&step=600&_=1475493021942

This would do the following:

Each 600 seconds read current temperature for the last 5 minutes (every point you have) and calculate average temperature, and do this between 1475483802.739 and 1475498202.739?

[juliusv]

The language in your last sentence is slightly ambiguous, so depending on what exactly you mean, it's correct or incorrect :)

Let me rephrase it: starting at timestamp 1475483802.739 and ending at timestamp 1475498202.739, the query "avg_over_time(...)" gets executed at regular instants that are 600 seconds apart. At every such instant, Prometheus calculates the average over all sample values (within each series) stretching back 5 minutes from that instant. This produces the output sample value for that instant.

Note that some samples are skipped completely, since your time averaging time window is 5 minutes, but your query resolution step is 10 minutes (600s).

[slavede]

Yeah, that's what I meant :). Thanks, it's clear explanation.

And yes, they are skipped, I've just put it there as an example.

Thank you very much for your help.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.