Non negligible divergence between recorded metric and original expression

[noushi]

What did you do?
To speed up console node list, I created a record rule:

node:cpu_iowait:cur_avg = 100 * (avg without(cpu, mode) (irate(node_cpu{mode='iowait'}[5m])))

What did you expect to see?
I expected the recorded metric to be almost identical to the original expression.

What did you see instead? Under which circumstances?
Instead, graphing

abs(100 * (avg without(cpu, mode) (irate(node_cpu{mode='iowait'}[5m]))) - node:cpu_iowait:cur_avg)

shows a divergence of more than 20% on multiple occasions:

DISCLAIMER: The server showing the highest divergence is one our most busiest.

Environment

• System information:

$ uname -srm
Linux 3.19.0-61-generic x86_64`

• Prometheus version:
  I'm using a Docker container.

From /status:

Version     0.20.0
Revision    aeab25c

• Prometheus configuration file:

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

rule_files:
   - rules/node_speedup.rules

scrape_configs:

  - job_name: 'node'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s

    static_configs:
        <node_list>
        ...

• Logs:

I didn't see any relevant entries.

[brian-brazil]

Given the default eval interval is 1m and you're scraping every 15s, this is an expected result when using irate. Try rate instead.

[noushi]

Thank you Brian, I'lll close this issue as I'm not working on it anymore.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.