Faster query processing using old expression than new one with recorded rule

[noushi]

What did you do?
To speed up console node list, I created a record rule:

node:cpu_usage:cur_avg = 100 * (1 - avg without(cpu, mode) (irate(node_cpu{job='node',mode='idle'}[5m])))
node:cpu_iowait:cur_avg = 100 * (avg without(cpu, mode) (irate(node_cpu{mode='iowait'}[5m])))

Then I changed node.html to use the new metrics.

What did you expect to see?

I expected to see a speed improvement, which I did see:

• page load went from 18secs to <12secs

What did you see instead? Under which circumstances?

• with the rule enabled, node.html is faster when using unchanged metrics (~7secs consistently) than when using recorded metrics (~10-12secs)

Environment

• System information:

$ uname -srm
Linux 3.19.0-61-generic x86_64`

• Prometheus version:
  I'm using a Docker container.

From /status:

Version     0.20.0
Revision    aeab25c

• Prometheus configuration file:

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

rule_files:
   - rules/node_speedup.rules

scrape_configs:

  - job_name: 'node'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s

    static_configs:
        <node_list>
        ...

• Logs:

I didn't see any relevant entries.

[fabxc]

Can you provide a diff of what you changed in the node template? This of course seems rather odd.

[noushi]

The meaningful difference is when I use

100 * (1 - avg without(cpu, mode) (irate(node_cpu{job='node',mode='idle'}[5m])))
100 * (avg without(cpu, mode) (irate(node_cpu{mode='iowait'}[5m])))

vs the corresponding recorded rules.

Following my discussion with @brian-brazil in #1832 , this turned out to be a good thing since the recorded rules are only updated once per minute, and I need the most recent results.

So I left the original expressions in node.html, and used recorded rules, which got me fresh results in less time.

This "feature" is probably not intended.

[beorn7]

Is this still a problem?

[brian-brazil]

Presuming it's still not an issue.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.