"No datapoints found." using rate on node_cpu

[pwaller]

prometheus, version 0.16.2 (branch: stable, revision: 287d9b2)
  build user:       @ee15eaddc546
  build date:       20160223-19:12:50
  go version:       1.5.3

I'm pretty convinced everything is working, except that I can't get the rate() function to work. Here I pick a specific node_cpu and try and try and compute the rate:

And the rate:

Am I doing something obviously wrong? I found #1022 but checking the clocks from the host running prometheus, the prometheus container and my browser and they all agree to within a second.

rate(node_cpu[1m]) also returns no data, whereas node_cpu by itself returns lots of data.

[brian-brazil]

Eyeballing that data, you have a scrape interval of 1m so rate(node_cpu[1m]) will often not have two data points to work with. I'd suggest using rate(node_cpu[5m]) instead.

[pwaller]

I think I'm coming to understand the problem. I thought I had a 1s scrape interval, but my scrape interval was only applying to the first scrape config and not the later ones.

How is the default scrape interval determined? I thought it was 10s, judging from the docs?

[pwaller]

Sorry. PEBKAC. I was reading the default scrape timeout, not the scrape interval. All solved!

The only slightly unfortunate thing was just how mysterious it was when I got no data back.

[brian-brazil]

Glad you're sorted out.

[pwaller]

Thanks for the exceptional response time 👍 👏

[davidlwillson]

I'll suggest that it would be cool if Prometheus could detect the problem and say something like: You're trying for every 30s, but your samples seem to be every 60s. Decrease the granularity of your report, or increase the granularity of your samples.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.