Prometheus query API failed to get data

[changsijay]

What did you do?
Use grafana to display chart.

What did you expect to see?
display properly on any time range

What did you see instead? Under which circumstances?
Sometimes the chart display correct but most of time it show no data points

Environment
Ubuntu 16.04

• System information:
  Linux 3.10.0-693.2.2.el7.x86_64 x86_64

• Prometheus version:
  prometheus, version 2.0.0 (branch: HEAD, revision: 0a74f98)
  build user: root@615b82cb36b6
  build date: 20171108-07:11:59
  go version: go1.9.2

• Prometheus configuration file:

# my global config
global:
  scrape_interval:     30m # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 30m # Evaluate rules every 15 seconds. The default is every 1 minute.

scrape_configs:
  - job_name: 'device_exporter'
    scrape_interval: 30m
    static_configs:
        - targets: ['my_host2:9002']

I use my own python exporter to export for current_device_count metrics every 30 minutes.
Then I add prometheus datasource with Scrape interval as 30 minutes

I created a dashboard with query sum(current_device_count)
The chart sometimes display properly, but maybe several minutes later, I refresh it, and became no data points.

I debug with the query URL:

which works fine, which range is 30days

http://my_host2:9090/api/v1/query_range?query=sum(current_device_count)&start=1513501507&end=1516064707&step=1800

some minutes later,
which failed to get data points, which range is 30days, too

http://my_host2:9090/api/v1/query_range?query=sum(current_device_count)&start=1513502383&end=1516094383&step=1800

If I change the failed case with step=300 I can get 628 items, I tried 600, 1200 are failed.

http://my_host2:9090/api/v1/query_range?query=sum(current_device_count)&start=1513502383&end=1516094383&step=300

I am not sure this is a bug or settings wrong.

[changsijay]

After writing some test script, I found the query only works every 30 minutes.
for example: 08:03, 08:33, 09:03, 09:33... etc

How could this happen?

[brian-brazil]

2m is the max safe scrape interval.

It makes more sense to ask questions like this on the prometheus-users mailing list rather than in a GitHub issue. On the mailing list, more people are available to potentially respond to your question, and the whole community can benefit from the answers provided.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.