Empty result set when querying with high step values on timestamped data.

[mar-v-in]

What did you do?

• 1. Creating a large dataset with timestamped historic data using the following statement (run in bash):

    echo -e "HTTP/1.1 200 OK\nContent-Type: text/plain; version=0.0.4\n\n"$(for m in `seq 8`; do for d in `seq 27`; do for h in `seq 4 23`; do echo -n "testval $h "$(date --date "$m/$d/2016 $h:22" +%s)"000\n"; done; done; done) | nc -l -p 8018

• 2. Start prometheus using large retention (prometheus -storage.local.retention 8640h) and a config that scrapes from localhost:8018.
• 3. After successful scraping, query prometheus for the data with different step values:

What did you expect to see?
http://127.0.0.1:9090/api/v1/query_range?query=testval&start=1470441600&end=1471046400&step=300 -> 139 values
http://127.0.0.1:9090/api/v1/query_range?query=testval&start=1470441600&end=1471046400&step=600 -> ~70 values

What did you see instead? Under which circumstances?
http://127.0.0.1:9090/api/v1/query_range?query=testval&start=1470441600&end=1471046400&step=300 -> 139 values
http://127.0.0.1:9090/api/v1/query_range?query=testval&start=1470441600&end=1471046400&step=600 -> no values

Environment

• System information:

  Linux 4.6.6-301.pf.skylake.fc24.x86_64 x86_64

• Prometheus version:

  prometheus, version 1.0.2 (branch: master, revision: d9b4079)
  build user: root@60e6cc1b2922
  build date: 20160824-11:10:31
  go version: go1.6.2

• Prometheus configuration file:

global:
  scrape_interval: 60s
  evaluation_interval: 60s
  scrape_timeout: 10s

scrape_configs:
  - job_name: self
    scrape_interval: 10s
    static_configs:
      - targets: ['localhost:9090']
  - job_name: test
    static_configs:
      - targets: ['localhost:8018']

Note
I don't know if what I do is actually intended to work, but the results are somewhat wrong anyway.

[brian-brazil]

You're running into staleness issues. When you make a request for $h:30 the last data point is 8 minutes away, so not returned. You probably want query with a range vector rather than query_range

[mar-v-in]

Indeed, that did the job.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.