Instant Query not returning the right value

[arnaudlemaignen]

Bug Report

What did you do?
Launch a correct instant query
What did you expect to see?
Always return a value (that should be the last one)
What did you see instead? Under which circumstances?
it returns no data sometimes

My case is not really standard.
I am querying a DB every 20 min (scrape_interval) with a timeout bit less than 20 min .
Those queries are really long to execute so I can not afford to query more frequently and/or decrease the timeout.

So I get 1 point every 20 min but when I am launching the instant query I have access to the real value only for 5 min, the 15min after it returns "no data"

You can see this in the image below that data are disconnected

Environment

• System information:

  Linux 3.10.0-862.el7.x86_64 x86_64

• Prometheus version:

prometheus, version 2.3.1 (branch: HEAD, revision: 188ca45)
build user: root@82ef94f1b8f7
build date: 20180619-15:56:22
go version: go1.10.3

• Prometheus configuration file:

global:
  scrape_interval: 30s
  scrape_timeout: 10s
  evaluation_interval: 30s
...
- job_name: db-exporter
  scrape_interval: 20m
  scrape_timeout: 1150s
  metrics_path: /metrics
  scheme: http
  static_configs:
  - targets:
    - platform-db-exporter:9285

[simonpasquier]

The maximum recommended value for scrape_interval is 2m because of staleness.

See https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness