Prometheus 2.x inconsistent results through the time

[onorua]

What did you do?
I do query:

curl 'http://prometheus-nodes-ca-ovh.afdevops.com:9090/api/v1/query?query=packetsByRange&start=2018-01-31T00:48:48Z&end=2018-01-31T01:18:48Z&step=30m'

What did you expect to see?
I expect result to be consistent, values should be either exists or does not exists all the time, because start and stop times never changed in the query.

What did you see instead? Under which circumstances?
I've got following:

MBP13:ansible-prometheus onorua$ curl 'http://prometheus-nodes-ca-ovh.afdevops.com:9090/api/v1/01-31T01:18:48Z&step=30m'e&start=2018-01-31T00:48:48Z&end=2018-0
{"status":"success","data":{"resultType":"vector","result":[]}}
MBP13:ansible-prometheus onorua$ curl 'http://prometheus-nodes-ca-ovh.afdevops.com:9090/api/v1/01-31T01:18:48Z&step=30m'e&start=2018-01-31T00:48:48Z&end=2018-0
{"status":"success","data":{"resultType":"vector","result":[]}}
MBP13:ansible-prometheus onorua$ curl 'http://prometheus-nodes-ca-ovh.afdevops.com:9090/api/v1/01-31T01:18:48Z&step=30m'e&start=2018-01-31T00:48:48Z&end=2018-0
{"status":"success","data":{"resultType":"vector","result":[]}}
MBP13:ansible-prometheus onorua$ curl 'http://prometheus-nodes-ca-ovh.afdevops.com:9090/api/v1/01-31T01:18:48Z&step=30m'e&start=2018-01-31T00:48:48Z&end=2018-0
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"packetsByRange","cluster":"checker-ny-1","colo":"checker","colo_type":"ny","container_name":"node-exporter","h":"23b6789","host_name":"abhi-checker-ny-1","host_type":"abhi-checker-ny","instance":
 ...

Important thing is, this metric is not regular, there is some tool Im not aware of which has metrics for 5 minutes and then 5 minutes there is no metrics:

Environment
dockerized Prometheus 2.1.0

• System information:

root@prometheus-nodes-ca-ovh:~# uname -srm
Linux 4.8.0-42-generic x86_64

• Prometheus version:

prometheus, version 2.1.0 (branch: HEAD, revision: 85f23d82a045d103ea7f3c89a91fba4a93e6367a)
  build user:       root@6e784304d3ff
  build date:       20180119-12:01:23
  go version:       go1.9.2

• Prometheus configuration file:

      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention=31d'
      - '--web.enable-admin-api'
      - '--storage.tsdb.max-block-duration=1d'
      - '--storage.tsdb.min-block-duration=2h'

config:

global:
  scrape_interval: 30s
  scrape_timeout: 20s
  evaluation_interval: 30s
  external_labels:
    prometheus: node
alerting:
  alertmanagers:
  - consul_sd_configs:
    - server: localhost:8500
      tag_separator: ','
      scheme: http
      services:
      - alertmanager
    scheme: http
    timeout: 10s
rule_files:
- /etc/prometheus/hss_recording.rules.yml
- /etc/prometheus/hss_alert.rules.yml
- /etc/prometheus/comscore_alert.rules.yml
- /etc/prometheus/css_alert.rules.yml
- /etc/prometheus/prometheus_alert.rules.yml
- /etc/prometheus/satellite_alert.rules.yml
- /etc/prometheus/efk_alert.rules.yml
- /etc/prometheus/recording.rules.yml
- /etc/prometheus/consul.rules.yml
- /etc/prometheus/storage_alert.rules.yml
- /etc/prometheus/bastion_alert.rules.yml
scrape_configs:
- job_name: efk-services
  scrape_interval: 30s
  scrape_timeout: 20s
  metrics_path: /metrics
  scheme: http
  consul_sd_configs:
  - server: localhost:8500
    tag_separator: ','
    scheme: http
    services:
    - node-exporter

• Logs:
  nothing related to this

[onorua]

we have the same behavior with 2.0 as well as 2.1 Prometheus servers.

[simonpasquier]

Can you share the Prometheus configuration file? Also it is better to ask such questions on the Prometheus users mailing list.

[onorua]

that is a bug report not a question

[simonpasquier]

I'm not familiar with the Consul SD but it looks like the Prometheus targets come and go. I would recommend to check the Targets page in the Prometheus UI and see if the expected endpoints are there while the metrics are missing.
Anything special in the Prometheus logs BTW?

[onorua]

Im really sorry to bother, it seems my colleague convinced me that the query is correct, while the proper one should be:

http://prometheus-nodes-ca-ovh.afdevops.com:9090/api/v1/query_range?query=packetsByRange&start=1517833500.684&end=1517840700.684&step=28

closing as invalid.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.