Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prometheus 2.x inconsistent results through the time #3799

Closed
onorua opened this issue Feb 5, 2018 · 6 comments

Comments

Projects
None yet
2 participants
@onorua
Copy link

commented Feb 5, 2018

What did you do?
I do query:

curl 'http://prometheus-nodes-ca-ovh.afdevops.com:9090/api/v1/query?query=packetsByRange&start=2018-01-31T00:48:48Z&end=2018-01-31T01:18:48Z&step=30m'

What did you expect to see?
I expect result to be consistent, values should be either exists or does not exists all the time, because start and stop times never changed in the query.

What did you see instead? Under which circumstances?
I've got following:

MBP13:ansible-prometheus onorua$ curl 'http://prometheus-nodes-ca-ovh.afdevops.com:9090/api/v1/01-31T01:18:48Z&step=30m'e&start=2018-01-31T00:48:48Z&end=2018-0
{"status":"success","data":{"resultType":"vector","result":[]}}
MBP13:ansible-prometheus onorua$ curl 'http://prometheus-nodes-ca-ovh.afdevops.com:9090/api/v1/01-31T01:18:48Z&step=30m'e&start=2018-01-31T00:48:48Z&end=2018-0
{"status":"success","data":{"resultType":"vector","result":[]}}
MBP13:ansible-prometheus onorua$ curl 'http://prometheus-nodes-ca-ovh.afdevops.com:9090/api/v1/01-31T01:18:48Z&step=30m'e&start=2018-01-31T00:48:48Z&end=2018-0
{"status":"success","data":{"resultType":"vector","result":[]}}
MBP13:ansible-prometheus onorua$ curl 'http://prometheus-nodes-ca-ovh.afdevops.com:9090/api/v1/01-31T01:18:48Z&step=30m'e&start=2018-01-31T00:48:48Z&end=2018-0
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"packetsByRange","cluster":"checker-ny-1","colo":"checker","colo_type":"ny","container_name":"node-exporter","h":"23b6789","host_name":"abhi-checker-ny-1","host_type":"abhi-checker-ny","instance":
 ...

Important thing is, this metric is not regular, there is some tool Im not aware of which has metrics for 5 minutes and then 5 minutes there is no metrics:
screen shot 2018-01-31 at 5 33 12 pm

Environment
dockerized Prometheus 2.1.0

  • System information:
root@prometheus-nodes-ca-ovh:~# uname -srm
Linux 4.8.0-42-generic x86_64
  • Prometheus version:
prometheus, version 2.1.0 (branch: HEAD, revision: 85f23d82a045d103ea7f3c89a91fba4a93e6367a)
  build user:       root@6e784304d3ff
  build date:       20180119-12:01:23
  go version:       go1.9.2
  • Prometheus configuration file:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention=31d'
      - '--web.enable-admin-api'
      - '--storage.tsdb.max-block-duration=1d'
      - '--storage.tsdb.min-block-duration=2h'

config:

global:
  scrape_interval: 30s
  scrape_timeout: 20s
  evaluation_interval: 30s
  external_labels:
    prometheus: node
alerting:
  alertmanagers:
  - consul_sd_configs:
    - server: localhost:8500
      tag_separator: ','
      scheme: http
      services:
      - alertmanager
    scheme: http
    timeout: 10s
rule_files:
- /etc/prometheus/hss_recording.rules.yml
- /etc/prometheus/hss_alert.rules.yml
- /etc/prometheus/comscore_alert.rules.yml
- /etc/prometheus/css_alert.rules.yml
- /etc/prometheus/prometheus_alert.rules.yml
- /etc/prometheus/satellite_alert.rules.yml
- /etc/prometheus/efk_alert.rules.yml
- /etc/prometheus/recording.rules.yml
- /etc/prometheus/consul.rules.yml
- /etc/prometheus/storage_alert.rules.yml
- /etc/prometheus/bastion_alert.rules.yml
scrape_configs:
- job_name: efk-services
  scrape_interval: 30s
  scrape_timeout: 20s
  metrics_path: /metrics
  scheme: http
  consul_sd_configs:
  - server: localhost:8500
    tag_separator: ','
    scheme: http
    services:
    - node-exporter
  • Logs:
    nothing related to this

@onorua onorua changed the title Prometheus 2.1.0 inconsistent results through the time Prometheus 2.x inconsistent results through the time Feb 5, 2018

@onorua

This comment has been minimized.

Copy link
Author

commented Feb 5, 2018

we have the same behavior with 2.0 as well as 2.1 Prometheus servers.

@simonpasquier

This comment has been minimized.

Copy link
Member

commented Feb 5, 2018

Can you share the Prometheus configuration file? Also it is better to ask such questions on the Prometheus users mailing list.

@onorua

This comment has been minimized.

Copy link
Author

commented Feb 5, 2018

that is a bug report not a question

@simonpasquier

This comment has been minimized.

Copy link
Member

commented Feb 5, 2018

I'm not familiar with the Consul SD but it looks like the Prometheus targets come and go. I would recommend to check the Targets page in the Prometheus UI and see if the expected endpoints are there while the metrics are missing.
Anything special in the Prometheus logs BTW?

@onorua

This comment has been minimized.

Copy link
Author

commented Feb 5, 2018

Im really sorry to bother, it seems my colleague convinced me that the query is correct, while the proper one should be:

http://prometheus-nodes-ca-ovh.afdevops.com:9090/api/v1/query_range?query=packetsByRange&start=1517833500.684&end=1517840700.684&step=28

closing as invalid.

@onorua onorua closed this Feb 5, 2018

@lock

This comment has been minimized.

Copy link

commented Mar 22, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Mar 22, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.