Context Deadline Exceeded

[SRAM85]

What did you do?
Using JMX_exporter to push metrics from baremetal serves to prometheus which is hosted in another baremetal server. we graph the metrics in grafana.

What did you expect to see?
Up and running targets with metrics information being scrapped

What did you see instead? Under which circumstances?
Targets for some jobs are up but the targets for jobs scrapping kafka metrics is down with "CONTEXT DEADLINE EXCEEDED" . i am not able to see metrics in grafana. Increasing the number of chunks or the memory of the jmx-exporter did not help. Restarting Prometheus did not help

Environment

• System information:

Linux 3.10.0-327.10.1.el7.x86_64 x86_64

• Prometheus version:

prometheus-1.5.2.linux-amd64

• Prometheus configuration file:

global:
  scrape_interval:     60s # By default, scrape targets every 15 seconds.
  evaluation_interval: 60s # By default, scrape targets every 15 seconds.

  external_labels:
      monitor: 'codelab-monitor'
  - job_name: 'Infrastructure_PROD_DFW'
    static_configs:
    - targets: ['bm1:10001','bm2:10001','bm3:10001','bm4:10001','bm5:10001','bm6:10001']
  
  - job_name: 'Infrastructure_PROD_DAL'
    static_configs:
    - targets: ['bm7:10001','bm8:10001','bm9:10001','bm10:10001','bm11:10001','bm12:10001']

  - job_name: 'Burrow'
    static_configs:
    - targets: ['bm13:9091'] 

• Logs:

context deadline exceeded

[juliusv]

context deadline exceeded indicates an timeout error pulling from the JMX exporter. Can you reach the /metrics endpoint on your JMX exporter successfully manually? Can you also reach it from the same machine that Prometheus runs on? How long does it take to load?

[SRAM85]

I am able to reach the endpoint /metrics and it takes like 5s -10s to load.

[juliusv]

@SRAM85 The default scrape timeout is 10 seconds - maybe sometimes it takes longer than that and that is the problem? You could try increasing it:

global:
  scrape_timeout: 30s
  ...

(or per scrape config)

[srikanthdixit]

Hello julisuv,

Below is my prometheus.yml configuration file.

my global config

global:
scrape_interval: 15s # By default, scrape targets every 15 seconds.
evaluation_interval: 15s # By default, scrape targets every 15 seconds.

scrape_timeout is set to the global default (10s).

Attach these labels to any time series or alerts when communicating with

external systems (federation, remote storage, Alertmanager).

external_labels:
monitor: 'codelab-monitor'

Load and evaluate rules in this file every 'evaluation_interval' seconds.

rule_files:

- "first.rules"

- "second.rules"

A scrape configuration containing exactly one endpoint to scrape:

Here it's Prometheus itself.

scrape_configs:

The job name is added as a label job=<job_name> to any timeseries scraped from this config.

• job_name: 'prometheus'

  Override the global default and scrape targets from this job every 5 seconds.

  scrape_interval: 5s

  metrics_path defaults to "/metrics"

scheme defaults to "http".

target_groups:
  - targets:
      - "10.9.64.47:9100"
      - "10.9.64.199:9100"
      - "10.9.64.212:9100"
      - "10.9.64.38:9100"
      - "10.9.64.40:9100"
      - "10.9.64.42:9100"
      - "10.9.64.45:9100"
      - "10.93.1.142:9100"
      - "10.93.1.144:9100"
      - "10.93.1.149:9100"

Issue: when i try to "wget" from promotheus server using "[root@1055401lvapp004 prometheus-0.18.0.linux-amd64]# wget http://10.9.64.47:9100/metrics
--2017-06-07 16:27:27-- http://10.9.64.47:9100/metrics
Connecting to 10.9.64.47:9100... connected.
HTTP request sent, awaiting response...

seems like request is sent to destination server but it didn't get respond back. Moreover, when i try to use /metrics123 or xyz i could see the request and respond successfully but not for /metrics.
here issue with /metrics service. In prometheus dashboard i could see all the endpoints are showing "DOWN" Could you please let me know what might be the issue with.

[xyr115]

Any movement on this?

[weskinner]

The fix for me was editing the NetworkPolicy associated with the Pod prometheus was trying to scrape.