HTTP request sent, awaiting response in Prometheus.

[srikanthdixit]

Hello,

This is my prometheus configuration file.

my global config

global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

scrape_timeout is set to the global default (10s).

Attach these labels to any time series or alerts when communicating with

external systems (federation, remote storage, Alertmanager).

external_labels:
monitor: 'codelab-monitor'

Load rules once and periodically evaluate them according to the global 'evaluation_interval'.

rule_files:

"first.rules"
"second.rules"
A scrape configuration containing exactly one endpoint to scrape:

Here it's Prometheus itself.

scrape_configs:

The job name is added as a label job=<job_name> to any timeseries scraped from this config.

job_name: 'prometheus'

metrics_path defaults to '/metrics'

scheme defaults to 'http'.

static_configs:

targets:
"10.93.1.134:9100"
"10.9.64.47:9100"

Issue No 1: When i try to wget from prometheus server to destination server. Seems like request has been sent but response is not getting. Below is the reference.

wget http://10.9.64.47:9100/metrics
--2017-06-08 13:17:09-- http://10.9.64.47:9100/metrics
Connecting to 10.9.64.47:9100... connected.
HTTP request sent, awaiting response...

Issue No 2: In prometheus dashboard i can see the target servers are showing DOWN not sure what would be the issue. I might be doing something wrong. Any help would be greatly appreciated. Please find attached screeen shot is what i'm facing.

[ghost]

Hello

Could you please provide any update on this

[ev1lm0nk3y]

Does the endpoint http://10.9.64.47:9100/metrics work from other servers, or even from that server itself (i.e. on 10.9.64.47 try running wget http://localhost:9100/metrics)? It looks like that server is listening on that port, but maybe taking a LONG time (>10s) to respond. If you see this result from multiple sources, your endpoint may not like the request for some reason.

As for Issue No. 2, if prometheus isn't able to get a response from your target(s) in the configured timeout, it will assume they are DOWN. Effectively your issues are linked; targets hang when asked for metrics and prometheus, rightfully, believes they aren't working.

[juliusv]

@srikanth789 You mention that wget http://10.9.64.47:9100/metrics also does not work, so it seems the problem is a network reachability problem and not related to the Prometheus server. In that case, Prometheus showing the targets as DOWN is expected.

Thus, closing the issue here.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.