State: DOWN, Error: context deadline exceeded

[dakala]

What did you do?

The metrics endpoint of my web application seems to be working. I can curl it and get good data. I've added my configuration to prometheus.yml

What did you expect to see?

I expect to see the metrics Prometheus has scrapped.

What did you see instead? Under which circumstances?

When I view the Prometheus targets page, my endpoint is listed but it's reported as down with the error, "context deadline exceeded".

Environment

• System information:

  Darwin 15.6.0 x86_64

• Prometheus version:

  prometheus, version 1.3.1 (branch: master, revision: be47695)
  build user: root@37f0aa346b26
  build date: 20161104-20:27:37
  go version: go1.7.3

• Alertmanager version:

  insert output of alertmanager -version here (if relevant to the issue)

• Prometheus configuration file:

# my global config
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
  evaluation_interval: 15s # By default, scrape targets every 15 seconds.
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: 'codelab-monitor'

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first.rules"
  # - "second.rules"
  - 'prometheus.rules'

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'my-job'
    scrape_interval: 20s
    static_configs:
      - targets: ['test-site.dev']
        labels:
          group: 'my-job-test'

  - job_name:       'example-random'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s

    static_configs:
      - targets: ['localhost:8080', 'localhost:8081']
        labels:
          group: 'production'

      - targets: ['localhost:8082']
        labels:
          group: 'canary'

Thanks for the attention.

[juliusv]

Does curl work from the Prometheus node as well? If so, try using tcpdump or Wireshark to find out what request Prometheus is sending. Something must be off.

[dakala]

@juliusv Thanks for your response but how do I check whether curl works from the Prometheus node? I'm working with only the Prometheus server. Hope I haven't missed out anything. Once again thanks.

[juliusv]

You mean you already did the curl from the Prometheus server's node? If so, the Prometheus must be doing something differently when sending the request. Then I would suggest using tcpdump or Wireshark on either the Prometheus node or the node where the service is to see what is going on on the wire.

[dakala]

Sorry but this is my first time of using a tool like Wireshark so bear with me. When I captured from Loopback lo0 and filtered on "http", I saw requests from the Prometheus endpoint (localhost:9090/metrics) and my application (promtest.dev/metrics) - I opened both in my browser. No scraping requests from Prometheus to my promtes.dev showed up.

With my application, I got this from the prometheus endpoint:

# TYPE prometheus_target_scrape_pool_sync_total counter
prometheus_target_scrape_pool_sync_total{scrape_job="my-job"} 1
# HELP prometheus_target_sync_length_seconds Actual interval to sync the scrape pool.
# TYPE prometheus_target_sync_length_seconds summary
prometheus_target_sync_length_seconds{scrape_job="my-job",quantile="0.01"} 0.00011974
prometheus_target_sync_length_seconds{scrape_job="my-job",quantile="0.05"} 0.00011974
prometheus_target_sync_length_seconds{scrape_job="my-job",quantile="0.5"} 0.00011974
prometheus_target_sync_length_seconds{scrape_job="my-job",quantile="0.9"} 0.00011974
prometheus_target_sync_length_seconds{scrape_job="my-job",quantile="0.99"} 0.00011974
prometheus_target_sync_length_seconds_sum{scrape_job="my-job"} 0.00011974
prometheus_target_sync_length_seconds_count{scrape_job="my-job"} 1

I changed the target to "foo.dev" (which doesn't exist), restarted Prometheus and got exactly the same output as above.

[grobie]

If you still have issues with this, send a question to the prometheus-users https://groups.google.com/forum/#!forum/prometheus-users mailing list.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.