EC2 Service discovery not working behind proxy

[adamron1]

What did you do?
Good question. It all began when we tried to set up prometheus in a private subnet in AWS. We require a proxy to get access to the outside world, but using the ec2 sd does not respect the environment variables set up for this. Can you imagine!

What did you expect to see?
A list of ec2 instances appearing in the targets page in prometheus.

What did you see instead? Under which circumstances?
Nothing!!! Checking the logs, we get a timeout when querying ec2. It seems as though it's not finding it. Sad face.

Environment

• System information:

  Linux 3.13.0-93-generic x86_64

• Prometheus version:

  prometheus, version 1.0.1 (branch: master, revision: be40190)
  build user: root@e881b289ce76
  build date: 20160722-20:00:12
  go version: go1.6.2

• Prometheus configuration file:

global:
  scrape_interval: "15s"     # By default, scrape targets every 15 seconds.
  evaluation_interval: "15s" # By default, evaluate rules every 15 seconds.

scrape_configs:
- job_name: "prometheus"
  scrape_interval: "15s"
  metrics_path: "/metrics"
  static_configs:
    - targets: ["localhost:9090"]
- job_name: "node_exporter"
  scrape_interval: "15s"
  metrics_path: "/metrics"
  honor_labels: true
  ec2_sd_configs:
  - region: us-west-2
    port: 9100

• Logs:

ubuntu@ip-10-161-71-70:~$ tail -f /var/log/prometheus/prometheus.log
time="2016-09-23T01:55:42Z" level=info msg="Done checkpointing fingerprint mappings in 2.021242ms." source="persistence.go:1394"
time="2016-09-23T01:55:42Z" level=info msg="Local storage stopped." source="storage.go:406"
time="2016-09-23T01:55:43Z" level=info msg="Starting prometheus (version=1.1.0, branch=master, revision=5ee84a96db6190d4fcdaf4eff74a09b52824a9aa)" source="main.go:73"
time="2016-09-23T01:55:43Z" level=info msg="Build context (go=go1.6.3, user=root@54c6975115bb, date=20160903-19:01:53)" source="main.go:74"
time="2016-09-23T01:55:43Z" level=info msg="Loading configuration file /opt/prometheus/prometheus.yml" source="main.go:221"
time="2016-09-23T01:55:43Z" level=info msg="Loading series map and head chunks..." source="storage.go:358"
time="2016-09-23T01:55:43Z" level=info msg="418 series loaded." source="storage.go:363"
time="2016-09-23T01:55:43Z" level=warning msg="No AlertManagers configured, not dispatching any alerts" source="notifier.go:176"
time="2016-09-23T01:55:43Z" level=info msg="Listening on :9090" source="web.go:233"
time="2016-09-23T01:55:43Z" level=info msg="Starting target manager..." source="targetmanager.go:75"
0x451c90
0x451c90
0x451c90
time="2016-09-23T01:57:43Z" level=error msg="could not describe instances: RequestError: send request failed\ncaused by: Post https://ec2.us-west-2.amazonaws.com/: dial tcp 54.240.251.131:443: i/o timeout" source="ec2.go:80"
0x451c90
0x451c90
0x451c90
time="2016-09-23T01:59:43Z" level=error msg="could not describe instances: RequestError: send request failed\ncaused by: Post https://ec2.us-west-2.amazonaws.com/: dial tcp 54.240.253.45:443: i/o timeout" source="ec2.go:90"

[adamron1]

Addressed in #2027

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.