Connect request to binding IP:PORT (>= v2.0 related)

[ghost]

What did you do?

This issue is solved for me, but let's say I would like to document it for others (and maybe it's a real bug, don't know).

I'm using prometheus in EC2 environment with EC2 instance discovery. The problem is, that network guys require that all the external http traffic goes trough the http proxy. So I get this requirement covered by exporting env proxy variables which native go http client will pickup:

HTTP_PROXY=http://proxyserver:1.2.3.4.5
HTTPS_PROXY=http://proxyserver:1.2.3.4.5
NO_PROXY=169.254.169.254

I keep the 169.254.169.254 there so the AWS lib can pickup the correct role and credentials for discovery.

This works fine with v1.7. After upgrading to 2.0 I've noticed a lot of failed connections to proxy. Surprisingly everything else worked fine including the discovery. Cutting the config down to empty global: section, I still saw the traffic. Quick strace shows that the prometheus main thread tries to repeatedly connect to 0.0.0.0:9090, which is as default binding i think, and since the HTTP_PROXY is configured, it tries to do it trough the proxy (bummer).

So the solution was to include 0.0.0.0 in no_proxy (silly, but fixes the issue):

NO_PROXY=169.254.169.254,0.0.0.0

What did you expect to see?

Close to no traffic to http proxy. Ideally no connect to binding address:port.

What did you see instead? Under which circumstances?

High volumes of http traffic to proxy. Attemps to connect to binding address:port.

Environment

• System information:

Linux 4.14.0-2-amd64 x86_64

• Prometheus version:

prometheus, version 2.1.0 (branch: HEAD, revision: 85f23d8)
build user: root@6e784304d3ff
build date: 20180119-12:01:23
go version: go1.9.2

• Prometheus configuration file:

global:

[simonpasquier]

Thanks for the report, it is indeed a bug. #4572 should fix it.