Blackbox exporter resolves target IP for URL instead of using DNS name when using proxy for http probes

[nzjustin]

Blackbox exporter resolves the IP of the target and uses it in the target URL when sending the http request. e.g.:

ts=2020-09-01T22:52:54.530930926Z caller=http.go:318 module=http_2xxproxy target=https://www.mytarget.com/page.html level=info msg="Resolving target address" ip_protocol=ip6
ts=2020-09-01T22:52:54.534245845Z caller=http.go:318 module=http_2xxproxy target=https://www.mytarget.com/page.html level=info msg="Resolved target address" ip=10.0.75.56
ts=2020-09-01T22:52:54.534429186Z caller=client.go:250 module=http_2xxproxy target=https://www.mytarget.com/page.html level=info msg="Making HTTP request" url=https://10.0.75.56/page.html host=www.mytarget.com

This is a problem because if the proxy server has an allow list based on destination address then we would need to manually add all the possible IP addresses to the allow list for each DNS name we want to probe.

This seems like a bug, as I can't see why resolving the IP first would be desirable when using a proxy. Not sure though as this is seems to be the default behavior even when not using a proxy, so I assume there is a probable reason why the IP is resolved before http requests are made?

If this is not a bug, is there some other way to configure blackbox exporter to not resolve IP when doing a request via proxy?

Logs and relevant info is below (identifying info changed to protect the innocent). As you can see, the first thing blackbox exporter does is resolve the IP and then send the request using the IP instead of DNS name to the proxy server, which rejects it.

Host operating system: output of uname -a

Linux d9f1d017f71b 3.10.0-1127.13.1.el7.x86_64 #1 SMP Fri Jun 12 14:34:17 EDT 2020 x86_64 GNU/Linux

blackbox_exporter version: output of blackbox_exporter -version

blackbox_exporter, version 0.16.0

What is the blackbox.yml module config.

relevant module:

modules:
  http_2xx_proxy:
    http:
      proxy_url: http://proxy.server.address:3128
    prober: http

What is the prometheus.yml scrape config.

  - job_name: 'web'
    metrics_path: /probe
    params:
      module: [web]  # placeholder for relabeling
    file_sd_configs:
      - files:
          - 'targets/blackbox_http.json'
    relabel_configs:
      - source_labels: [module]
        target_label: __param_module
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox.exporter.host:19115

What logging output did you get from adding &debug=true to the probe URL?

Logs for the probe:
ts=2020-09-01T22:52:54.530735823Z caller=main.go:304 module=http_2xxproxy target=https://www.mytarget.com/page.html level=info msg="Beginning probe" probe=http timeout_seconds=9.5
ts=2020-09-01T22:52:54.530930926Z caller=http.go:318 module=http_2xxproxy target=https://www.mytarget.com/page.html level=info msg="Resolving target address" ip_protocol=ip6
ts=2020-09-01T22:52:54.534245845Z caller=http.go:318 module=http_2xxproxy target=https://www.mytarget.com/page.html level=info msg="Resolved target address" ip=10.0.75.56
ts=2020-09-01T22:52:54.534429186Z caller=client.go:250 module=http_2xxproxy target=https://www.mytarget.com/page.html level=info msg="Making HTTP request" url=https://10.0.75.56/page.html host=www.mytarget.com
ts=2020-09-01T22:52:54.534838323Z caller=main.go:119 module=http_2xxproxy target=https://www.mytarget.com/page.html level=error msg="Error for HTTP request" err="Get https://10.0.75.56/page.html: proxyconnect tcp: dial tcp :0: connect: connection refused"
ts=2020-09-01T22:52:54.534894887Z caller=main.go:119 module=http_2xxproxy target=https://www.mytarget.com/page.html level=info msg="Response timings for roundtrip" roundtrip=0 start=2020-09-01T22:52:54.534598896Z dnsDone=2020-09-01T22:52:54.534598896Z connectDone=2020-09-01T22:52:54.534793768Z gotConn=0001-01-01T00:00:00Z responseStart=0001-01-01T00:00:00Z end=0001-01-01T00:00:00Z
ts=2020-09-01T22:52:54.535027324Z caller=main.go:304 module=http_2xxproxy target=https://www.mytarget.com/page.html level=error msg="Probe failed" duration_seconds=0.004223545

What did you do that produced an error?

Specified target url using it's DNS name

What did you expect to see?

Blackbox exporter to make the http request using the target's DNS name in the request URL

What did you see instead?

Blackbox exporter resolved the target's IP and sent the http request using the targets IP in the destination URL instead of it's DNS name.

[brian-brazil]

Dupe of #264. The proxy_url is not a feature that is supported, it just happens to come with the http config library we use. I recommend running the blackbox exporter on the other side of your proxy.

[cameronkerrnz]

Please note that proxy_url is documented in https://github.com/prometheus/blackbox_exporter/blob/master/CONFIGURATION.md

If it is not supported, it probably should not be mentioned there.

Cheers,
Cameron

[brian-brazil]

It's there in case someone has an odd use for it and it is a valid config option. The HTTP module however is for testing HTTP, not testing HTTP proxies.

[cameronkerrnz]

Our use case is not to test a proxy; merely to test that a website we rely on is available. The only issue is that our proxy has ACLs based on source IP and destination domain; our standard security posture is that servers (including our monitoring servers) should go out via a whitelisted proxy; in this way we achieve some control over egress traffic from our data centre. In this use case, control over whether or not the result is IPv4 or IPv6 becomes much less relevant compared to the value in knowing that a cloud service we rely on has gone down.

On Thu, 3 Sep 2020 at 7:34 PM, Brian Brazil ***@***.***> wrote: It's there in case someone has an odd use for it. The HTTP module however is for testing HTTP, not testing HTTP proxies. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#687 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHFFTQZR4JVIH2GX4S4BFLSD5BIRANCNFSM4QSOWLEQ> . --

-- Cameron Kerr <cameron.kerr.nz@gmail.com> See my blog at http://distracted-it.blogspot.co.nz/ (previously http://humbledown.org/) Skype me on cameron.kerr.nz

[brian-brazil]

I'm not going to rehash arguments which have already been rehashed at length. #264 covers what solutions would be acceptable here.