Can't bind to localhost or 127.0.0.1, service is always visible to the outside

[Mindtoeye]

Host operating system: output of uname -a

Linux 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30) x86_64 GNU/Linux

blackbox_exporter version: output of blackbox_exporter --version

blackbox_exporter, version 0.19.0 (branch: HEAD, revision: 5d575b8)
build user: root@2b0258d5a55a
build date: 20210510-12:56:44
go version: go1.16.4
platform: linux/amd64

What is the blackbox.yml module config.

modules:
  http_prometheus:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2"]
      method: GET
      fail_if_ssl: false
      fail_if_not_ssl: true
      tls_config:
        insecure_skip_verify: true

What is the prometheus.yml scrape config.

What logging output did you get from adding &debug=true to the probe URL?

From /var/log/syslog:

Aug 10 14:15:05 infra-01 blackbox_exporter[41318]: level=info ts=2021-08-10T18:15:05.701Z caller=main.go:385 msg="Listening on address" address=:9115

What did you do that produced an error?

I configured my service config file (/lib/systemd/system/blackbox.service) with:

Description=Blackbox Exporter Service
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=blackbox
Group=blackbox
ExecStart=/usr/local/bin/blackbox_exporter --config.file=/etc/blackbox/blackbox.yml --web.listen-address="127.0.0.1:9115"

Restart=always

[Install]

Point a browser to our public web address of the server running blackbox_exporter

What did you expect to see?

I was expecting to see nothing, as in: a connection could not be established

What did you see instead?

I could pull up the /metrics page and the /config page

[anemyte]

Can't reproduce that by running the binary directly. Have you made systemctl daemon-reload after making changes to /lib/systemd/system/blackbox.service?

My test:

❯ docker run --name bb --rm -p 9115:9115 prom/blackbox-exporter --web.listen-address="127.0.0.1:9115" --config.file=/etc/blackbox_exporter/config.yml
level=info ts=2021-08-24T09:07:51.364Z caller=main.go:224 msg="Starting blackbox_exporter" version="(version=0.19.0, branch=HEAD, revision=5d575b88eb12c65720862e8ad2c5890ba33d1ed0)"
level=info ts=2021-08-24T09:07:51.364Z caller=main.go:225 build_context="(go=go1.16.4, user=root@2b0258d5a55a, date=20210510-12:56:44)"
level=info ts=2021-08-24T09:07:51.365Z caller=main.go:237 msg="Loaded config file"
level=info ts=2021-08-24T09:07:51.365Z caller=main.go:385 msg="Listening on address" address=127.0.0.1:9115
level=info ts=2021-08-24T09:07:51.365Z caller=tls_config.go:191 msg="TLS is disabled." http2=false

❯ curl localhost:9115                                                                                                                                
curl: (56) Recv failure: Connection reset by peer

❯ docker exec -it bb nc localhost:9115
GET /metrics HTTP/1.1
Host: localhost
Connection: close

HTTP/1.1 200 OK
Content-Type: text/plain; version=0.0.4; charset=utf-8
Date: Tue, 24 Aug 2021 09:09:38 GMT
Connection: close
Transfer-Encoding: chunked

800
# HELP blackbox_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which blackbox_exporter was built.
# TYPE blackbox_exporter_build_info gauge
blackbox_exporter_build_info{branch="HEAD",goversion="go1.16.4",revision="5d575b88eb12c65720862e8ad2c5890ba33d1ed0",version="0.19.0"} 1
# HELP blackbox_exporter_config_last_reload_success_timestamp_seconds Timestamp of the last successful configuration reload.
# TYPE blackbox_exporter_config_last_reload_success_timestamp_seconds gauge
blackbox_exporter_config_last_reload_success_timestamp_seconds 1.6297960713653445e+09
...

[Mindtoeye]

Maybe this will be useful for someone. A lot of the directions out there use this configuration for the systemd service:

--web.listen-address="127.0.0.1:9115"

When you use that the service starts without error but the setting won't be used. Changed the line to this to make it pick it up:

--web.listen-address=127.0.0.1:9115