Cannot scrape targets specified by mDNS name

[moio]

What did you do?

I am trying to scrape a target that runs node_exporter by its mDNS name (published through Avahi) in my local network (see configuration below).

What did you expect to see?

The same behavior as specifying the IP address in targets (which works as expected).

What did you see instead? Under which circumstances?

State of the endpoint is DOWN in http://ptometheus.local:9090/targets, while ping works as expected.

Environment

• System information:

# uname -srm
Linux 4.4.21-69-default x86_64

• Prometheus version:

Latest released:

# /opt/prometheus/prometheus -version
prometheus, version 1.5.2 (branch: master, revision: bd1182d29f462c39544f94cc822830e1c64cf55b)
  build user:       root@a8af9200f95d
  build date:       20170210-14:41:22
  go version:       go1.7.5

Also reproduced on self-built master:

# /opt/prometheus/prometheus -version
prometheus, version 1.5.2 (branch: master, revision: 29f05680a29afe9aa6cf119563e99723a41765bd)
  build user:       silvio@lenovo
  build date:       20170328-11:31:33
  go version:       go1.7.3

• Prometheus configuration file:

global:
  scrape_interval: 5s
scrape_configs:
  - job_name: 'avahi'
    static_configs:
      - targets: ['avahi.local:9100']

[msiebuhr]

Slightly related, I recently implemented mDNS service discovery.

It looks for things advertising _prometheus-http._tcp (and ditto HTTPS) over zeroconf/mDNS and regularly dumps what it found. You won't need the zeroconf hostnames, but you will need to publish your service for consumption by prometheus.

[brian-brazil]

We support DNS, with would require specific mDNS support to be added.

[discordianfish]

Do we want to support that? I just realize my new home router isn't adding dhcp clients with their hostname to DNS, so would like to use mdns for my setup at home to discover exporters.

[brian-brazil]

We've a moratorium on new SDs, and mDNS isn't exactly something I'd expect to see in a production setup.

[discordianfish]

Arguably it's by far the most common SD mechanism. Not in your typical server infrastructure but using Prometheus in a home network/IoT is IMO a reasonable use case.

[brian-brazil]

There's always file_sd.

[msiebuhr]

I think you're confusing two faces of mDNS:

1. RFC6762 - .local hostnames and resolving them on a local network.
2. RFC6763 - mDNS service discovery (eg. _http._tcp.local for discovering all local web servers)

They happen to share the same means (and name), but to achieve two quite different goals.

As to the first, it is worth noting that Go hasn't always been nice to resolve .local names, ex: golang/go#16739. I haven't played with it on any recent Go-versions, so I can't attest to the current state of things. Wrapping things in file_sd won't make much difference in this case.

I sort-of-skirted the issue by using the service-discovery part (see https://github.com/msiebuhr/prometheus-mdns-sd) where I return the resolved addresses to Prometheus. (Look at #1903 for some history and further discussion).

[discordianfish]

@msiebuhr What were the main reasons to close the PR? I'd actually love to see this in Prometheus, with the influx of people using Prometheus for IoT use cases etc.

[msiebuhr]

It drags in quite a few large dependencies (low-level networking to get mDNS working), and I was told in private conversation (perhaps @brian-brazil ; cannot remember) that new service discovery mechanism preferably should be built outside core prometheus.

Edit: Just re-read @brian-brazil mention a moratorium on new SD's above. That may be where I got it from...

But yes, I generally find it weird (from a hobbyist/IoT point-of-view) that Prometheus comes with complex discovery mechanisms for some really big systems and everyone else must implement some custom cron/daemon thing poking at file_sd.

[discordianfish]

@msiebuhr I fully agree with this. I don't know why proprietary, vendor specific SD mechanisms for things like triton or azure are in scope while the most widely deployed open mechanism isn't.

I know that @SuperQ would also like to see mDNS support. So if we can find lazy consensus here, I'd probably call in a official vote on this.

[brian-brazil]

We continue to have a moratorium on new SD mechanisms, and once the example file_sd code is in the plan is to close all existing feature requests and point to that instead.

SDs take a lot of work to maintain, and history shows that there's little interest in doing so.

[SuperQ]

I don't agree with this plan.

[brian-brazil]

There's no sign of an end to the conditions that gave us the moratorium, and I'd rather clearly set expectations with the various requestors of SDs - some of whom have been waiting for 2+ years - than continue to give them false hope. Between the file_sd example and setting expectations I'm hoping that there'll be a surge in the development&maintenance of these by 3rd parties, rather than the current stream of +1s on issues which are ignored.

This structure gives us the option of pulling in some of the SDs into Prometheus in the future when things are healthier, but we continue to be unable to accept new SDs or even maintain what we already have.