DNS discovery should not trim final dot

[bboreham]

I wish to question this line. It is commented:

// Remove the final dot from rooted DNS names to make them look more usual.

but why do we care how it looks? A rooted name has a different meaning from a non-rooted name.

More practically, in a typical Kubernetes installation it causes 4-5 pointless DNS lookups as the search path is traversed.

[simonpasquier]

The code has been there since the inception of the DNS SD [1] but I agree that it should be removed. Maybe @juliusv has some opinion?

[1] https://github.com/prometheus/prometheus/pull/297/files#diff-de6c481fa836cf8897a7b6d1ae60e5d5R74

[juliusv]

I don't remember what motivated that, but it certainly seems bogus and IMO should just be fixed.

[grobie]

This is not as easy as it looks. The returned target name is used in many contexts, not only in the context of DNS lookups. In the context of HTTP as Host header or in the context of TLS as server name, the final dot is not only uncommon but usually unsupported.

I expect a solution to the problem to require a more complex solution, e.g. by using different names for DNS lookups and HTTP requests, or by making the DNS lookup at an earlier stage. Open for ideas and discussion @bboreham.

Possible workarounds:

• configure /etc/resolv.conf ndots option in a way to avoid additional search domain lookups
• configure a relabel rule to add the final dot to all returned SRV targets
• use templated DNS responses to return a NXDOMAIN response with a high negative TTL for such pointless DNS lookups (easy with CoreDNS)

More details can be found in this PR: #4766

[bboreham]

Can we start by agreeing on the nature of the problem, which I think is something like:

“Prometheus alters the SD results it gets from a DNS SRV lookup by removing any final dot, transforming fully-qualified names into relative names. Removing this artefact proved to inconvenience some users, so we choose to retain it”

If we agree on this, I think my suggestion would be to add another Prometheus SD type which uses the unaltered DNS response.

[juliusv]

@bboreham Fully agreed on the problem description. We probably wouldn't need a completely new SD type, but just add an option to the DNS SD?