Env variable to specify nameserver for local testing

[mwitkow]

Use case:
I want to develop Prometheus alerts, configs and rules locally. However, my resolv.conf points to our corp DNS, which is different from the prod DNS that Prometheus uses to resolv.conf. We do have a corp2prod bridge, so I can temporarily override /etc/resolv.conf with the bridge to prod DNS, but that is a) cumbersome b) affects other processes

Can we have an ENV variable PROMETHEUS_NAMESERVERS that will be used instead of /etc/resolv.conf for local dev?

@miekg for context.

[juliusv]

We should get away from hard-depending on the presence of /etc/resolv.conf in any case, as that's OS-specific and won't work e.g. under Windows. We were actually considering converting back to the standard library's DNS functions for DNS-SD, which are correctly using OS-specific default lookup mechanisms (/etc/resolv.conf on Unix, direct GetHostByName syscall on Windows). We mainly fell back to @miekg's custom DNS library back then due to bugs in the standard library's DNS implementation. Those might have been fixed now, but we'd need to verify that: #360

In any case, going back to the stdlib would conflict with your goal here (but would be nice for other reasons), since the stdlib's DNS resolver can't be configured to not use /etc/resolv.conf under Unix at all :-/ Further, defining an env variable like PROMETHEUS_NAMESERVERS would be problematic in the sense that Prometheus is doing other network requests (like sending samples to the remote storage or alerts to Alertmanager), where we're using the standard library's normal HTTP functions, which would be unaffected by this flag. Unfortunately you can't just tell the Go stdlib to use a different DNS server for all HTTP requests. You could only do things like manually resolving any names via @miekg's library before sending requests to the raw IP address, but then you'd also have to set Host headers yourself. So this variable would be confusing, as it would only affect a subset of the DNS lookups that Prometheus does.

So I think this is a valid use case, but I am not sure how it can be implemented nicely and consistently without causing a lot of extra code noise. @miekg - maybe any brilliant ideas? :)

[grobie]

I feel unsure about adding code to prometheus for such special use cases. The interface on linux is to lookup DNS servers via /etc/resolv.conf and in my experience the vast majority of tools doesn't give you an option to override that, so I'm not sure Prometheus should be an exception here.

@mwitkow-io What about running your prometheus in a docker container locally, which allows you to specify DNS servers via the --dns=... flag? https://docs.docker.com/articles/networking/#dns

[miekg]

[ Quoting notifications@github.com in "Re: [prometheus] Env variable to sp..." ]

We should get away from hard-depending on the presence of /etc/resolv.conf in
any case, as that's OS-specific and won't work e.g. under Windows. We were
actually considering converting back to the standard library's DNS functions
for DNS-SD, which are correctly using OS-specific default lookup mechanisms
(/etc/resolv.conf on Unix, direct GetHostByName syscall on Windows). We
mainly fell back to @miekg's custom DNS library back then due to bugs in the
standard library's DNS implementation. Those might have been fixed now, but
we'd need to verify that: #360

... and also note that names like "nodes..com" as wildcard addresses won't be
supported in that setup, as they are considered invalid names by glibc (and
maybe windows as well). For this use-case SkyDNS says "nodes.any.com" and
"nodes..com" are equivalent.

Looking at the CL linked from the prometheus issue, it seems they still don't
implement EDNS0, so you can't use a larger bufsize than 512 for udp queries.

In any case, going back to the stdlib would conflict with your goal here (but
would be nice for other reasons), since the stdlib's DNS resolver can't be
configured to not use /etc/resolv.conf under Unix at all :-/ Further,
defining an env variable like PROMETHEUS_NAMESERVERS would be problematic in
the sense that Prometheus is doing other network requests (like sending samples
to the remote storage or alerts to Alertmanager), where we're using the
standard library's normal HTTP functions, which would be unaffected by this
flag. Unfortunately you can't just tell the Go stdlib to use a different DNS
server for all HTTP requests. You could only do things like manually resolving
any names via @miekg's library before sending requests to the raw IP address,
but then you'd also have to set Host headers yourself. So this variable would
be confusing, as it would only affect a subset of the DNS lookups that
Prometheus does.

So I think this is a valid use case, but I am not sure how it can be
implemented nicely and consistently without causing a lot of extra code noise.
@miekg - maybe any brilliant ideas? :)

..Maybe... 2 things.

1. I can easily add a RESOLVCONF env variable to my dns lib that has the path of
   a resolv.conf like file and if that is set it will be used instead of
   /etc/resolv.conf.
2. IF prometheus is compiled to use glibc (and not the built-in Go
   resolver), you can LD_PRELOAD a lib the points to a different set of
   nameserver - this will effect all other prometheus lookups.

This will give the desired effect with no code changes to prometheus and a small
(couple of lines) change to my lib.

Thoughts?

/Miek

Miek Gieben

[juliusv]

@miekg That's great info, I wasn't aware of the wildcard issues that SkyDNS users would be having with the standard library.

I don't think we want to do 2), but out of curiosity, how would you do it? Use a custom dialer for the net.http clients which does lookups via a manually linked-in glibc?

1. sounds perfect from our side, since it would be no change to Prometheus at all, and nothing that we'd document officially. But would it be sufficient from your side if only DNS-SD requests use the custom nameserver?

I tend to agree with @grobie on the other points.

[miekg]

Re 2) isn't this the default for Go?

% go build
% ldd skydns 
    linux-vdso.so.1 =>  (0x00007ffd62fed000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f81f181b000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f81f1456000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f81f1a60000)

Any lookup SkyDNS would do via net.LookupXXX would be handled by glibc, AFAICT an LD_PRELOAD trick would still work here. So no code changes needed.

I also like the idea of using docker for local tests though.

[sdurrheimer]

docker is the best approach for your tests

[juliusv]

@miekg Yes, when you import net, glibc gets linked in, but as far as I can see from tracing a net.LookupAddr call through the code, it looks like it does not delegate to glibc at the point where it's about nameserver configuration:

net.LookupAddr (http://golang.org/src/net/lookup.go)
  -> lookupAddr (http://golang.org/src/net/lookup_unix.go)
    -> lookup (http://golang.org/src/net/dnsclient_unix.go) # This reads hardcoded /etc/resolv.conf
      -> tryOneName (http://golang.org/src/net/dnsclient_unix.go)
        -> this loops through all servers from /etc/resolv.conf, connects manually, does its own DNS requests, etc.

[miekg]

[ Quoting notifications@github.com in "Re: [prometheus] Env variable to sp..." ]

@miekg Yes, when you import net, glibc gets linked in, but as far as I can see from tracing a net.LookupAddr call through the code, it looks like it does not delegate to glibc at the point where it's about nameserver configuration:

net.LookupAddr (http://golang.org/src/net/lookup.go)
 -> lookupAddr (http://golang.org/src/net/lookup_unix.go)
   -> lookup (http://golang.org/src/net/dnsclient_unix.go) # This reads hardcoded /etc/resolv.conf
     -> tryOneName (http://golang.org/src/net/dnsclient_unix.go)
       -> this loops through all servers from /etc/resolv.conf, connects manually, does its own DNS requests, etc.

What a f*cking mess this is... Because if you look at net.LookupCNAME it has:

name, err, ok := cgoLookupCNAME(name)                                                                          
if !ok {                                                                                                        
     cname, err = goLookupCNAME(name)                                                                        
}                                                                                                               
return       

Which uses glibc immediately. My my, so what you hit depends on the actual RR
type you ask.... Unless you compile with +netgo which will always use the go
impl.

[juliusv]

Yep - so due to all that I wouldn't want to add an env variable for this into Prometheus itself and would advocate for going the Docker route instead. Or hey, if Docker is too heavy-weight, how about a simple chroot?

Closing this for now.

[mwitkow]

Yea, this seems pretty heavyweight after all. Docker seems like a good temporary solution.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.