prometheus links with libc.musl in the docker container

[miekg]

Alpine linux is great, but it's libc musl is pretty broken wrt to DNS lookups. When ldd-ing prometheus from the standard container I see:

/prometheus # cd /bin
/bin # ldd prometheus 
    /lib/ld-musl-x86_64.so.1 (0x7fe05b27e000)
    libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7fe05b27e000)

Which in turn looks like it does use the libc (and the resolver) making dns lookup flaky. I also though prometheus was compiled with -netgo to bypass this issue?

[fabxc]

Prometheus is not compiled with -netgo – personally, I never tried it. Do you know more about the implications of using it? Any effects on performance, memory usage?

If there are only upsides to it, we can certainly use it.

[miekg]

The upside is that every lookup goes through the same (Go) code path. I haven't measured the performance and mem usage, but I think it should be OK, i.e. no worse then with my go dns.

You can't get more broken dns lookups than with musl.

[beorn7]

We need to be careful WRT performance. A single Prometheus server can monitor thousands of targets and therefore create a lot of (regular, not necessarily SRV) DNS lookups. (We tickled a bug in the Go stdlib because of the lookup volume. The bug was in the way the Go stdlib interacts with the libc call, and we could have worked around it with -netgo, IIRC, but we didn't, presumably for performance reasons...)

Needs to be discussed with the folks once they are back from vacation.

[miekg]

[ Quoting notifications@github.com in "Re: [prometheus] prometheus links w..." ]

We need to be careful WRT performance. A single Prometheus server can monitor
thousands of targets and therefore create a lot of (regular, not necessarily
SRV) DNS lookups. (We tickled a bug in the Go stdlib because of the lookup
volume. The bug was in the way the Go stdlib interacts with the libc call, and
we could have worked around it with -netgo, IIRC, but we didn't, presumably for
performance reasons...)

Yes, this is why you then defaulted to my go dns lib. Which would not be
impacted by this change at all.

Whenever you do a lookup via net.LookupXXX or open a a connection to
alertmanager or whatever, i.e. when using net.Dial you hit musl libc
for some queries, using -netgo will force these queries to always use the
Go code path.

Needs to be discussed with the folks once they are back from vacation.

/Miek

Miek Gieben

[juliusv]

@sdurrheimer Hmm, didn't you work on using glibc again in Docker images? Was that only in your private Dockerfiles so far?

[juliusv]

(and yes, musl is completely broken with respect to DNS)

[sdurrheimer]

The current docker image is using glibc but in the master branch (corresponding to the 0.15.0rc). The stable branch is still using the old image.

[juliusv]

Ah ok, so this issue here should resolve itself when we release 0.15.0. @miekg Sounds good?

[miekg]

Falling back to glibc is fine.

[juliusv]

Cool, closing this here then, since it is fixed in the Dockerfile at HEAD.

[sdurrheimer]

@miekg Having a feedback if the addition of glibc in the docker image has solved your problem would be awesome.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.