New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Not resolving using search domain #8

Closed
NoumanSaleem opened this Issue Feb 20, 2015 · 94 comments

Comments

Projects
None yet
@NoumanSaleem

NoumanSaleem commented Feb 20, 2015

Hostname is not properly resolving within the container when setting --dns-search on Docker.

Docker settings:

DOCKER_OPTS="--bip=172.17.42.1/16 --dns=172.17.42.1 --dns=10.0.2.15 --dns-search=service.consul"

gliderlabs/alpine

vagrant@local:~$ docker run --rm -it gliderlabs/alpine sh
/ # ping consul
ping: bad address 'consul'
/ # ping consul.service.consul
PING consul.service.consul (10.211.55.22): 56 data bytes
64 bytes from 10.211.55.22: seq=0 ttl=64 time=0.045 ms

progrium/busybox

vagrant@local:~$ docker run --rm -it progrium/busybox
/ # ping consul
PING consul (10.211.55.22): 56 data bytes

gliderlabs/registrator#111

@andyshinn

This comment has been minimized.

Collaborator

andyshinn commented Feb 20, 2015

Looks like musl doesn't support domain and search in resolv.conf: http://wiki.musl-libc.org/wiki/Functional_differences_from_glibc#Name_Resolver_.2F_DNS. We're researching workarounds to this issue. Thanks for reporting!

@brycekahle

This comment has been minimized.

brycekahle commented Mar 4, 2015

@andyshinn the musl behavior of querying all nameservers in /etc/resolv.conf in parallel and using the the first response is unexpected. Perhaps that could be added to the documentation?

@andyshinn

This comment has been minimized.

Collaborator

andyshinn commented Mar 5, 2015

Yep, agree it should be noted somewhere. I'll add some more information to the docs about the caveats soon.

@pikeas

This comment has been minimized.

pikeas commented Mar 22, 2015

@andyshinn The docs are a good start, but they raise further questions. It would be great if the section on dnsmasq (http://gliderlabs.viewdocs.io/docker-alpine/caveats) could be fleshed out to provide a full solution.

@andyshinn

This comment has been minimized.

Collaborator

andyshinn commented Mar 24, 2015

I tried to provide a general example of the use case where you want to send internal requests to a Consul server and the rest out to upstream name servers. What specifically do you think the example is missing?

@pikeas

This comment has been minimized.

pikeas commented Mar 24, 2015

A brief description of dnsmasq and how to use it - on the host, as an additional container, etc. In other words, specific instructions on use, rather than "maybe try this workaround".

@progrium

This comment has been minimized.

Contributor

progrium commented Mar 24, 2015

I think it could use a full dnsmasq config for the custom+upstream DNS
scenario so people can figure out how to run it on the host, then a pointer
to a good generic dnsmasq Docker container that would be easy to apply that
configuration for people that want to run it in a container, and lastly,
once available, a pointer to our magic resolve container.

On Tue, Mar 24, 2015 at 3:36 PM, Aris Pikeas notifications@github.com
wrote:

A brief description of dnsmasq and how to use it - on the host, as an
additional container, etc. In other words, specific instructions on use,
rather than "maybe try this workaround".


Reply to this email directly or view it on GitHub
#8 (comment)
.

Jeff Lindsay
http://progrium.com

@thockin

This comment has been minimized.

thockin commented May 7, 2015

I was so hopeful for docker-alpine - I need a mini package for DNS utils. Alas, DNS in alpine seems to be totally busted. A shame.

@tifayuki

This comment has been minimized.

tifayuki commented May 29, 2015

I set up a dnsmasq container. In resolv.dnsmasq.conf, I added

nameserver 8.8.8.8
search example.com

In alpine container I set the dns pointing to that dnsmasq container.

The problem is when I do ping base in alpine container, the log of dnsmasq shows that only base is sent to 8.8.8.8, but base.example.com is never queried.

@andyshinn

This comment has been minimized.

Collaborator

andyshinn commented May 29, 2015

The 3 dnsmasq options that might append a domain to unqualified names is --expand-hosts, --domain, and --auth-zone. Can you try setting these to see if it changes the behavior? Unfortunately, the only other way to do this is with some host file manipulation on the dnsmasq container.

@neilellis

This comment has been minimized.

neilellis commented Jul 15, 2015

This cannot be solved using dnsmasq, dnsmasq assumes glibc has expanded the hostname from the search entry in /etc/resolv.conf.

If you are in an environment (like Tutum) where links don't work because of this:

https://gist.github.com/neilellis/8983d6977f45f126df28

Is my workaround in conjunction with running dnsmasq:

dnsmasq --resolv-file=/etc/dnsmasq-resolv.conf --addn-hosts=/etc/hosts.links --no-daemon

The full image is at https://github.com/vizzbuzz/base-alpine

@progrium

This comment has been minimized.

Contributor

progrium commented Jul 15, 2015

Alternatively we have a container called Resolvable that helps:
https://github.com/gliderlabs/resolvable

@hypergig

This comment has been minimized.

hypergig commented Jul 28, 2015

Just because no one has mentioned it yet.. DNS search/domain resolution is really important in a kubernetes cluster. Kubernetes services are used for just about every integration point and they all resolve through DNS. Kubernetes injects the right information into /etc/resolv.conf to do this.

The fact that I can't connect to myservice.mynamespace from within an alpine container prevents me from doing anything useful. For example, I have a service called etcd.system that resolves to etcd host for http puts and gets. A very common thing is to write to etcd via curl for application configuration, etc. Alpine is the smallest base image that easily lets you download and use curl, but because my kubernetes service, etcd.system, isn't resolvable I am dead in the water.

I now have to use a 100 meg image to curl, when the 6 meg one would have done fine.

Just food for thought.

@progrium

This comment has been minimized.

Contributor

progrium commented Jul 28, 2015

Totally agree. Hopefully this puts pressure on musl developers to address this problem. If anybody knows the best way to do this, please share.

In the meantime, we're probably adding search domain support to Resolvable.

@JeanMertz

This comment has been minimized.

JeanMertz commented Aug 30, 2015

I guess this is still dead in the water. Ran into the exact same use-case as @hypergig, using Kubernetes, hitting DNS problems, finding out Alpine doesn't support DNS search.

The people at Kubernetes (specifically @thockin) seem to be aware of it at kubernetes/kubernetes#10163, but no dice so far.

@macropin

This comment has been minimized.

macropin commented Sep 3, 2015

I'd like to see this resolved too. We're using skydns for service discovery.

Has anyone raised this upstream with the musl devs?

@thockin

This comment has been minimized.

thockin commented Sep 3, 2015

I seem to recall not finding a way to file a bug with them.
On Sep 2, 2015 6:11 PM, "Andrew Cutler" notifications@github.com wrote:

I'd like to see this resolved too. We're using skydns for service
discovery.

Has anyone raised this upstream with the musl devs?


Reply to this email directly or view it on GitHub
#8 (comment)
.

@progrium

This comment has been minimized.

Contributor

progrium commented Sep 3, 2015

I've not found a way yet but if anybody finds a way or files an issue be sure to share the link.

@andyshinn

This comment has been minimized.

Collaborator

andyshinn commented Sep 4, 2015

I've sent an email to the musl mailing list. It can be viewed at http://www.openwall.com/lists/musl/2015/09/04/3. I'm still gathering some information to answer all of Rich's questions (based on DNS use cases in Consul, Kubernetes and SkyDNS). If you have any specific use cases, please let me know or weigh in directly on the mailing list thread.

@thockin

This comment has been minimized.

thockin commented Sep 4, 2015

Awesome. Let me know if there is anything I can do to help.

On Fri, Sep 4, 2015 at 12:22 PM, Andy Shinn notifications@github.com
wrote:

I've sent an email to the musl mailing list. It can be viewed at
http://www.openwall.com/lists/musl/2015/09/04/3. I'm still gathering some
information to answer all of Rich's questions (based on DNS use cases in
Consul, Kubernetes and SkyDNS). If you have any specific use cases, please
let me know or weigh in directly on the mailing list thread.


Reply to this email directly or view it on GitHub
#8 (comment)
.

@davidkbainbridge

This comment has been minimized.

davidkbainbridge commented Sep 28, 2015

Any update on this? Just ran into it this morning.

@janeczku

This comment has been minimized.

janeczku commented Oct 14, 2015

Sure, there is Alpine-kubernetes: A base image that comes with a special sauce for giving SEARCH domain powers to Docker Alpine. See here: https://github.com/janeczku/docker-alpine-kubernetes.

@mhart

This comment has been minimized.

mhart commented Oct 22, 2015

It looks like Rich's questions here are still unanswered: http://www.openwall.com/lists/musl/2015/09/04/5

Can anyone familiar with what Kubernetes et al need in this regard respond there? Or should we get Rich involved here? (I can't find him on GitHub though, so that might be out of the question)

@andyshinn

This comment has been minimized.

Collaborator

andyshinn commented Oct 23, 2015

I think @thockin just re-ignited the conversation. Sorry I've been so lax on the subject. I want to be respectful of everyone's goals and didn't really know how to reasonably answer the questions yet based on the communities goals.

@tuannvm

This comment has been minimized.

tuannvm commented Apr 24, 2017

same issue with alpine 3.5 and kubernetes 1.5.x :(
Sometimes it works, and sometimes does not.

bash-4.3# nslookup zookeeper.kafka
nslookup: can't resolve '(null)': Name does not resolve

nslookup: can't resolve 'zookeeper.kafka': Name does not resolve
bash-4.3# nslookup zookeeper.kafka
nslookup: can't resolve '(null)': Name does not resolve

Name:      zookeeper.kafka
Address 1: 10.3.0.142
bash-4.3#
@danielo515

This comment has been minimized.

danielo515 commented May 5, 2017

I'm also facing this issue on rancher, specifically connecting to mongodb databases. My image is unable to resolve the name of a link coming from another stack. However, if the links are within the same stack it works just fine. I don't understand why exactly.

I am now using a 300mb image when my average image size with alpine was 90mb. I would love to come back to alpine when this gets fixed.

@dchambers

This comment has been minimized.

dchambers commented May 5, 2017

@danielo515 / @tuannvm: this is fixed from an Alpine perspective, but it's not as tolerant of non-conforming DNS servers, as was the case with Rancher's DNS until they recently fixed it for example.

@danielo515

This comment has been minimized.

danielo515 commented May 5, 2017

@dchambers do you know the minimum rancher version that works? I want to check mine. In wich Alpine version is this considered fixed ?

@dchambers

This comment has been minimized.

dchambers commented May 5, 2017

I'm not 100% sure because I didn't perform the Rancher upgrade myself, but I know our cluster is definitely now working. More information in this issue.

@dchambers

This comment has been minimized.

dchambers commented May 5, 2017

In wich Alpine version is this considered fixed ?

@danielo515, just saw the other part of your question and the answer is Alpine 3.4.

@tuannvm

This comment has been minimized.

tuannvm commented May 9, 2017

Still have an issue with alpine 3.4 / kube-dns:1.9
Hope that kube-dns:2.x.x would fix this.

rk295 added a commit to rk295/docker-git2consul that referenced this issue May 19, 2017

Bumping to Alpine 3.5.
Mostly to avoid
[this](gliderlabs/docker-alpine#8) issue.
@chenww

This comment has been minimized.

chenww commented May 26, 2017

alpine does not support multiple nameservers as well. Only one

@maxbbender

This comment has been minimized.

maxbbender commented Jan 24, 2018

Any updates on this? Using Jenkins based alpine images with dynamic slaves through kubernetes 1.8.3. Some alpine images can resolve external addresses and others can't. Both have the same /etc/resolv.conf files.

@Jean85

This comment has been minimized.

Jean85 commented Feb 6, 2018

I've stumbled onto this same issue, and I've solved it removing the a search directive, which was triggering a CloudFlare DNS (buggy) resolution.

I've found out about the solution with this blog post: https://blog.maio.me/alpine-kubernetes-dns/

@richfelker

This comment has been minimized.

richfelker commented Feb 7, 2018

Cloudflare has EWONTFIX'd this issue according to:

kubernetes/dns#119 (comment)

@Jean85

This comment has been minimized.

Jean85 commented Feb 7, 2018

Exactly, that's why the only solution was removing the triggering domain from the search directive on my end.

YonatanKiron pushed a commit to YonatanKiron/kafka-docker that referenced this issue Mar 25, 2018

Yonatan Kiron
Update kafka 10 to newer alpine
Motivation - current version doesn't use /etc/resolv.conf which prevents
kubernetes usage with the current image.

See gliderlabs/docker-alpine#8
@inodb

This comment has been minimized.

inodb commented May 22, 2018

This is closed, but I'm still having problems in the latest release running on kubernetes (kops on AWS):

kubectl run -i --tty alpine --image=alpine --restart=Never -- nslookup kubernetes.default
nslookup: can't resolve '(null)': Name does not resolve

Name:      kubernetes.default
Address 1: 100.64.0.1 kubernetes.default.svc.cluster.local

in busybox nslookup works fine:

kubectl run -i --tty busybox --image=busybox --restart=Never -- nslookup kubernetes.default
Server:    100.64.0.10
Address 1: 100.64.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.default
Address 1: 100.64.0.1 kubernetes.default.svc.cluster.local
@Cloven

This comment has been minimized.

Cloven commented May 22, 2018

@inodb

This comment has been minimized.

inodb commented May 22, 2018

@Cloven nice, thanks for clarifying! Seems like it could be nice to have a p big warning sign for using alpine images on k8s somewhere

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment