update docker's resolv.conf file with options ndots:5 #10266

ArtfulCoder · 2015-06-23T23:37:57Z

Proof-of-concept PR for #10161 issue.
superseded #10241 PR

manually verified that it works
PR needs to be fixed up since ndots entry is done twice in the /etc/resolv.conf file(innocuous) amongst other things
the earlier PR was more complex where we would unmount docker's resolv.conf file and mount our own
in this PR, we simply modify the /etc/resolv.conf file that docker is mounting.

ArtfulCoder · 2015-06-23T23:42:21Z

k8s-bot · 2015-06-23T23:56:51Z

GCE e2e build/test passed for commit 9d00fb0aad750800e15c7b89335286b364787fc6.

ArtfulCoder · 2015-06-24T00:58:55Z

In this PR, we simply add the "options ndots:3" to the docker generated resolv.conf file.
Far simpler that the #10241 PR which requires mount/umount

k8s-bot · 2015-06-24T06:01:18Z

GCE e2e build/test passed for commit e7bfa71be37ad070ff19629c5e0da3dc7d76db33.

smarterclayton · 2015-06-24T13:04:05Z

The challenge is now that user code can race against the resolver - if someone has a very fast script that makes a curl call, we probably won't beat them. This makes it possible to use ndots, so it's an improvement, but I worry that it will be unreliable for users.

ArtfulCoder · 2015-06-24T13:16:55Z

In reality, all containers of the pod share the same dns resolv file, because docker sees that the other containers have marked the pause container as the container to use for networking namespace.

The pause container gets created first and we update the resolv file for the pause container.
Other containers of the pod get created later with the updated resolv file.

So we should not have this problem.

For the pause container too, (whose only purpose is to hold the networking namespace), the resolv.conf file gets updated in the same second that it got created, per kubelet log timestamps..

So pause serves has two important functions:

hold the networking namespace
share the same dns resolv file

My guess is that the code we had to create resolv.conf per container of the pod was probably never working since docker always uses the pause container's resolv.conf file for other containers of the same pod.

This is similar to how docker uses the host's resolv.conf file when dns options is not specified

smarterclayton · 2015-06-24T13:29:49Z

Is there no case today where we recreate the infra pod without tearing down the old containers?

ArtfulCoder · 2015-06-24T13:30:52Z

If the infra pod is teared down, the networking namespace goes with it and old containers of the same pod have to be recreated.

You can verify this by manually killing a pause container and see that containers of that pod restart.

k8s-bot · 2015-06-24T14:03:07Z

GCE e2e build/test passed for commit 4003cd350411d7ec02254cbf53eebe3c43231189.

k8s-bot · 2015-06-24T14:06:12Z

GCE e2e build/test passed for commit 8d0d71279e66e1e9b2378d520e0c1473187da311.

smarterclayton · 2015-06-24T15:46:38Z

pkg/kubelet/dockertools/manager.go

@@ -28,6 +28,7 @@ import (
 	"strconv"
 	"strings"
 	"sync"
+	//"syscall"


Remove this

ArtfulCoder · 2015-06-24T15:49:40Z

note to self: tests, address comments, add a lock to not update the same file twice.

smarterclayton · 2015-06-24T15:49:42Z

pkg/kubelet/dockertools/manager.go

+}
+
+func appendToFile(filePath, stringToAppend string) error {
+	f, err := os.OpenFile(filePath, os.O_APPEND|os.O_WRONLY, 0644)


Will this fail if the file doesn't exist (due to another issue, like docker getting stopped and tearing down the container) and if so will it be a clean error?

smarterclayton · 2015-06-24T15:53:13Z

What would prevent us from updating the file? Kubelet killed during restart? I would expect only one execution path here.

On Jun 24, 2015, at 11:49 AM, Abhi Shah notifications@github.com wrote:

note to self: tests, address comments, add a lock to not update the same file twice.

—
Reply to this email directly or view it on GitHub.

thockin · 2015-06-24T16:39:16Z

I haven't done review yet, but pathologically, we need to set ndots to 5 if
we want _dns._udp.kube-dns.default.svc to resolve.

On Wed, Jun 24, 2015 at 8:53 AM, Clayton Coleman notifications@github.com
wrote:

What would prevent us from updating the file? Kubelet killed during
restart? I would expect only one execution path here.

On Jun 24, 2015, at 11:49 AM, Abhi Shah notifications@github.com
wrote:

note to self: tests, address comments, add a lock to not update the same
file twice.

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub
#10266 (comment)
.

k8s-bot · 2015-06-24T16:59:57Z

GCE e2e build/test passed for commit 7e53e5ccbb7dbf6616a5b26749a07ff82eeb8c2b.

ArtfulCoder · 2015-06-25T01:11:41Z

@thockin @smarterclayton Thanks for the PR comments. I think I have addressed them all.
The e2e test is updated and should be green in the next run. (It is green on my test cluster)
I did a small refactor as well in the e2e test just because I was in there..

thockin · 2015-06-25T04:37:11Z

LGTM. We can reduce the test case back to one when we do a new dnsutils container.

thockin · 2015-06-25T04:37:46Z

@smarterclayton if you're not around, I'll get @dchen1107 to provide 2nd LGTM

ArtfulCoder · 2015-06-25T05:09:40Z

@thockin I already created a new dnsutils container. It is part of the PR. Its called jessie-dnsutils..

I didn't quite follow your new-container-comment..

ArtfulCoder · 2015-06-25T05:10:40Z

@k8s-bot test this please

thockin · 2015-06-25T05:15:27Z

I want to build a new dnsutils container that is not based on debian at
all. when I do that this test is not that useful any more.

On Wed, Jun 24, 2015 at 10:11 PM, Abhi Shah notifications@github.com
wrote:

@k8s-bot https://github.com/k8s-bot test this please

—
Reply to this email directly or view it on GitHub
#10266 (comment)
.

dchen1107 · 2015-06-25T05:23:54Z

ok to test

ArtfulCoder · 2015-06-25T05:26:34Z

@thockin ok, thanks

dchen1107 · 2015-06-25T05:30:26Z

LGTM

ArtfulCoder · 2015-06-25T13:55:33Z

@k8s-bot test this please
ok to test

update docker's resolv.conf file with options ndots:5

ArtfulCoder · 2015-06-25T18:54:07Z

http://kubekins.dls.corp.google.com/view/Critical%20Builds/job/kubernetes-e2e-gce/7202/

Jenkins GCE e2e passed.

ghaskins · 2016-07-18T20:57:12Z

This change appears to have broken kubernetes in my environment w.r.t. accessing external DNS for reasons I do not fully understand yet. Is there a way to make this optional?

thockin · 2016-08-03T07:19:44Z

@ghaskins only if you don't want kubernetes DNS to work. You can probably reduce it from 5 to 3, but that may change again in the future...

karteek · 2016-08-08T21:49:25Z

This change seems to make external DNS lookups very slow.
Any external DNS request with less than 5 dots, seem to make n requests (number of search domains) before resolving correctly.

root@mypod-controller-203fa# nslookup -debug
> google.com
Server:     10.3.0.10
Address:    10.3.0.10#53

------------
    QUESTIONS:
    google.com.my-namespace.svc.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  cluster.local
    origin = ns.dns.cluster.local
    mail addr = hostmaster.cluster.local
    serial = 1470690000
    refresh = 28800
    retry = 7200
    expire = 604800
    minimum = 60
    ttl = 60
    ADDITIONAL RECORDS:
------------
** server can't find google.com.my-namespace.svc.cluster.local: NXDOMAIN
Server:     10.3.0.10
Address:    10.3.0.10#53

------------
    QUESTIONS:
    google.com.svc.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  cluster.local
    origin = ns.dns.cluster.local
    mail addr = hostmaster.cluster.local
    serial = 1470690000
    refresh = 28800
    retry = 7200
    expire = 604800
    minimum = 60
    ttl = 60
    ADDITIONAL RECORDS:
------------
** server can't find google.com.svc.cluster.local: NXDOMAIN
Server:     10.3.0.10
Address:    10.3.0.10#53

------------
    QUESTIONS:
    google.com.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  cluster.local
    origin = ns.dns.cluster.local
    mail addr = hostmaster.cluster.local
    serial = 1470690000
    refresh = 28800
    retry = 7200
    expire = 604800
    minimum = 60
    ttl = 60
    ADDITIONAL RECORDS:
------------
** server can't find google.com.cluster.local: NXDOMAIN
Server:     10.3.0.10
Address:    10.3.0.10#53

------------
    QUESTIONS:
    google.com.us-west-2.compute.internal, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ADDITIONAL RECORDS:
------------
** server can't find google.com.us-west-2.compute.internal: NXDOMAIN
Server:     10.3.0.10
Address:    10.3.0.10#53

------------
    QUESTIONS:
    google.com, type = A, class = IN
    ANSWERS:
    ->  google.com
    internet address = 172.217.3.174
    ttl = 173
    AUTHORITY RECORDS:
    ADDITIONAL RECORDS:
------------
Non-authoritative answer:
Name:   google.com
Address: 172.217.3.174

Is there any way, I can use kubernetes DNS without slowing down external DNS resolution?

thockin · 2016-08-08T23:27:16Z

Can you characterize "slow" I see low milliseconds for DNS lookups, even
with ndots 5

On Mon, Aug 8, 2016 at 2:50 PM, Karteek E notifications@github.com wrote:

This change seems to make external DNS lookups very slow.
Any external DNS request with less than 5 dots, seem to make n requests
(number of search domains) before resolving correctly.

root@mypod-controller-203fa# nslookup -debug

google.com
Server: 10.3.0.10
Address: 10.3.0.10#53
QUESTIONS:
google.com.my-namespace.svc.cluster.local, type = A, class = IN
ANSWERS:
AUTHORITY RECORDS:
->  cluster.local
origin = ns.dns.cluster.local
mail addr = hostmaster.cluster.local
serial = 1470690000
refresh = 28800
retry = 7200
expire = 604800
minimum = 60
ttl = 60
ADDITIONAL RECORDS:
** server can't find google.com.my-namespace.svc.cluster.local: NXDOMAIN
Server: 10.3.0.10
Address: 10.3.0.10#53
QUESTIONS:
google.com.svc.cluster.local, type = A, class = IN
ANSWERS:
AUTHORITY RECORDS:
->  cluster.local
origin = ns.dns.cluster.local
mail addr = hostmaster.cluster.local
serial = 1470690000
refresh = 28800
retry = 7200
expire = 604800
minimum = 60
ttl = 60
ADDITIONAL RECORDS:
** server can't find google.com.svc.cluster.local: NXDOMAIN
Server: 10.3.0.10
Address: 10.3.0.10#53
QUESTIONS:
google.com.cluster.local, type = A, class = IN
ANSWERS:
AUTHORITY RECORDS:
->  cluster.local
origin = ns.dns.cluster.local
mail addr = hostmaster.cluster.local
serial = 1470690000
refresh = 28800
retry = 7200
expire = 604800
minimum = 60
ttl = 60
ADDITIONAL RECORDS:
** server can't find google.com.cluster.local: NXDOMAIN
Server: 10.3.0.10
Address: 10.3.0.10#53
QUESTIONS:
google.com.us-west-2.compute.internal, type = A, class = IN
ANSWERS:
AUTHORITY RECORDS:
ADDITIONAL RECORDS:
** server can't find google.com.us-west-2.compute.internal: NXDOMAIN
Server: 10.3.0.10
Address: 10.3.0.10#53
QUESTIONS:
google.com, type = A, class = IN
ANSWERS:
->  google.com
internet address = 172.217.3.174
ttl = 173
AUTHORITY RECORDS:
ADDITIONAL RECORDS:
Non-authoritative answer:
Name: google.com
Address: 172.217.3.174

Is there any way, I can use kubernetes DNS without slowing down external
DNS resolution?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#10266 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVB3m74f7reXa7CoSYHtUEtab061Oks5qd6SagaJpZM4FKUZI
.

karteek · 2016-08-08T23:49:01Z

I don't have exact numbers to quantify the slowness.
But, on high scale, each request to an external REST api gets 20-100 milliseconds slower which adds up.

This is primarily due to two reasons IMO

One is due to ndots 5,
KubeDNS doesn't seem to be caching DNS results (https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-dns/app/server.go#L140)

thockin · 2016-08-09T04:28:00Z

kube-dns has a dnsmasq in front of it doign caching, supposedly.

On Mon, Aug 8, 2016 at 4:49 PM, Karteek E notifications@github.com wrote:

I don't have exact numbers to quantify the slowness.
But, on high scale, each request to an external REST api gets 20-100
milliseconds slower which adds up.

This is primarily due to two reasons IMO

One is due to ndots 5,

KubeDNS doesn't seem to be caching DNS results (https://github.com/
kubernetes/kubernetes/blob/master/cmd/kube-dns/app/server.go#L140)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#10266 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVEfF1HutbztUa2DsYpo51uDXHeApks5qd8CdgaJpZM4FKUZI
.

girishkalele · 2016-08-09T04:37:35Z

@karteek

There is dnsmasq caching in 1.3 - could you run a describe on your kube-dns RC ?

There are two questions in my mind that I need to verify -

latency of a negative response from kube-dns should be extremely low
does dnsmasq cache negative responses from kube-dns to speed up the progression through the N searches

karteek · 2016-08-09T04:49:15Z

@girishkalele

I understand that there is dnsmasq caching. Probably thats why I didn't see much difference in dnsperf results even after enabling caching on kube-dns.

This is my kube-dns RC

➜  ~ kubectl --namespace=kube-system describe rc kube-dns-v19
Name:       kube-dns-v19
Namespace:  kube-system
Image(s):   gcr.io/google_containers/kubedns-amd64:1.6,gcr.io/google_containers/kube-dnsmasq-amd64:1.3,gcr.io/google_containers/exechealthz-amd64:1.1
Selector:   k8s-app=kube-dns,version=v19
Labels:     k8s-app=kube-dns
        kubernetes.io/cluster-service=true
        version=v19
Replicas:   12 current / 12 desired
Pods Status:    12 Running / 0 Waiting / 0 Succeeded / 0 Failed
No volumes.
No events.

Edit: Removed irrelevant information.

googlebot added the cla: yes label Jun 23, 2015

ArtfulCoder added the team/cluster label Jun 23, 2015

ArtfulCoder added this to the v1.0-candidate milestone Jun 23, 2015

ArtfulCoder mentioned this pull request Jun 23, 2015

mount resolv.conf #10241

Closed

ArtfulCoder force-pushed the kubelet_resolv branch 2 times, most recently from 4003cd3 to 8d0d712 Compare June 24, 2015 13:48

smarterclayton reviewed Jun 24, 2015
View reviewed changes

pkg/kubelet/dockertools/manager.go

@@ -28,6 +28,7 @@ import (

"strconv"

"strings"

"sync"

//"syscall"

Copy link

Contributor

smarterclayton Jun 24, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this

smarterclayton reviewed Jun 24, 2015
View reviewed changes

ArtfulCoder force-pushed the kubelet_resolv branch 3 times, most recently from d5951d9 to e001aa2 Compare June 24, 2015 16:59

ArtfulCoder force-pushed the kubelet_resolv branch from 26ecc0e to e68d5dd Compare June 25, 2015 01:10

thockin added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 25, 2015

dchen1107 added the ok-to-merge label Jun 25, 2015

update docker's resolv.conf file with options ndots:5

23caf44

ArtfulCoder force-pushed the kubelet_resolv branch from e68d5dd to 23caf44 Compare June 25, 2015 06:34

mbforbes added a commit that referenced this pull request Jun 25, 2015

Merge pull request #10266 from ArtfulCoder/kubelet_resolv

2bb4cf3

update docker's resolv.conf file with options ndots:5

mbforbes merged commit 2bb4cf3 into kubernetes:master Jun 25, 2015

thockin mentioned this pull request Jun 25, 2015

DNS searches across namespaces is BROKEN on some OSes #10161

Closed

thockin assigned girishkalele Aug 9, 2016

This was referenced Jul 7, 2018

Minishift DNS Issues When Host Has Search Domain Set minishift/minishift#2560

Closed

DNS resolution fails if default search domain has a wildcard match openshift/origin#17316

Open

update docker's resolv.conf file with options ndots:5 #10266

update docker's resolv.conf file with options ndots:5 #10266

Conversation

ArtfulCoder commented Jun 23, 2015

ArtfulCoder commented Jun 23, 2015

k8s-bot commented Jun 23, 2015

ArtfulCoder commented Jun 24, 2015

k8s-bot commented Jun 24, 2015

smarterclayton commented Jun 24, 2015

ArtfulCoder commented Jun 24, 2015

smarterclayton commented Jun 24, 2015

ArtfulCoder commented Jun 24, 2015

k8s-bot commented Jun 24, 2015

k8s-bot commented Jun 24, 2015

smarterclayton Jun 24, 2015

Choose a reason for hiding this comment

ArtfulCoder commented Jun 24, 2015

smarterclayton Jun 24, 2015

Choose a reason for hiding this comment

smarterclayton commented Jun 24, 2015

thockin commented Jun 24, 2015

k8s-bot commented Jun 24, 2015

ArtfulCoder commented Jun 25, 2015

thockin commented Jun 25, 2015

thockin commented Jun 25, 2015

ArtfulCoder commented Jun 25, 2015

ArtfulCoder commented Jun 25, 2015

thockin commented Jun 25, 2015

dchen1107 commented Jun 25, 2015

ArtfulCoder commented Jun 25, 2015

dchen1107 commented Jun 25, 2015

ArtfulCoder commented Jun 25, 2015

ArtfulCoder commented Jun 25, 2015

ghaskins commented Jul 18, 2016

thockin commented Aug 3, 2016

karteek commented Aug 8, 2016

thockin commented Aug 8, 2016

karteek commented Aug 8, 2016

thockin commented Aug 9, 2016

girishkalele commented Aug 9, 2016

karteek commented Aug 9, 2016 • edited

karteek commented Aug 9, 2016 •

edited