Kube-dns add-on should accept option ndots for SkyDNS or document ConfigMap alternative subPath

[bogdando]

BUG REPORT:

Kubernetes version (use kubectl version):
Kubernetes v1.3.5+coreos.0,
Kube-dns add-on v19 from gcr.io/google_containers/kubedns-amd64:1.7

Environment:

• Cloud provider or hardware configuration: on-premise VMs
• OS (e.g. from /etc/os-release): Ubuntu Xenial
• Kernel (e.g. uname -a): 4.4.0-38-generic
• Install tools: Kube-dns add-on v19 from gcr.io/google_containers/kubedns-amd64:1.7
• Others:

What happened:
ndots:5 is hardcoded to the containers' base /etc/resolv.conf by kubelet running with --cluster_dns, --cluster_domain, --resolv-conf=/etc/resolv.conf flags.

What you expected to happen:
ndots should be configurable via the kubedns app definition or a configmap to allow users to chose if skydns shall be attempting absolute domains or utilizing the search domains, f.e.:

        args:
        # command = "/kube-dns"
        - --domain=cluster.local.
        - --ndots=5
        - --dns-port=10053

AFAICT, DNS SRV records expect ndots:7, thus will fail to resolv via skydns (or maybe not! #33554 (comment))
Also, this might affect DNS performance by generating undesired additional resolve queries for suggested search subdomains before actually trying the absolute domain, when a number of dots in the initial query exceeds the given ndots threshold.

How to reproduce it (as minimally and precisely as possible):
Deploy kube dns cluster add-in, check containers' /etc/resolv.conf within pods

Anything else do we need to know:
This is rather a docs issue, see #35525 (comment) for details and please address that in docs.

[MrHohn]

Why would resolving SRV records fail?

Suppose a hostname with X dot is queried and X < ndots threshhold, it will have the search paths appended before it is sent to the name server?

For the SRV record _my-port-name._my-port-protocol.my-svc.my-namespace.svc.cluster.local, I think below inputs will act properly:

• _my-port-name._my-port-protocol.my-svc.my-namespace
• _my-port-name._my-port-protocol.my-svc.my-namespace.svc
• _my-port-name._my-port-protocol.my-svc.my-namespace.svc.cluster.local

Given the search paths is:

• default.svc.cluster.local
• svc.cluster.local
• cluster.local

Please correct me if I miss something.

[macb]

Though SRV records may not fail, the ndots configuration should be exposed somewhere. The current 5 is a pretty rough default to expect everyone to adhere to. It results in something like 10+ dns queries per non-cluster domain attempted in our clusters.

#14051 (comment) describes some scenarios where lowering it "won't work". That doesn't actually seem to be the case, it just changes the behavior from first resolving all search domains and then trying the absolute domain to instead attempting the absolute domain then trying the search domains.

cc @thockin (linked your comment above)

Our clusters use cluster domains along the lines of: .int.clustername.region.internal.my.domain
This allows us to manage PKI off of our .internal.my.domain domain. This seems to get expanded to making search domains quite pricey when combined with the hosts /etc/resolv.conf search domains (6 configured search domains).

We have an existing application that lives outside our clusters at:
application.internal.my.domain

The 3 dots means we'll attempt the 6 cluster search domains before actually trying the absolute domain (which ultimately is resolved by the regional DNS servers outside the k8s cluster). That's 12 DNS queries (A and AAAA for each search domain) which ultimately fail since application.internal.my.domain doesn't exist within our cluster.

If instead we could configure ndots to say ndots:3 the absolute domain is first attempted. If it NXDOMAINs, all of the search domains will be applied just as before. 3 in this case would also keep the previous behavior of application, application.default, application.default.svc utilizing the search domains first.

In any case, it would give the option to the user to decide which is more important for their usage; attempting absolute domains or utilizing the search domains.

[thockin]

I don't think that behavior is consistent. The resolvers I have seen never try search if the query has >= ndots dots.

$ k run thdbg --rm --restart=Never -ti --image=ubuntu
Waiting for pod default/thdbg to be running, status is Pending, pod ready: false
Waiting for pod default/thdbg to be running, status is Pending, pod ready: false
Waiting for pod default/thdbg to be running, status is Pending, pod ready: false
Waiting for pod default/thdbg to be running, status is Pending, pod ready: false
If you don't see a command prompt, try pressing enter.

root@thdbg:/# apt-get update >/dev/null 2>&1

root@thdbg:/# apt-get install -y dnsutils >/dev/null 2>&1

root@thdbg:/# cat /etc/resolv.conf 
search default.svc.cluster.local svc.cluster.local cluster.local google.internal c.thockin-dev.internal
nameserver 10.0.0.10
options ndots:5

root@thdbg:/# cat > /etc/resolv.conf << EOF
> search default.svc.cluster.local svc.cluster.local cluster.local google.internal c.thockin-dev.internal
> nameserver 10.0.0.10
> options ndots:1
> EOF

root@thdbg:/# cat /etc/resolv.conf 
search default.svc.cluster.local svc.cluster.local cluster.local google.internal c.thockin-dev.internal
nameserver 10.0.0.10
options ndots:1

root@thdbg:/# nslookup kubernetes
Server:     10.0.0.10
Address:    10.0.0.10#53

Non-authoritative answer:
Name:   kubernetes.default.svc.cluster.local
Address: 10.0.0.1

root@thdbg:/# nslookup kubernetes.default
Server:     10.0.0.10
Address:    10.0.0.10#53

** server can't find kubernetes.default: NXDOMAIN

[macb]

From the resolv.conf man page:

The default for n is 1, meaning that if there are any dots in a name, the name will be tried first as an absolute name before any search list elements are appended to it.

Implies search domains should still be respected even if the absolute domain is attempted. It does seem not everything respects that though. Seems like something still best left up to the user to determine for themselves?

[thockin]

We had a rough proposal from someone to add a DNS policy for
"ClusterNoSearch" or something that was the same as before, but only set
ndots:1 for the requesting pod.

I would accept such a PR...

On Oct 6, 2016 6:56 AM, "Mac Browning" notifications@github.com wrote:

From the resolv.conf man page:

The default for n is 1, meaning that if there are any dots in a name, the
name will be tried first as an absolute name before any search list
elements are appended to it.

Implies search domains should still be respected even if the absolute
domain is attempted. It does seem not everything respects that though.
Seems like something still best left up to the user to determine for
themselves?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#33554 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVNbMlEzaPmIInzweCYhxABYDqkxGks5qxP33gaJpZM4KHmmB
.

[macb]

tl;dr:

It seems like libbind/netresolv and musl are consistent in ignoring search domains when the ndots option is met. However glibc (and other similar implementations such as golang's dns impl) would continue to respect search domains with a lower ndots.

I wouldn't want to just go from ndots:5 to ndots:1 since the vast majority of search signaling could still be preserved with ndots:3 in our case. Though given the choice ndots:1 is likely more useful for users that have a lot of services outside their clusters since it will be 1 failed absolute domain request then falling back to search for internal requests instead of 10+ failed searches before falling back to the absolute domain for external services.

Do you have the ClusterNoSearch proposal handy? A search didn't turn anything up in issues.

---

With ndots:1 and an otherwise normal k8s resolv.conf:

nslookup

All of the bind utilities (dig, nslookup, etc) use libbind (or whatever its called)

root@macb-debug:/# nslookup kubernetes.default
Server:         172.17.240.2
Address:        172.17.240.2#53

** server can't find kubernetes.default: NXDOMAIN

And tcpdump from that request where it just tries the absolute:

17:35:56.959854 IP 172.17.41.7.44089 > 172.17.240.2.53: 56391+ A? kubernetes.default. (36)
17:35:56.963720 IP 172.17.240.2.53 > 172.17.41.7.44089: 56391 NXDomain 0/0/0 (36)

This makes sense given bind documents ndots as:

+ndots=D
Set the number of dots that have to appear in name to D for it to be considered absolute. The default value is that defined using the ndots statement in /etc/resolv.conf, or 1 if no ndots statement is present. Names with fewer dots are interpreted as relative names and will be searched for in the domains listed in the search or domain directive in /etc/resolv.conf.

curl

However, if I use curl, it will resolve:

root@macb-debug:/# curl http://kubernetes.default -v
* Rebuilt URL to: http://kubernetes.default/
* Hostname was NOT found in DNS cache
*   Trying 172.17.240.1...

And tcpdump from that request where we can see it trying the absolute then falling back to search domains:

17:34:14.753159 IP 172.17.41.7.42383 > 172.17.240.2.53: 5843+ A? kubernetes.default. (36)
17:34:14.753246 IP 172.17.41.7.42383 > 172.17.240.2.53: 5736+ AAAA? kubernetes.default. (36)
17:34:14.753510 IP 172.17.240.2.53 > 172.17.41.7.42383: 5843 NXDomain 0/0/0 (36)
17:34:14.753604 IP 172.17.240.2.53 > 172.17.41.7.42383: 5736 NXDomain 0/0/0 (36)
17:34:14.753727 IP 172.17.41.7.60146 > 172.17.240.2.53: 48770+ A? kubernetes.default.default.svc.int.frog.nyc3.internal.my.domain. (88)
17:34:14.753798 IP 172.17.41.7.60146 > 172.17.240.2.53: 46767+ AAAA? kubernetes.default.default.svc.int.frog.nyc3.internal.my.domain. (88)
17:34:14.754942 IP 172.17.240.2.53 > 172.17.41.7.60146: 46767 NXDomain 0/1/0 (259)
17:34:14.755472 IP 172.17.240.2.53 > 172.17.41.7.60146: 48770 NXDomain 0/1/0 (259)
17:34:14.755721 IP 172.17.41.7.34184 > 172.17.240.2.53: 36544+ A? kubernetes.default.svc.int.frog.nyc3.internal.my.domain. (80)
17:34:14.755860 IP 172.17.41.7.34184 > 172.17.240.2.53: 26345+ AAAA? kubernetes.default.svc.int.frog.nyc3.internal.my.domain. (80)
17:34:14.756634 IP 172.17.240.2.53 > 172.17.41.7.34184: 36544 1/0/0 A 172.17.240.1 (96)
17:34:14.756660 IP 172.17.240.2.53 > 172.17.41.7.34184: 26345 0/0/0 (80)

golang (1.7)

Tried out a simple go program(go1.7):

package main

import "net/http"

func main() {
        http.Get("http://kubernetes.default")
}

root@macb-debug:~# ./main

And the tcpdump:

17:42:55.799352 IP 172.17.41.7.54780 > 172.17.240.2.53: 49389+ AAAA? kubernetes.default. (36)
17:42:55.801342 IP 172.17.41.7.52301 > 172.17.240.2.53: 61981+ A? kubernetes.default. (36)
17:42:55.805505 IP 172.17.240.2.53 > 172.17.41.7.54780: 49389 NXDomain 0/0/0 (36)
17:42:55.805636 IP 172.17.240.2.53 > 172.17.41.7.52301: 61981 NXDomain 0/0/0 (36)
17:42:55.806171 IP 172.17.41.7.45104 > 172.17.240.2.53: 32960+ AAAA? kubernetes.default.default.svc.int.frog.nyc3.internal.my.domain. (88)
17:42:55.806411 IP 172.17.41.7.47877 > 172.17.240.2.53: 41316+ A? kubernetes.default.default.svc.int.frog.nyc3.internal.my.domain. (88)
17:42:55.807575 IP 172.17.240.2.53 > 172.17.41.7.47877: 41316 NXDomain 0/1/0 (259)
17:42:55.807816 IP 172.17.240.2.53 > 172.17.41.7.45104: 32960 NXDomain 0/1/0 (259)
17:42:55.808179 IP 172.17.41.7.54148 > 172.17.240.2.53: 11120+ AAAA? kubernetes.default.svc.int.frog.nyc3.internal.my.domain. (80)
17:42:55.808339 IP 172.17.240.2.53 > 172.17.41.7.54148: 11120 0/0/0 (80)
17:42:55.808365 IP 172.17.41.7.40203 > 172.17.240.2.53: 65043+ A? kubernetes.default.svc.int.frog.nyc3.internal.my.domain. (80)
17:42:55.808502 IP 172.17.240.2.53 > 172.17.41.7.40203: 65043 1/0/0 A 172.17.240.1 (96)

The golang stdlib spells out the case nicely in the unix dnsclient.

musl

musl documents its difference at least:

queries with fewer dots than the ndots configuration variable are processed with search first then tried literally (just like glibc), but those with at least as many dots as ndots are only tried in the global namespace (never falling back to search, which glibc would do if the name is not found in the global DNS namespace)

[thockin]

So we can not depend on this behavior. Unfortunately, resolv.conf behavior
is under-specified here and in other ways.

The "proposal" was an offhand remark in a bug or email or something. If
this is really a pain point, I would be happy for someone to open a new
proposal (small) on this.

On Thu, Oct 6, 2016 at 11:15 AM, Mac Browning notifications@github.com
wrote:

tl;dr:

It seems like libbind/netresolv and musl are consistent in ignoring
search domains when the ndots option is met. However glibc (and other
similar implementations such as golang's dns impl) would continue to
respect search domains with a lower ndots.

I wouldn't want to just want to go from ndots:5 to ndots:1 since the vast
majority of search signaling could still be preserved with ndots:3 in our
case. Though given the choice ndots:1 is likely more useful for users
that have a lot of services outside their clusters since it will be 1
failed absolute domain request then falling back to search for internal
requests instead of 10+ failed searches before falling back to the absolute
domain for external services.

Do you have the ClusterNoSearch proposal handy? That search didn't turn

anything up in issues.

With ndots:1 and an otherwise normal k8s resolv.conf: nslookup

All of the bind utilities (dig, nslookup, etc) use libbind

root@macb-debug:/# nslookup kubernetes.default
Server: 172.17.240.2
Address: 172.17.240.2#53

** server can't find kubernetes.default: NXDOMAIN

And tcpdump from that request where it just tries the absolute:

17:35:56.959854 IP 172.17.41.7.44089 > 172.17.240.2.53: 56391+ A? kubernetes.default. (36)
17:35:56.963720 IP 172.17.240.2.53 > 172.17.41.7.44089: 56391 NXDomain 0/0/0 (36)

This makes sense given bind documents ndots as:

+ndots=D
Set the number of dots that have to appear in name to D for it to be
considered absolute. The default value is that defined using the ndots
statement in /etc/resolv.conf, or 1 if no ndots statement is present. Names
with fewer dots are interpreted as relative names and will be searched for
in the domains listed in the search or domain directive in /etc/resolv.conf.

curl

However, if I use curl, it will resolve:

root@macb-debug:/# curl http://kubernetes.default -v

• Rebuilt URL to: http://kubernetes.default/
• Hostname was NOT found in DNS cache
• Trying 172.17.240.1...

And tcpdump from that request where we can see it trying the absolute
then falling back to search domains:

17:34:14.753159 IP 172.17.41.7.42383 > 172.17.240.2.53: 5843+ A? kubernetes.default. (36)
17:34:14.753246 IP 172.17.41.7.42383 > 172.17.240.2.53: 5736+ AAAA? kubernetes.default. (36)
17:34:14.753510 IP 172.17.240.2.53 > 172.17.41.7.42383: 5843 NXDomain 0/0/0 (36)
17:34:14.753604 IP 172.17.240.2.53 > 172.17.41.7.42383: 5736 NXDomain 0/0/0 (36)
17:34:14.753727 IP 172.17.41.7.60146 > 172.17.240.2.53: 48770+ A? kubernetes.default.default.svc.int.frog.nyc3.internal.my.domain. (88)
17:34:14.753798 IP 172.17.41.7.60146 > 172.17.240.2.53: 46767+ AAAA? kubernetes.default.default.svc.int.frog.nyc3.internal.my.domain. (88)
17:34:14.754942 IP 172.17.240.2.53 > 172.17.41.7.60146: 46767 NXDomain 0/1/0 (259)
17:34:14.755472 IP 172.17.240.2.53 > 172.17.41.7.60146: 48770 NXDomain 0/1/0 (259)
17:34:14.755721 IP 172.17.41.7.34184 > 172.17.240.2.53: 36544+ A? kubernetes.default.svc.int.frog.nyc3.internal.my.domain. (80)
17:34:14.755860 IP 172.17.41.7.34184 > 172.17.240.2.53: 26345+ AAAA? kubernetes.default.svc.int.frog.nyc3.internal.my.domain. (80)
17:34:14.756634 IP 172.17.240.2.53 > 172.17.41.7.34184: 36544 1/0/0 A 172.17.240.1 (96)
17:34:14.756660 IP 172.17.240.2.53 > 172.17.41.7.34184: 26345 0/0/0 (80)

golang (1.7)

Tried out a simple go program(go1.7):

package main

import "net/http"

func main() {
http.Get("http://kubernetes.default")
}

root@macb-debug:~# ./main

And the tcpdump:

17:42:55.799352 IP 172.17.41.7.54780 > 172.17.240.2.53: 49389+ AAAA? kubernetes.default. (36)
17:42:55.801342 IP 172.17.41.7.52301 > 172.17.240.2.53: 61981+ A? kubernetes.default. (36)
17:42:55.805505 IP 172.17.240.2.53 > 172.17.41.7.54780: 49389 NXDomain 0/0/0 (36)
17:42:55.805636 IP 172.17.240.2.53 > 172.17.41.7.52301: 61981 NXDomain 0/0/0 (36)
17:42:55.806171 IP 172.17.41.7.45104 > 172.17.240.2.53: 32960+ AAAA? kubernetes.default.default.svc.int.frog.nyc3.internal.my.domain. (88)
17:42:55.806411 IP 172.17.41.7.47877 > 172.17.240.2.53: 41316+ A? kubernetes.default.default.svc.int.frog.nyc3.internal.my.domain. (88)
17:42:55.807575 IP 172.17.240.2.53 > 172.17.41.7.47877: 41316 NXDomain 0/1/0 (259)
17:42:55.807816 IP 172.17.240.2.53 > 172.17.41.7.45104: 32960 NXDomain 0/1/0 (259)
17:42:55.808179 IP 172.17.41.7.54148 > 172.17.240.2.53: 11120+ AAAA? kubernetes.default.svc.int.frog.nyc3.internal.my.domain. (80)
17:42:55.808339 IP 172.17.240.2.53 > 172.17.41.7.54148: 11120 0/0/0 (80)
17:42:55.808365 IP 172.17.41.7.40203 > 172.17.240.2.53: 65043+ A? kubernetes.default.svc.int.frog.nyc3.internal.my.domain. (80)
17:42:55.808502 IP 172.17.240.2.53 > 172.17.41.7.40203: 65043 1/0/0 A 172.17.240.1 (96)

The golang stdlib spells out the case nicely in the unix dnsclient
https://golang.org/src/net/dnsclient_unix.go.
musl

musl documents its difference at least:

queries with fewer dots than the ndots configuration variable are
processed with search first then tried literally (just like glibc), but
those with at least as many dots as ndots are only tried in the global
namespace (never falling back to search, which glibc would do if the name
is not found in the global DNS namespace)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#33554 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVEdryFhbsGNSfsnEM8txtzw5-Xmcks5qxTqqgaJpZM4KHmmB
.

[bogdando]

This conversation definitely fixed white spaces I had with understanding ndots in action, thank you for details - I've updated the bug description. This has nothing to the bug relevance tho.

[bogdando]

@thockin hardcoded ndots:5 is still a pain point, despite on implementation details of DNS stack. This keeps this issue relevant.

[bogdando]

NOTE: this is rather a docs issue, see #35525 (comment)
I suggest to fix this in docs, as described by @fluxrad