New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pods can't resolve DNS on nodes with IPv6-only nameservers #58180

Closed
drf opened this Issue Jan 12, 2018 · 19 comments

Comments

Projects
None yet
7 participants
@drf

drf commented Jan 12, 2018

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened:

Installing a new cluster with kubeadm on a set of baremetal machines with the following content in /etc/resolv.conf

nameserver 2001:4860:4860::8888

Results in a working cluster, but pods aren't capable of resolving DNS such as google.it (internal resolution works fine). Apparently, pods can't understand IPv6-only nameservers: changing /etc/resolv.conf to

nameserver 2001:4860:4860::8888
nameserver 8.8.8.8

fixes the issue.

What you expected to happen:

Pods should be capable of resolving DNS through an IPv6 nameserver.

How to reproduce it (as minimally and precisely as possible):

Schedule a pod on a node with IPv6 only nameservers.

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version): 1.9.1
  • Cloud provider or hardware configuration: Baremetal VMs
  • OS (e.g. from /etc/os-release): Debian Stretch
  • Kernel (e.g. uname -a): Linux 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04) x86_64 GNU/Linux
  • Install tools: kubeadm
  • Others: N/A
@cmluciano

This comment has been minimized.

Show comment
Hide comment
Member

cmluciano commented Jan 16, 2018

@danehans

This comment has been minimized.

Show comment
Hide comment
@danehans
Contributor

danehans commented Jan 17, 2018

@leblancd

This comment has been minimized.

Show comment
Hide comment
@leblancd

leblancd Jan 17, 2018

Contributor

@drf - Using a single IPv6 nameserver address in the /etc/resolv.conf of each master/worker node should work (that's what I have in my IPv6 cluster). What's required for this to work, though, is that the kube-dns pod itself is getting an IPv6 address configured (e.g. set by your CNI network/IPAM plugin), and the kube-dns pod needs to be able to connect (using that pod IPv6 address as a source IP) to your target, external DNS controller (2001:4860:4860::8888).

If the kube-dns pod is getting assigned a private (or ULA) IPv6 address (begins with fd00:), then source NAT'ing of some form will be required between the kube-dns pod and the external DNS controller. If your CNI network plugin supports masquerading, then you can configure that. Otherwise, you'll need to set up NAT66 or other form of source NATing in your setup.

Can you exec into your kube-dns pod (kubectl exec -it -n kube-system sh') and check:

  • IPv6 address(es)
  • IPv6 routes / default gateway
  • Try 'ping6 2001:4860:4860::8888'

If the ping6's don't work, then it's an IPv6 connectivity/routing/snat issue.

Contributor

leblancd commented Jan 17, 2018

@drf - Using a single IPv6 nameserver address in the /etc/resolv.conf of each master/worker node should work (that's what I have in my IPv6 cluster). What's required for this to work, though, is that the kube-dns pod itself is getting an IPv6 address configured (e.g. set by your CNI network/IPAM plugin), and the kube-dns pod needs to be able to connect (using that pod IPv6 address as a source IP) to your target, external DNS controller (2001:4860:4860::8888).

If the kube-dns pod is getting assigned a private (or ULA) IPv6 address (begins with fd00:), then source NAT'ing of some form will be required between the kube-dns pod and the external DNS controller. If your CNI network plugin supports masquerading, then you can configure that. Otherwise, you'll need to set up NAT66 or other form of source NATing in your setup.

Can you exec into your kube-dns pod (kubectl exec -it -n kube-system sh') and check:

  • IPv6 address(es)
  • IPv6 routes / default gateway
  • Try 'ping6 2001:4860:4860::8888'

If the ping6's don't work, then it's an IPv6 connectivity/routing/snat issue.

@drf

This comment has been minimized.

Show comment
Hide comment
@drf

drf Jan 18, 2018

Hello @leblancd and thanks for the thorough explanation. I plan on reinstalling the cluster early next week, I'll start from scratch and report back to you. To avoid wasting each others' time - I have tried as CNI Calico Flannel and Canal, all of them installed through the kubeadm instructions. Are you aware of any limitations concerning IPv6 about them?

drf commented Jan 18, 2018

Hello @leblancd and thanks for the thorough explanation. I plan on reinstalling the cluster early next week, I'll start from scratch and report back to you. To avoid wasting each others' time - I have tried as CNI Calico Flannel and Canal, all of them installed through the kubeadm instructions. Are you aware of any limitations concerning IPv6 about them?

@leblancd

This comment has been minimized.

Show comment
Hide comment
@leblancd

leblancd Jan 18, 2018

Contributor

@drf - Sounds good, let me know if I can help when you restart your setup. From what I understand, the Calico CNI plugin supports IPv6, but Flannel does not yet support IPv6 (Flannel support for IPv6 might be in progress?). Since Canal uses Flannel for plumbing, I don't think that it supports IPv6 either, but maybe someone who's more familiar can correct me.

Contributor

leblancd commented Jan 18, 2018

@drf - Sounds good, let me know if I can help when you restart your setup. From what I understand, the Calico CNI plugin supports IPv6, but Flannel does not yet support IPv6 (Flannel support for IPv6 might be in progress?). Since Canal uses Flannel for plumbing, I don't think that it supports IPv6 either, but maybe someone who's more familiar can correct me.

@drf

This comment has been minimized.

Show comment
Hide comment
@drf

drf Jan 18, 2018

@leblancd cool, thanks for the insight - I'll stick to Calico for the sake of considering the issue belonging to Kubernetes or not.

drf commented Jan 18, 2018

@leblancd cool, thanks for the insight - I'll stick to Calico for the sake of considering the issue belonging to Kubernetes or not.

@danehans

This comment has been minimized.

Show comment
Hide comment
@danehans

danehans Jan 18, 2018

Contributor

cc'ing @caseydavenport @squeed for Calico/Flannel IPv6 support input.

Contributor

danehans commented Jan 18, 2018

cc'ing @caseydavenport @squeed for Calico/Flannel IPv6 support input.

@drf

This comment has been minimized.

Show comment
Hide comment
@drf

drf Jan 22, 2018

Installed new cluster with Calico on it, here's what happens:

/ # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN qlen 1
    link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP 
    link/ether 62:80:2c:76:7b:0e brd ff:ff:ff:ff:ff:ff
    inet 192.168.41.65/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::6080:2cff:fe76:7b0e/64 scope link 
       valid_lft forever preferred_lft forever
/ # ping6 2001:4860:4860::8888
PING 2001:4860:4860::8888 (2001:4860:4860::8888): 56 data bytes
ping6: sendto: Network unreachable
/ # ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=59 time=11.719 ms
/ # ip -6 route show
fe80::/64 dev eth0  metric 256 
unreachable default dev lo  metric -1  error -101
ff00::/8 dev eth0  metric 256 
unreachable default dev lo  metric -1  error -101
/ # ip route show
default via 169.254.1.1 dev eth0 
169.254.1.1 dev eth0

This is a cluster with a master+agent, on both hosts ping6 2001:4860:4860::8888 works as expected. resolv.conf on the host looks like:

nameserver 2001:4b78::53

Willing to help if you need more info

drf commented Jan 22, 2018

Installed new cluster with Calico on it, here's what happens:

/ # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN qlen 1
    link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP 
    link/ether 62:80:2c:76:7b:0e brd ff:ff:ff:ff:ff:ff
    inet 192.168.41.65/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::6080:2cff:fe76:7b0e/64 scope link 
       valid_lft forever preferred_lft forever
/ # ping6 2001:4860:4860::8888
PING 2001:4860:4860::8888 (2001:4860:4860::8888): 56 data bytes
ping6: sendto: Network unreachable
/ # ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=59 time=11.719 ms
/ # ip -6 route show
fe80::/64 dev eth0  metric 256 
unreachable default dev lo  metric -1  error -101
ff00::/8 dev eth0  metric 256 
unreachable default dev lo  metric -1  error -101
/ # ip route show
default via 169.254.1.1 dev eth0 
169.254.1.1 dev eth0

This is a cluster with a master+agent, on both hosts ping6 2001:4860:4860::8888 works as expected. resolv.conf on the host looks like:

nameserver 2001:4b78::53

Willing to help if you need more info

@drf

This comment has been minimized.

Show comment
Hide comment
@drf

drf Jan 22, 2018

Additional info: cluster installed with kubeadm and running atop Docker 17.03 on stretch.

drf commented Jan 22, 2018

Additional info: cluster installed with kubeadm and running atop Docker 17.03 on stretch.

@drf

This comment has been minimized.

Show comment
Hide comment
@drf

drf Jan 22, 2018

Also: note that the cluster was initialized with --pod-network-cidr=192.168.0.0/16 as per documentation

drf commented Jan 22, 2018

Also: note that the cluster was initialized with --pod-network-cidr=192.168.0.0/16 as per documentation

@leblancd

This comment has been minimized.

Show comment
Hide comment
@leblancd

leblancd Jan 22, 2018

Contributor

Hi @drf - Your command outputs indicate that Calico isn't configured for IPv6, and without the required IPv6 config, the kube-dns pod won't be able to connect to your external DNS server via IPv6 address. 2 issues that I see in your command outputs:

  • The pod's eth0 has only a link local (LLA) IPv6 address:
  inet6 fe80::6080:2cff:fe76:7b0e/64 scope link 

In addition to the LLA address, you'll want the pod configured with either a unique local (ULA, begins with fd00:) or global unicast (GUA, typically begins with 2001:) address to allow the pod to communicate off-node.

  • The pod has no IPv6 default route to route (e.g to route to your external DNS server).

If you're interested in testing IPv6 operation, I would suggest the following:
(1) Switch to IPv6-only configuration (only IPv6 addresses on master/worker nodes, and only IPv6 addresses on the pods, and only IPv6 addresses in your kubeadm config). Kubernetes doesn't yet support dual-stack operation (as of release 1.9.1). You might be able to get some dual-stack functionality working with Kubernetes 1.9.1, but there would be some unexpected quirks (only IPv4 service addresses, and Kubernetes would only be aware of / would have a preference for the IPv4 address for a pod).
(2) Configure Calico for IPv6-only (reference: https://docs.projectcalico.org/v2.5/reference/cni-plugin/configuration). The reference link shows dual-stack config (the Calico CNI plugin supports dual-stack), but Kubernetes itself doesn't yet support dual-stack operation.
(3) If your pods use ULA (private) addresses (e.g. addresses beginning with fd00:), then those addresses are not applicable outside of the scope of your cluster, so you may have to consider configure masquerading in the CNI plugin (I'm not too familiar with the Calico plugin, so I don't know if that's available), or some other form of source NAT'ing (e.g. NAT66) to allow the pod to communicate with external IPv6-addressed servers such as your DNS server.

Can you please try configuring your cluster as described above?

Contributor

leblancd commented Jan 22, 2018

Hi @drf - Your command outputs indicate that Calico isn't configured for IPv6, and without the required IPv6 config, the kube-dns pod won't be able to connect to your external DNS server via IPv6 address. 2 issues that I see in your command outputs:

  • The pod's eth0 has only a link local (LLA) IPv6 address:
  inet6 fe80::6080:2cff:fe76:7b0e/64 scope link 

In addition to the LLA address, you'll want the pod configured with either a unique local (ULA, begins with fd00:) or global unicast (GUA, typically begins with 2001:) address to allow the pod to communicate off-node.

  • The pod has no IPv6 default route to route (e.g to route to your external DNS server).

If you're interested in testing IPv6 operation, I would suggest the following:
(1) Switch to IPv6-only configuration (only IPv6 addresses on master/worker nodes, and only IPv6 addresses on the pods, and only IPv6 addresses in your kubeadm config). Kubernetes doesn't yet support dual-stack operation (as of release 1.9.1). You might be able to get some dual-stack functionality working with Kubernetes 1.9.1, but there would be some unexpected quirks (only IPv4 service addresses, and Kubernetes would only be aware of / would have a preference for the IPv4 address for a pod).
(2) Configure Calico for IPv6-only (reference: https://docs.projectcalico.org/v2.5/reference/cni-plugin/configuration). The reference link shows dual-stack config (the Calico CNI plugin supports dual-stack), but Kubernetes itself doesn't yet support dual-stack operation.
(3) If your pods use ULA (private) addresses (e.g. addresses beginning with fd00:), then those addresses are not applicable outside of the scope of your cluster, so you may have to consider configure masquerading in the CNI plugin (I'm not too familiar with the Calico plugin, so I don't know if that's available), or some other form of source NAT'ing (e.g. NAT66) to allow the pod to communicate with external IPv6-addressed servers such as your DNS server.

Can you please try configuring your cluster as described above?

@drf

This comment has been minimized.

Show comment
Hide comment
@drf

drf Jan 22, 2018

@leblancd I think I see what's happening now. The pod CIDR is configured through IPv4 only by kubeadm - and I've found some literature of people trying to achieve an IPv6 configuration with kubeadm before (http://blog.michali.net/2017/02/11/using-kubeadm-and-calico-plugin-for-ipv6-addresses/). I guess that when initializing with 192.168.0.0/16 as CIDR, kubeadm automatically configures Calico in IPv4 mode.

TBH, I believe this leads us to a number of possible outcomes.

  1. Kubeadm should give the option to deploy an IPv6 cluster (apparently, passing an IPv6 subnet to kubeadm init doesn't work)
  2. When/if Kubernetes will support dual-stack IPv4/IPv6, have kubeadm support that

So ultimately I believe this might be a kubeadm issue.

drf commented Jan 22, 2018

@leblancd I think I see what's happening now. The pod CIDR is configured through IPv4 only by kubeadm - and I've found some literature of people trying to achieve an IPv6 configuration with kubeadm before (http://blog.michali.net/2017/02/11/using-kubeadm-and-calico-plugin-for-ipv6-addresses/). I guess that when initializing with 192.168.0.0/16 as CIDR, kubeadm automatically configures Calico in IPv4 mode.

TBH, I believe this leads us to a number of possible outcomes.

  1. Kubeadm should give the option to deploy an IPv6 cluster (apparently, passing an IPv6 subnet to kubeadm init doesn't work)
  2. When/if Kubernetes will support dual-stack IPv4/IPv6, have kubeadm support that

So ultimately I believe this might be a kubeadm issue.

@rpothier

This comment has been minimized.

Show comment
Hide comment
@rpothier

rpothier Jan 22, 2018

Contributor

@drf Kubeadm determines if it is an IPv6 deployment based of the api-advertise-addresses.
so if api-advertise-addresses is v6, it's a v6 deployment.

Contributor

rpothier commented Jan 22, 2018

@drf Kubeadm determines if it is an IPv6 deployment based of the api-advertise-addresses.
so if api-advertise-addresses is v6, it's a v6 deployment.

@drf

This comment has been minimized.

Show comment
Hide comment
@drf

drf Jan 22, 2018

@rpothier so it is correct to leave the POD cidr with an IPv4 subnet and specify as the api-advertise-address the IPv6 address of the master? If so, I guess this is not a bug but it might be worth it to update the documentation

drf commented Jan 22, 2018

@rpothier so it is correct to leave the POD cidr with an IPv4 subnet and specify as the api-advertise-address the IPv6 address of the master? If so, I guess this is not a bug but it might be worth it to update the documentation

@leblancd

This comment has been minimized.

Show comment
Hide comment
@leblancd

leblancd Jan 22, 2018

Contributor

@drf - The assignment of pod IPs and the addition of IP routes on the pods is the responsibility of the CNI IPAM plugin. Setting kubeadm's --pod-network-cidr flag isn't intended to automatically set pod IP ranges; you still need to set the CNI network config files (including IPAM sections) on the worker nodes. Once you select the subnet ranges in the CNI network config files, then the --pod-network-cidr that you use for kubeadm init should span all of the subnet ranges that are defined in those CNI network config files. (If you have a mismatch, kube-proxy may error when it tries to set up some source NAT'ing for out-of-cluster addresses in ip6tables.) The kubeadm init flags that are needed to trigger IPv6 operation are --apiserver-advertise-address and --service-cidr. Re. eventual dual-stack support, yes, you're spot on that kubeadm will need changes to support multiple service-cidr's and multiple pod-network-cidr's.

Contributor

leblancd commented Jan 22, 2018

@drf - The assignment of pod IPs and the addition of IP routes on the pods is the responsibility of the CNI IPAM plugin. Setting kubeadm's --pod-network-cidr flag isn't intended to automatically set pod IP ranges; you still need to set the CNI network config files (including IPAM sections) on the worker nodes. Once you select the subnet ranges in the CNI network config files, then the --pod-network-cidr that you use for kubeadm init should span all of the subnet ranges that are defined in those CNI network config files. (If you have a mismatch, kube-proxy may error when it tries to set up some source NAT'ing for out-of-cluster addresses in ip6tables.) The kubeadm init flags that are needed to trigger IPv6 operation are --apiserver-advertise-address and --service-cidr. Re. eventual dual-stack support, yes, you're spot on that kubeadm will need changes to support multiple service-cidr's and multiple pod-network-cidr's.

@drf

This comment has been minimized.

Show comment
Hide comment
@drf

drf Jan 22, 2018

@leblancd pretty clear then, thanks :) FWIW, I think this issue can be closed as it is a misconfiguration issue as it stands, but probably it has some documentation bits which would be quite useful to other people trying to go IPv6

drf commented Jan 22, 2018

@leblancd pretty clear then, thanks :) FWIW, I think this issue can be closed as it is a misconfiguration issue as it stands, but probably it has some documentation bits which would be quite useful to other people trying to go IPv6

@fejta-bot

This comment has been minimized.

Show comment
Hide comment
@fejta-bot

fejta-bot Apr 22, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot commented Apr 22, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@fejta-bot

This comment has been minimized.

Show comment
Hide comment
@fejta-bot

fejta-bot May 22, 2018

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

fejta-bot commented May 22, 2018

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

@fejta-bot

This comment has been minimized.

Show comment
Hide comment
@fejta-bot

fejta-bot Jun 21, 2018

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

fejta-bot commented Jun 21, 2018

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment