cluster networking is broken?

[liyimeng]

helm install job never succeed, it seem that it is not possible to reach dns server.

alpine:/home/alpine/k3s/dist/artifacts# ./k3s kubectl  get all -n kube-system 
NAME                             READY   STATUS             RESTARTS   AGE
pod/coredns-7748f7f6df-tp7fq     1/1     Running            1          104m
pod/helm-install-traefik-g5rmk   0/1     CrashLoopBackOff   21         104m

NAME               TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
service/kube-dns   ClusterIP   10.43.0.10   <none>        53/UDP,53/TCP,9153/TCP   104m

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/coredns   1/1     1            1           104m

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/coredns-7748f7f6df   1         1         1       104m

NAME                             COMPLETIONS   DURATION   AGE
job.batch/helm-install-traefik   0/1           104m       104m

./k3s kubectl   -n kube-system logs -f pod/helm-install-traefik-g5rmk
+ export HELM_HOST=127.0.0.1:44134+ 
tiller --listen=127.0.0.1:44134 --storage=secret
+ HELM_HOST=127.0.0.1:44134
+ helm init --client-only
[main] 2019/02/08 20:48:52 Starting Tiller v2.12.3 (tls=false)
[main] 2019/02/08 20:48:52 GRPC listening on 127.0.0.1:44134
[main] 2019/02/08 20:48:52 Probes listening on :44135
[main] 2019/02/08 20:48:52 Storage driver is Secret
[main] 2019/02/08 20:48:52 Max history per release is 0
Creating /root/.helm 
Creating /root/.helm/repository 
Creating /root/.helm/repository/cache 
Creating /root/.helm/repository/local 
Creating /root/.helm/plugins 
Creating /root/.helm/starters 
Creating /root/.helm/cache/archive 
Creating /root/.helm/repository/repositories.yaml 
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com 
Error: Looks like "https://kubernetes-charts.storage.googleapis.com" is not a valid chart repository or cannot be reached: Get https://kubernetes-charts.storage.googleapis.com/index.yaml: dial tcp: lookup kubernetes-charts.storage.googleapis.com on 10.43.0.10:53: read udp 10.42.0.4:39333->10.43.0.10:53: i/o timeout

Verify by running a busy box

alpine:/home/alpine/k3s/dist/artifacts# ./k83s kubectl run -i --tty busybox --image=busybox --restart=Never -- sh
ash: ./k83s: not found
alpine:/home/alpine/k3s/dist/artifacts# ./k3s kubectl run -i --tty busybox --image=busybox --restart=Never -- sh
If you don't see a command prompt, try pressing enter.
/ # 
/ # ping 10.43.0.10
PING 10.43.0.10 (10.43.0.10): 56 data bytes
^C
--- 10.43.0.10 ping statistics ---
7 packets transmitted, 0 packets received, 100% packet loss
/ # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
3: eth0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue 
    link/ether 32:03:33:52:8c:19 brd ff:ff:ff:ff:ff:ff
    inet 10.42.0.6/24 brd 10.42.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::3003:33ff:fe52:8c19/64 scope link 
       valid_lft forever preferred_lft forever
/ # ping 10.43.0.10
PING 10.43.0.10 (10.43.0.10): 56 data bytes
^C
--- 10.43.0.10 ping statistics ---
6 packets transmitted, 0 packets received, 100% packet loss
/ # ping 10.42.0.6
PING 10.42.0.6 (10.42.0.6): 56 data bytes
64 bytes from 10.42.0.6: seq=0 ttl=64 time=0.109 ms
64 bytes from 10.42.0.6: seq=1 ttl=64 time=0.108 ms
64 bytes from 10.42.0.6: seq=2 ttl=64 time=0.106 ms
^C
--- 10.42.0.6 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.106/0.107/0.109 ms

[ibuildthecloud]

@liyimeng can you ensure the br_netfilter module is loaded. The agent is supposed to load this module but it seems to not always work. I'm troubleshooting that now.

[ibuildthecloud]

@liyimeng FYI, it you are running in a container you need bind mount in /lib/modules/$(uname -r):/lib/modules/$(uname -r):ro so that modules can be loaded

[liyimeng]

I do have it loaded

alpine:/home/alpine/k3s/dist/artifacts# lsmod | grep netfilter br_netfilter 20480 0 bridge 163840 1 br_netfilter

When I do a troubleshooting like this:
https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/#running-commands-in-a-pod

`
/ # wget -O- hostnames
Connecting to hostnames (10.43.129.144:80)
hostnames-85bc9c579-rtr4r

•                100% |***********************************************************************************************************************|    26  0:00:00 ETA
  

/ # nslookup hostnames
Server: 10.43.0.10
Address: 10.43.0.10:53

** server can't find hostnames.default.svc.cluster.local: NXDOMAIN

*** Can't find hostnames.svc.cluster.local: No answer
*** Can't find hostnames.cluster.local: No answer
*** Can't find hostnames.default.svc.cluster.local: No answer
*** Can't find hostnames.svc.cluster.local: No answer
*** Can't find hostnames.cluster.local: No answer

/ # nslookup hostnames.default
;; connection timed out; no servers could be reached
/ # cat /etc/hosts

Kubernetes-managed hosts file.

127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-mcastprefix
fe00::1 ip6-allnodes
fe00::2 ip6-allrouters
10.42.0.7 busybox
/ # arp
10-42-0-9.hostnames.default.svc.cluster.local (10.42.0.9) at 96:12:b9:85:cf:5c [ether] on eth0
10-42-0-8.hostnames.default.svc.cluster.local (10.42.0.8) at 1e:87:4b:df:77:2a [ether] on eth0

? (10.42.0.1) at 4e:b8:bd:7b:10:7b [ether] on eth0
10-42-0-5.kube-dns.kube-system.svc.cluster.local (10.42.0.5) at ba:e7:34:61:0a:bc [ether] on eth0
10-42-0-10.hostnames.default.svc.cluster.local (10.42.0.10) at 4e:2b:cf:de:e2:9e [ether] on eth0
`

how strange it is, I actually can reach the hostnames service when run wget, but nslookup failed. I guess it is something wrong on forwarding the packet from pod to service, or wise verse.

Do we have tube-proxy or ipvs to map between service and pods?

[liyimeng]

OK, I see that we use kube-proxy, at least iptables for this. String enough, nslookup work on the host!
just not inside the pods!

`
nslookup www.google.com 10.43.0.10
Server: 10.43.0.10
Address 1: 10.43.0.10

Name: www.google.com
Address 1: 216.58.207.196 arn11s04-in-f4.1e100.net
Address 2: 2a00:1450:400e:809::2004 ams15s32-in-x04.1e100.net
alpine:/home/alpine/k3s/dist/artifacts# nslookup hostnames.default.svc.cluster.local 10.43.0.10
Server: 10.43.0.10
Address 1: 10.43.0.10

nslookup: can't resolve 'hostnames.default.svc.cluster.local': Name does not resolve
`

[liyimeng]

BTW, ip forwarding is on

alpine:/home/alpine/k3s/dist/artifacts# cat /proc/sys/net/ipv4/ip_forward 1

[ibuildthecloud]

@liyimeng is there any way I can reproduce your setup?

[liyimeng]

@ibuildthecloud Here is what I have done:

• create a KVM VM
• install alpine linux, v3.9.0 likely.
• install necessary software, like git and docker
• check out latest k3s
• build and run k3s
• I then see helm install job never success, the log says google is not reachable
• ./k3s kubectl run -i --tty busybox --image=busybox --restart=Never -- sh
• try to make an nslookup www.google.com in the pod, it fails for me.

[aaliddell]

Seen the same issue when installing on an existing system. When running on a clean install, there are no issues.

After some testing, the issue appears to be in having existing iptables rules that have a default DROP policy on the INPUT. After setting the input policy to ACCEPT, the issue appears to be resolved. Therefore, either a note needs to be added to the docs or k3s needs to setup its own iptables rules on the INPUT chain to insure that traffic does not hit the default policy, which is usually DROP or REJECT for security reasons.

[jose-sanchezm]

In a fresh installation over CentOS 7.5 I'm getting the same issue:

# kubectl get pods --all-namespaces
NAMESPACE     NAME                         READY   STATUS             RESTARTS   AGE
default       ds4m-0                       1/1     Running            0          25m
kube-system   coredns-7748f7f6df-6j8kh     0/1     CrashLoopBackOff   9          25m
kube-system   helm-install-traefik-bprt6   0/1     CrashLoopBackOff   9          25m

# kubectl logs coredns-7748f7f6df-6j8kh --namespace kube-system
E0305 16:36:17.678604       1 reflector.go:205] github.com/coredns/coredns/plugin/kubernetes/controller.go:322: Failed to list *v1.Namespace: Get https://10.43.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.43.0.1:443: connect: no route to host
E0305 16:36:19.680598       1 reflector.go:205] github.com/coredns/coredns/plugin/kubernetes/controller.go:322: Failed to list *v1.Namespace: Get https://10.43.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.43.0.1:443: connect: no route to host
.:53
2019-03-05T16:36:21.676Z [INFO] CoreDNS-1.3.0
2019-03-05T16:36:21.676Z [INFO] linux/amd64, go1.11.4, c8f0e94
CoreDNS-1.3.0
linux/amd64, go1.11.4, c8f0e94
2019-03-05T16:36:21.676Z [INFO] plugin/reload: Running configuration MD5 = 3ef0d797df417f2c0375a4d1531511fb
E0305 16:36:23.686625       1 reflector.go:205] github.com/coredns/coredns/plugin/kubernetes/controller.go:322: Failed to list *v1.Namespace: Get https://10.43.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.43.0.1:443: connect: no route to host
E0305 16:36:23.690587       1 reflector.go:205] github.com/coredns/coredns/plugin/kubernetes/controller.go:315: Failed to list *v1.Service: Get https://10.43.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.43.0.1:443: connect: no route to host

# kubectl logs helm-install-traefik-bprt6 --namespace kube-system
+ export HELM_HOST=127.0.0.1:44134
+ HELM_HOST=127.0.0.1:44134
+ helm init --client-only
+ tiller --listen=127.0.0.1:44134 --storage=secret
[main] 2019/03/05 16:35:55 Starting Tiller v2.12.3 (tls=false)
[main] 2019/03/05 16:35:55 GRPC listening on 127.0.0.1:44134
[main] 2019/03/05 16:35:55 Probes listening on :44135
[main] 2019/03/05 16:35:55 Storage driver is Secret
[main] 2019/03/05 16:35:55 Max history per release is 0
Creating /root/.helm 
Creating /root/.helm/repository 
Creating /root/.helm/repository/cache 
Creating /root/.helm/repository/local 
Creating /root/.helm/plugins 
Creating /root/.helm/starters 
Creating /root/.helm/cache/archive 
Creating /root/.helm/repository/repositories.yaml 
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com 
Error: Looks like "https://kubernetes-charts.storage.googleapis.com" is not a valid chart repository or cannot be reached: Get https://kubernetes-charts.storage.googleapis.com/index.yaml: dial tcp: lookup kubernetes-charts.storage.googleapis.com on 10.43.0.10:53: read udp 10.42.1.2:48204->10.43.0.10:53: read: no route to host

My firewalld configuration:

# firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: eth0
  sources: 
  services: dhcpv6-client ssh
  ports: 4789/udp 6443/tcp
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

br_netfilter module is loaded:

# lsmod | grep netfilter
br_netfilter           22256  0 
bridge                151336  2 br_netfilter,ebtable_broute

Which extra rules do I have to configure to get it working?

[aaliddell]

I've not used firewalld before, but essentially you need to add a rule equivalent to this iptables rule:

iptables -I INPUT 1 -i cni0 -s 10.42.0.0/16 -j ACCEPT

This rule says, permit incoming packets from interface (bridge) cni0, with source in range 10.42.0.0/16. If you wanted to be more granular, you could individually open each port, rather than accept any.

[jose-sanchezm]

I've added the rule and core-dns logs way less errors (although there are still some) but helm-install-traefik continues crashing continuously with the same error. Do I need another rule for it?

[aaliddell]

Does firewalld have a log somewhere of what packets it is blocking or a way to enable such a log? If so, look there to see what might still be getting dropped.

[briandealwis]

I'm seeing this with VMWare Photon 3.0. Adding @aaliddell's snippet to /etc/systemd/scripts/ip4save has done the trick.

[sahlex]

@briandealwis can you please point out which exact snippet you are referring to?

[briandealwis]

This comment:

iptables -I INPUT 1 -i cni0 -s 10.42.0.0/16 -j ACCEPT