CentOS: DNS service access issues

[murali-reddy]

I never ran into this issue on Debian or Ubuntu. Kube-router does not have any distro specific logic, but service proxy provided by kube-router seems to have some problem on CentOS.

In the below e.g from cluster deployed with kube-router on CentOS, frontend of guestbook app trying to resolve redis service, but it fails.

08:41:51.510914 IP 100.96.2.4.57965 > ip-172-20-34-141.domain: 41256+ A? redis-slave.default.svc.cluster.local. (55)
08:41:51.510946 IP 100.96.2.4.57965 > 100.96.2.2.domain: 41256+ A? redis-slave.default.svc.cluster.local. (55)
08:41:51.510969 IP 100.96.2.4.57965 > ip-172-20-34-141.domain: 48752+ AAAA? redis-slave.default.svc.cluster.local. (55)
08:41:51.510973 IP 100.96.2.4.57965 > 100.96.2.2.domain: 48752+ AAAA? redis-slave.default.svc.cluster.local. (55)
08:41:51.511300 ARP, Reply 100.96.2.4 is-at 0a:58:64:60:02:04 (oui Unknown), length 28
08:41:51.511320 IP 100.96.2.4 > 100.96.2.2: ICMP 100.96.2.4 udp port 57965 unreachable, length 145
08:41:51.511422 IP 100.96.2.4 > 100.96.2.2: ICMP 100.96.2.4 udp port 57965 unreachable, length 107

On cluster deployed with Debian it succeeds

09:37:19.079264 IP 100.96.1.4.52150 > 100.64.0.10.domain: 6494+ A? redis-slave.default.svc.cluster.local. (55)
09:37:19.079300 IP 100.96.1.4.52150 > 100.96.1.2.domain: 6494+ A? redis-slave.default.svc.cluster.local. (55)
09:37:19.079323 IP 100.96.1.4.52150 > 100.64.0.10.domain: 14286+ AAAA? redis-slave.default.svc.cluster.local. (55)
09:37:19.079328 IP 100.96.1.4.52150 > 100.96.1.2.domain: 14286+ AAAA? redis-slave.default.svc.cluster.local. (55)
09:37:19.079681 ARP, Reply 100.96.1.4 is-at 0a:58:64:60:01:04 (oui Unknown), length 28
09:37:19.079795 IP 100.96.1.4.33874 > 100.70.57.10.6379: Flags [S], seq 2568390226, win 29200, options [mss 1460,sackOK,TS val 212631 ecr 0,nop,wscale 9], length 0
09:37:19.080819 IP 100.96.1.4.33874 > 100.70.57.10.6379: Flags [.], ack 1573856540, win 58, options [nop,nop,TS val 212631 ecr 212233], length 0
09:37:19.080859 IP 100.96.1.4.33874 > 100.70.57.10.6379: Flags [P.], seq 0:27, ack 1, win 58, options [nop,nop,TS val 212631 ecr 212233], length 27: RESP "GET" "messages"

Need to find the root cause.

[murali-reddy]

Ok. its well documented step and discussed in this bug kubernetes/kubernetes#33798

https://kubernetes.io/docs/concepts/cluster-administration/network-plugins/#network-plugin-requirements

the plugin connects containers to a Linux bridge, the plugin must set the net/bridge/bridge-nf-call-iptables sysctl to 1 to ensure that the iptables proxy functions correctly

Things run fine once we set net.bridge.bridge-nf-call-iptables is set to 1