Kubeapi not accessible on gateway loadbalancer IP

[nitinsatish]

In 5th step I am not able to get gateway working. All the control nodes are working and responding on 6443. However the LB isn't responding. Not sure how to troubleshoot the ipvs LB.

ubuntu@gateway:~$ sudo ipvsadm -L
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  kubernetes.kubenet:6443 wrr
  -> control0.kubenet:6443        Route   1      52         0
  -> control1.kubenet:6443        Route   1      52         0
  -> control2.kubenet:6443        Route   1      53         0

But connecting to LB is timing out.

curl -v --cacert auth/ca.pem https://kubernetes:6443/healthz
*   Trying 192.168.1.21:6443...


# telnet 192.168.1.21 6443
Trying 192.168.1.21...

The IP is pingable though.

# ping 192.168.1.21
PING 192.168.1.21 (192.168.1.21) 56(84) bytes of data.
64 bytes from 192.168.1.21: icmp_seq=1 ttl=64 time=0.363 ms
64 bytes from 192.168.1.21: icmp_seq=2 ttl=64 time=0.222 ms
64 bytes from 192.168.1.21: icmp_seq=3 ttl=64 time=0.197 ms
64 bytes from 192.168.1.21: icmp_seq=4 ttl=64 time=0.173 ms
^C
--- 192.168.1.21 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3060ms
rtt min/avg/max/mdev = 0.173/0.238/0.363/0.073 ms

[ghik]

Hi Nitin,

I'm assuming you're trying to contact the gateway from your host machine (Linux or macOS?).
Try contacting kubernetes:6443 from the VMs, in particular from the gateway VM itself.
Your ipvsadm output shows active connections, which likely means that nodes are successfully talking to the Kubernetes API.

If everything works fine from VMs, then I would suspect the problem is somewhere on your host system, e.g. a firewall blocking the traffic. It's hard to say more without digging deeper into your system settings. Things you can try:

• inspecting the traffic with tcpdump or Wireshark
• inspecting your firewall rules

[nitinsatish]

I tried redoing the gateway VM(reset the qcow image and configured it again) as it looks like the problem is in connecting to gateway from host. gateway -> control and even host ->control VMs on 6443 are working fine. Still the issue continues. :(

I see that all ipvs connections are in SYN_RECV state. None in ESTABLISHED state.

ubuntu@gateway:~$ sudo ipvsadm -lnc |head -40
IPVS connection entries
pro expire state       source             virtual            destination
TCP 00:55  SYN_RECV    192.168.1.16:44972 192.168.1.21:6443  192.168.1.12:6443
TCP 00:56  SYN_RECV    192.168.1.13:51946 192.168.1.21:6443  192.168.1.13:6443
TCP 00:14  SYN_RECV    192.168.1.13:51860 192.168.1.21:6443  192.168.1.11:6443
TCP 00:56  SYN_RECV    192.168.1.11:40926 192.168.1.21:6443  192.168.1.13:6443
TCP 00:46  SYN_RECV    192.168.1.12:47210 192.168.1.21:6443  192.168.1.11:6443
TCP 00:32  SYN_RECV    192.168.1.11:35258 192.168.1.21:6443  192.168.1.12:6443
TCP 00:58  SYN_RECV    192.168.1.13:45610 192.168.1.21:6443  192.168.1.11:6443
TCP 00:55  SYN_RECV    192.168.1.16:52884 192.168.1.21:6443  192.168.1.13:6443
TCP 00:49  SYN_RECV    192.168.1.13:55946 192.168.1.21:6443  192.168.1.12:6443
TCP 00:16  SYN_RECV    192.168.1.11:42584 192.168.1.21:6443  192.168.1.11:6443
TCP 00:10  SYN_RECV    192.168.1.12:44950 192.168.1.21:6443  192.168.1.12:6443
TCP 00:21  SYN_RECV    192.168.1.12:48412 192.168.1.21:6443  192.168.1.12:6443
TCP 00:11  SYN_RECV    192.168.1.12:48428 192.168.1.21:6443  192.168.1.13:6443
TCP 00:42  SYN_RECV    192.168.1.13:55902 192.168.1.21:6443  192.168.1.13:6443
TCP 00:57  SYN_RECV    192.168.1.11:52176 192.168.1.21:6443  192.168.1.12:6443
TCP 00:09  SYN_RECV    192.168.1.13:55216 192.168.1.21:6443  192.168.1.13:6443
TCP 00:55  SYN_RECV    192.168.1.11:52156 192.168.1.21:6443  192.168.1.13:6443
TCP 00:21  SYN_RECV    192.168.1.12:48436 192.168.1.21:6443  192.168.1.11:6443
TCP 00:34  SYN_RECV    192.168.1.11:35286 192.168.1.21:6443  192.168.1.11:6443
TCP 00:38  SYN_RECV    192.168.1.11:45620 192.168.1.21:6443  192.168.1.11:6443
TCP 00:12  SYN_RECV    192.168.1.16:60748 192.168.1.21:6443  192.168.1.12:6443
TCP 00:56  SYN_RECV    192.168.1.13:45590 192.168.1.21:6443  192.168.1.13:6443
TCP 00:48  SYN_RECV    192.168.1.11:40882 192.168.1.21:6443  192.168.1.11:6443
TCP 00:17  SYN_RECV    192.168.1.15:36120 192.168.1.21:6443  192.168.1.13:6443
TCP 00:38  SYN_RECV    192.168.1.12:38298 192.168.1.21:6443  192.168.1.13:6443
TCP 00:52  SYN_RECV    192.168.1.12:43442 192.168.1.21:6443  192.168.1.13:6443
TCP 00:07  SYN_RECV    192.168.1.11:34888 192.168.1.21:6443  192.168.1.13:6443
TCP 00:09  SYN_RECV    192.168.1.12:44930 192.168.1.21:6443  192.168.1.11:6443
TCP 00:41  SYN_RECV    192.168.1.14:51002 192.168.1.21:6443  192.168.1.13:6443

However, I don't see any iptables rules preventing this on host or gateway.

ubuntu@gateway:~$ sudo iptables -nL
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Host

# iptables -nL
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Sorry about troubling you with this.

[ghik]

Oh, so the connections aren't actually working, that's a good info.

What's your host system? (i.e. what does uname -a say?)

[nitinsatish]

My host system is an ESXi VM.

$ uname -a
Linux nitin-vm 5.15.0-25-generic #25-Ubuntu SMP Wed Mar 30 15:54:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

BTW, I am able to reach to kubeapi directly on control VMs.

# curl --cacert auth/ca.pem https://control0:6443/healthz ; echo
ok

My assumption is that this is an issue with gateway VM.

[ghik]

Okay, let's check the network setup on gateway & control VMs. Make sure that you've done everything listed in the guide for the control & gateway VMs, i.e.

• make sure the gateway machine has the virtual IP properly assigned (you can check it with ip addr, but your previous ping results suggest that everything's fine here)
• make sure that control machines also recognize the virtual IP as their own (check with ip addr, the VIP should be assigned to loopback interface)
• make sure that arp_ignore and arp_annouce kernel options are properly set on control machines

[nitinsatish]

The issue was IP 192.168.1.21 had got unbound from control VMs. I think this happened when I rebooted the control VMs. After adding 192.168.1.21 to lo, everything is working good.

Thanks for helping me and sorry for taking your time. Somehow the idea that gateway VM is the problem got stuck in my head. I should checked the control VMs.