connection from hostNetwork to remote Pod or ClusterIP does not work

[mangelajo]

In k8s, connecting from a worker node (or pod with hostNetwork) to another pod or clusterIP works, see example:

$ kubectl get pods -o wide
NAME                       READY   STATUS    RESTARTS   AGE   IP           NODE          NOMINATED NODE   READINESS GATES
netshoot-785ffd8c8-t2tn5   1/1     Running   509        21d   10.42.4.34   subm-a-w-1    <none>           <none>
test-5db4cfb4bd-z5q7l      1/1     Running   0          16d   10.42.3.24   subm-a-gw-1   <none>           <none>

$ ./netshoot-hostnet.sh subm-a-w-1
bash-5.0# ping 10.42.4.34
PING 10.42.4.34 (10.42.4.34) 56(84) bytes of data.
64 bytes from 10.42.4.34: icmp_seq=1 ttl=64 time=0.063 ms
64 bytes from 10.42.4.34: icmp_seq=2 ttl=64 time=0.072 ms
^C
--- 10.42.4.34 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.063/0.067/0.072/0.004 ms

bash-5.0# ping 10.42.3.24
PING 10.42.3.24 (10.42.3.24) 56(84) bytes of data.
64 bytes from 10.42.3.24: icmp_seq=1 ttl=63 time=0.691 ms
64 bytes from 10.42.3.24: icmp_seq=2 ttl=63 time=0.639 ms
^C
--- 10.42.3.24 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.639/0.665/0.691/0.026 ms

bash-5.0# ip a | grep 10.42 -B 2
6: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8951 qdisc noqueue state UNKNOWN group default 
    link/ether 7e:4f:17:bc:77:db brd ff:ff:ff:ff:ff:ff
    inet 10.42.4.0/32 scope global flannel.1
bash-5.0# exit

The current implementation fails to work this way across clusters, because of the way vx-submariner and the iptable rules are configured, as soon as a remote PodIP or ClusterIP is targeted, the routing rules and vx-submariner implementation, the connection will be sourced with the vx-submariner IP, which is not routable across clusters.

This was discovered trying to implement the use case described in the following diagram, which works in a single cluster:

The solution to the problem was to ensure that any traffic leaving vx-submariner for the hostNetwork would use the PodIP network address of the worker node.

[mangelajo]

We also need to add additional testing verify this type of connectivity.

[sridhargaddam]

The necessary fix is now merged in Submariner repo.