adding iptables-wrappers script to entrypoint

[dockerpac]

By default, iptables binary running in kube-vip image will use legacy iptables, even on systems that uses nft (tested on Rocky Linux 8.6).
It's not recommended to mix usage of both iptables-legacy and iptables-nft.

This PR adds the execution of iptables-wrapper-installer.sh from https://github.com/kubernetes-sigs/iptables-wrappers/blob/master/iptables-wrapper-installer.sh.
This script will automatically detect the version of iptables used on the system based on some auto detection mechanisms.

[thebsdbox]

I'll test this out, but in practice (until we move to eBPF) leaving legacy iptables managing the mangling and egress NAT work has been stable as it means Kubernetes doesn't know it is there.

[dockerpac]

I'll test this out, but in practice (until we move to eBPF) leaving legacy iptables managing the mangling and egress NAT work has been stable as it means Kubernetes doesn't know it is there.

Hmm, at least in my configuration (Mirantis Kubernetes Engine 3.6 on Rocky Linux 8.6), rules are added to legacy mangle and NAT tables, but it seems they are not used.

pac@mke-0:~$ kubectl -n kube-system exec -it kube-vip-ds-92fdb  -- sh -c 'echo "iptables-legacy" && iptables -t nat -nvL POSTROUTING && echo "iptables-nft" && iptables-nft -t nat -nvL POSTROUTING'
iptables-legacy
Chain POSTROUTING (policy ACCEPT 564 packets, 34298 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 SNAT       all  --  *      *       10.250.163.3         0.0.0.0/0            mark match 0x40/0x40 /* a3ViZS12aXAK=kube-vip */ to:192.168.10.161
iptables-nft
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
  535 32118 cali-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:O3lYWMrLQYEMJtB5 */
 1290 77830 KUBE-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */
  221 13260 MASQUERADE  all  --  *      !br-5dce58eb37a1  172.19.0.0/16        0.0.0.0/0
    0     0 MASQUERADE  all  --  *      docker_gwbridge  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match src-type LOCAL
   32  1920 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0
    0     0 MASQUERADE  all  --  *      !docker_gwbridge  172.18.0.0/16        0.0.0.0/0
    0     0 MASQUERADE  tcp  --  *      *       172.19.0.2           172.19.0.2           tcp dpt:12378
    0     0 MASQUERADE  tcp  --  *      *       172.19.0.2           172.19.0.2           tcp dpt:6444
    0     0 MASQUERADE  tcp  --  *      *       172.19.0.2           172.19.0.2           tcp dpt:2376
    0     0 MASQUERADE  tcp  --  *      *       172.19.0.3           172.19.0.3           tcp dpt:12389
pac@mke-0:~$
pac@mke-0:~$ # Traffic from a Pod inside the cluster with a Service with egress annotation that contacts an application outside of the cluster
pac@mke-0:~$ kubectl exec -it deploy/whoami -- sh -c 'for i in $(seq 100) ; do curl -s 192.168.10.150 > /dev/null ; done'
pac@mke-0:~$
pac@mke-0:~$
pac@mke-0:~$ kubectl -n kube-system exec -it kube-vip-ds-92fdb  -- sh -c 'echo "iptables-legacy" && iptables -t nat -nvL POSTROUTING && echo "iptables-nft" && iptables-nft -t nat -nvL POSTROUTING'
iptables-legacy
Chain POSTROUTING (policy ACCEPT 571 packets, 34718 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 SNAT       all  --  *      *       10.250.163.3         0.0.0.0/0            mark match 0x40/0x40 /* a3ViZS12aXAK=kube-vip */ to:192.168.10.161
iptables-nft
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
  642 38538 cali-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:O3lYWMrLQYEMJtB5 */
 1297 78250 KUBE-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */
  221 13260 MASQUERADE  all  --  *      !br-5dce58eb37a1  172.19.0.0/16        0.0.0.0/0
    0     0 MASQUERADE  all  --  *      docker_gwbridge  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match src-type LOCAL
   32  1920 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0
    0     0 MASQUERADE  all  --  *      !docker_gwbridge  172.18.0.0/16        0.0.0.0/0
    0     0 MASQUERADE  tcp  --  *      *       172.19.0.2           172.19.0.2           tcp dpt:12378
    0     0 MASQUERADE  tcp  --  *      *       172.19.0.2           172.19.0.2           tcp dpt:6444
    0     0 MASQUERADE  tcp  --  *      *       172.19.0.2           172.19.0.2           tcp dpt:2376
    0     0 MASQUERADE  tcp  --  *      *       172.19.0.3           172.19.0.3           tcp dpt:12389

[dockerpac]

Now with the new image that is using iptables-nft, Egress traffic is correctly using the VIP, and we can see that the rule is used

$ kubectl -n kube-system exec -it kube-vip-ds-kvtg7  -- sh -c 'echo "iptables-nft" && iptables -t nat -nvL POSTROUTING'
iptables-nft
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 SNAT       all  --  *      *       64.0.0.0             0.0.0.0/0            mark match 0x40/0x40 /* a3ViZS12aXAK=kube-vip */ to:192.168.10.161
 1207 72410 KUBE-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */
  257 15420 MASQUERADE  all  --  *      !br-dbccedf581b4  172.19.0.0/16        0.0.0.0/0
    0     0 MASQUERADE  all  --  *      docker_gwbridge  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match src-type LOCAL
   32  1920 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0
    0     0 MASQUERADE  all  --  *      !docker_gwbridge  172.18.0.0/16        0.0.0.0/0
    0     0 MASQUERADE  tcp  --  *      *       172.19.0.2           172.19.0.2           tcp dpt:12378
    0     0 MASQUERADE  tcp  --  *      *       172.19.0.2           172.19.0.2           tcp dpt:6444
    0     0 MASQUERADE  tcp  --  *      *       172.19.0.2           172.19.0.2           tcp dpt:2376
    0     0 MASQUERADE  tcp  --  *      *       172.19.0.3           172.19.0.3           tcp dpt:12389
   74  4454 cali-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:O3lYWMrLQYEMJtB5 */

$ kubectl exec -it deploy/whoami -- sh -c 'for i in $(seq 100) ; do curl -s 192.168.10.150 > /dev/null ; done'

$ kubectl -n kube-system exec -it kube-vip-ds-kvtg7  -- sh -c 'echo "iptables-nft" && iptables -t nat -nvL POSTROUTING'
iptables-nft
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
  100  6000 SNAT       all  --  *      *       64.0.0.0             0.0.0.0/0            mark match 0x40/0x40 /* a3ViZS12aXAK=kube-vip */ to:192.168.10.161
 1209 72530 KUBE-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */
  257 15420 MASQUERADE  all  --  *      !br-dbccedf581b4  172.19.0.0/16        0.0.0.0/0
    0     0 MASQUERADE  all  --  *      docker_gwbridge  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match src-type LOCAL
   32  1920 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0
    0     0 MASQUERADE  all  --  *      !docker_gwbridge  172.18.0.0/16        0.0.0.0/0
    0     0 MASQUERADE  tcp  --  *      *       172.19.0.2           172.19.0.2           tcp dpt:12378
    0     0 MASQUERADE  tcp  --  *      *       172.19.0.2           172.19.0.2           tcp dpt:6444
    0     0 MASQUERADE  tcp  --  *      *       172.19.0.2           172.19.0.2           tcp dpt:2376
    0     0 MASQUERADE  tcp  --  *      *       172.19.0.3           172.19.0.3           tcp dpt:12389
   76  4574 cali-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:O3lYWMrLQYEMJtB5 */

By the way, I had a conflict with Calico which is deployed in Insert mode by default and is adding the cali-POSTROUTING rule at the beginning of the POSTROUTING chain by default, and prevents the kube-vip SNAT rule to be executed. I had to reconfigure it in append mode, but I'll create another issue for that, not sure if that's the right way to deal with it.

kubectl -n kube-system exec deploy/calico-kube-controllers -c calico-kube-controllers -- calicoctl patch felixConfiguration default --patch '{"spec":{"chainInsertMode": "append"}}'