Improve support for MetalLB in L2 mode (Kubeproxy IPVS strict ARP)

[rastislavs]

User Story
MetalLB in L2 mode relies on ARP: for each virtual service IP (LoadBalancer IP), one leader-elected node in the cluster should respond to an ARP request for a LoadBalancer IP.

In case that IPVS mode is used for kube-proxy (which is the default in KKP), this concept is a bit broken. Each node in the cluster has an kube-ipvs0 interface with all LoadBalancer service IPs applied on it. This causes that by default, every node (not only the leader-elected one) replies to an ARP request for a LoadBalancer IP.

This should not cause any big issues for short-lived requests, as any node in the cluster is capable of forwarding the packet to the target service backends. But in case when persistent connections are needed, this may cause connections drops, whenever an upstream router issues a new ARP request for a LoadBalancer IP - as it will use the first answer, the target node may change randomly.

This issue can be reproduced more frequently if the upstream router does not extend the timeout of entries in the ARP cache with each received frame, but re-issues ARP request periodically - which is often the case for firewalls, to prevent ARP spoofing.

Acceptance criteria
Allow to enable strictARP mode in the KubeProxyConfiguration in KKP, which should prevent the issue.

[youssefazrak]

Related to metallb/metallb#153

[toschneck]

As this issue effect multiple customer installation, we should potential provide some fix soon. As potential most folks are not aware of this.

For now I created a runbook, how this can be fixed manual:
kubermatic/community-components#25

@shaase-ctrl please think how much prio this topic has for KKP and KubeOne:
kubermatic/kubeone#1409

[rastislavs]

We just discussed on the sig-networking meeting that we would like to make the strictARP mode configurable in the cluster API, to allow disabling it in case of any issues in specific environments.

[mate4st]

Just want to confirm that, according to @rastislavs, the default value for strictARP should be true?

[youssefazrak]

@WeirdMachine yes, it should be set to true.
Metallb has an updated doc: https://metallb.universe.tf/installation/