Enable externalTrafficPolicy=Local for layer2 mode

[zsek]

Is this a bug report or a feature request?: Feature request

What happened:
Having no clue about BGP, I opted for the Layer 2 setup.
All works fine except one little detail. Layer 2 does not honor the externalTrafficPolicy (as documented). Unfortunately this is a show stopper in case of web services since you have to know the source IP for any production web service for many reasons (from simple stats to identifying abuse/hack attempts etc). Could this be changed to use NodePort instead of ClusterIP?

What you expected to happen:
Well it works as expected :)

It would be great to get the source IP on REMOTE_ADDR on the ingress controller (ingress-nginx in my case). This is possible if metallb would honor externalTrafficPolicy and the ingress-controller is deployed as a DaemonSet on all nodes running a metallb speaker.

How to reproduce it (as minimally and precisely as possible):
Metallb is configured as a loadbalancer
Ingress-nginx is configured to handle ingress
nginx pod is configured to display a phpinfo(); page.

The phpinfo page gives cluster specific IPs as $_SERVER['REMOTE_ADDR'] and $_SERVER['HTTP_X_FORWARDED_FOR']

$_SERVER['HTTP_X_FORWARDED_FOR'] 10.244.0.0
$_SERVER['REMOTE_ADDR'] 10.244.1.80
(Note: 10.244.1.80 is the IP of nginx-ingress-controller)

Anything else we need to know?:

Environment:

• MetalLB version: 0.62
• Kubernetes version: 1.9.7
• BGP router type/version: n/a - layer 2 mode used
• OS (e.g. from /etc/os-release): Ubuntu 16.04.4 LTS
• Kernel (e.g. uname -a):4.4.0-122-generic

PS: Thank you for the great work!! :)

[danderson]

Thanks for the suggestion! I'm already planning to do this, it's tracked in #195 because removing leader election is a prerequisite to make this work.

You can read the proposed design in the other bug, but the bottom line is: yes, we can make this work, but the tradeoff is that only 1 machine will receive traffic. It will behave completely like a VRRP active/passive failover. Unfortunately, due to the limitations of layer 2 protocols, that's the best we can do :(

I'll keep this bug open in addition to #195, so that it's obvious to other people that we're planning this, the other bug doesn't make it obvious.