Get rid of leader election for layer-2 mode

[danderson]

Layer 2 modes (ARP and NDP) requires a single owner machine for each service IP. Currently we do this by running master election and having the winner own all IPs.

This is a little sub-optimal in several ways:

• Leader election in k8s is kinda expensive in terms of control plane qps
• Electing a machine regardless of what it's running means we are forced to use externalTrafficPolicy=Cluster, so we lose source IP information
• We cannot shard traffic load by IP (arguably this is a feature, but it's not a particularly compelling one)

So, here's a proposal: let's get rid of the leader election, and replace it with a deterministic node selection algorithm. The controller logic remains unchanged (allocates an IP). On the speaker, we would do the following:

• Based on the Endpoints object for the service, construct a list of nodes that have a pod for that service. In python pseudocode, that list would be [x.node for x in endpoints]
• Hash the node names and service name together, to produce a service-dependent (but deterministic) set of hashes for the nodes.
• Do a weighted alphabetical sort of the hashes, such that the first element of the list is the alphabetically first hash with the largest number of local pods
• Pick that first element as the "owner" of this service, and make it announce that IP.

This algorithm results in a couple of properties:

• Each IP can be owned by different nodes. In fact, due to the service-dependent hashing, it's likely that services will uniformly distribute throughout the cluster.
• Services will prefer to attach to nodes that have multiple serving pods for the service, to better distribute load.
• There is no explicit locking, similar to consistent hashing in Maglev each speaker simply arrives independently at the same conclusions.
• We can once again allow externalTrafficPolicy: Local for layer2 mode services, which removes one of the major downsides of using ARP/NDP today (no client IP visibility).
• One downside is that split-brain is more likely, because if a node gets cut off from the control plane it may not realize that conditions around it have changed, and so we might end up with multiple machines thinking that they own an IP. We can either accept this as a tradeoff (ARP and NDP behave somewhat gracefully in the presence of a split brain), or keep a concept similar to leader election that just pings a lease in the cluster, and speakers who don't see that lease increasing stop all advertisement on the assumption that they've lose communications with the control plane. This however has the significant downside that the cluster will stop all announcements if the control plane goes down, rather than gracefully keep announcing the last known state. I think I would prefer just accepting the split brain in that case.

@miekg @mdlayher Thoughts on this proposal?

[miekg]

[ Quoting <notifications@github.com> in "[google/metallb] Consider getting r..." ]

This algorithm results in a couple of properties:

This sgtm (also explains why I see 4 qps of config traffic w/ leader election) I also agree with allowing a split brain situation, instead of doing something complex. /Miek

…

-- Miek Gieben

[mdlayher]

This all seems reasonable to me. My only big concern would be ending up with hotspots due to whatever the hashing algorithm is doing.

[steven-sheehy]

This would definitely solve two of our problems with MetalLB layer2 mode: HA of IP announcing (we only have one master so can't leader elect if master down) and source IP visibility. Any idea what release this is targeted for?

[danderson]

My rough plan is to have BGPv6 support and this bug in 0.7.

Unfortunately no timeline for when that'll happen, since I'm a lone developer in his spare time :(

[mrbobbytables]

We just encountered this issue today ourselves.

The proposal looks good, and I am in favor of just letting ARP/NDP sort it out. This seems to be more in line with Kubernetes as a whole with workload and services not being completely dependent on the control plane being available.

re: @mdlayher thoughts on hotspots -- I cannot speak for everyone's use-cases, but for us this is a specific need where we map services to 'edge-nodes'. We are already targeting specific nodes with these deployment, so its somewhat deterministic already.
One possibility to refine this, would be to check for an optional annotation on the service where a node, or ordered list of nodes could be supplied to function as a manual version of the 'node selection algorithm'. This would allow for some level of operator control or override regarding which nodes would announce the IP.

[anandsinghkunwar]

Does it make sense to add the service namespace as well in the hash in nodename + servicename hash?

[xeor]

Should this works now, that being getting the real ip of the requester...

I am using..

• arp mode (also tried bgp mode)
• LoadBalancer with externalTrafficPolicy: "Local"
• LoadBalancer > Ingress (nginx-controller) > Service (ClusterIP) > k8s.gcr.io/echoserver:1.4
• metallb 0.7.3 from helm..
• nginx ingress 0.22.0 from helm..

Still getting x-real-ip=10.40.0.0.. Or am I missing something? Been fiddling a lot with this now, so now I'm wondering if this issue is solved or not..

[zerkms]

@xeor check ingress access logs first, does it have the real ip address there

[xeor]

Just checked it's log during all my experimenting.. It shows 10.40.0.0 everytime..

[zerkms]

What is even 10.40.0.0?

[danderson]

Please use the mailing list for support questions, not a closed bug :)

[xeor]

Sorry, but I wanted to confirm that this was actually fixed and should have worked.. If it should, I'll continue my quest on figuring it out.. :)