Failover time very high in layer2 mode

[ghaering]

Is this a bug report or a feature request?:

Both, probably.

What happened:

I tested with layer 2 mode and simulated node failure by shutting down the node that the load balancer IP was on. What then happened it that it took approx. 5 minutes for the IP to be switched to the other node in the cluster (1 master, 2 nodes). After much experimentation I came to the conclusion that the node being down and being "NotReady" did not initiate the switch of the IP address. The 5 minute timeout seems to be caused by the default pod eviction timeout of Kubernetes, which is 5 minutes. That means it takes 5 minutes for a pod on a node that is not available to be deleted. Default "node monitor grace period is 40 seconds, btw.". So that means it currently takes almost 6 minutes with default confguration for an IP address to be switched.

I made things a lot better by decreasing both settings like this:
- --pod-eviction-timeout=20s
- --node-monitor-grace-period=20s
in /etc/kubernetes/manifests/kube-controller-manager.yaml

This makes MetalLB switch the IP in case of node failure in the sub-minute range.

What you expected to happen:

To be honest what I would expect that the whole process takes maybe max. 5 seconds.

How to reproduce it (as minimally and precisely as possible):

Create a Kubernetes 1.11.1 cluster with kubeadm (single master, two nodes). Calico networking.

kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.7.2/manifests/metallb.yaml
kubectl apply -f metallb-cfg.yml
kubectl apply -f tutorial-2.yaml

➜ metallb-test cat metallb-cfg.yml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: default
protocol: layer2
addresses:
- 10.115.195.206-10.115.195.208

Then

watch curl --connect-timeout 1 http://10.115.195.206

to see if the nginx app is reachable.

Then

kubectl logs -f --namespace metallb-system speaker-xxxxxxxxx

To see which node has the IP address assigned at the moment.
ssh into the machine and "poweroff".

Wait for how long it takes until the "watch curl" is successful again.

Anything else we need to know?:

Environment:

• MetalLB version: v0.7.2
• Kubernetes version: v1.11.1
• BGP router type/version: N/A
• OS (e.g. from /etc/os-release): CentOS 7
• Kernel (e.g. uname -a):Linux cp-k8s-ghdev02-node-01.ewslab.eos.lcl 3.10.0-693.el7.x86_64 Implement BGP add-path #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

[danderson]

This is definitely a bug. In MetalLB 0.6, failover time was at max 10 seconds, because we had explicit leadership elections and so we could detect node failure much faster. In 0.7, for scaling, I switched to trusting k8s to tell us the state of the cluster. Unfortunately, I assumed that it was way better than it actually is :(

So, failover time for layer2 is definitely unacceptable in 0.7. Node failures should recover in seconds, not minutes. Now, how can we fix that?...

The obvious choice is to reintroduce leader election. But, to make 0.7's features work, now the leader election has to be per-service. That means a linearly growing amount of k8s control plane traffic for each LB service, more CPU consumption to manage the leadership state, and probably less good distribution of services across nodes because a leadership race is more open to racing.

For reference, if we set the leader election timeout to 10s, that means we should ping the object every ~5s to renew our lease, so that's 0.2qps of k8s control plane traffic for every LoadBalancer service in the cluster. That can get huge pretty quickly :/

Another option would be to maintain a separate "healthy speakers" object of some kind, where each speaker pings its liveness periodically. Then each speaker could still do stateless decisions for leadership, and just filter the list of potential pods based on which speakers are alive. This is still leader election, but now the qps cost is O(nodes) instead of O(services). This won't work either I think, many services have way more nodes than LB services.

Either way, we need to resolve this, because 0.7 has made the failover time of layer2 mode unacceptable.

[danderson]

Okay, one more possibility: make the speakers explicitly probe each other for health. We bypass the control plane completely, the speakers just probe each other in a mesh so that they know exactly who is healthy and who is not. Then they can use that information to filter the list of "feasible" pods when they're calculating who is the leader.

O(n^2) network traffic, but it's very light network traffic compared to the cost of talking to the k8s control plane. There's still the risk of split brain in this setup, because there could be a network partition that triggers different health visibility in different sections of the cluster.

[danderson]

Okay, one more idea: if the node transition from Ready to NotReady happens pretty quickly, we can just make all speakers monitor all nodes, and only allow pods on Ready nodes to be considered for leadership.

That's the quickest fix in terms of the current codebase, but from your description that will only lower the failover time to ~40s, which is still high (although better than 6min). @ghaering , do I understand correctly that it takes 40s before the node status becomes NotReady?

[danderson]

Oh, and one more idea: change the architecture of MetalLB, and make the controller responsible for selecting an announcing node for each service, instead of making the speakers do it. The controller has a more global view of the world, and it could healthcheck the speakers explicitly to help it make decisions. As a bonus, it would make it much easier to have a status page on the controller that shows the full state of the cluster... Possibly. Need to think about that, it's a huge change to MetalLB.

[mxey]

MetaLB 0.7 uses endpoint readiness, which is based on pod readiness. The Kubernetes node lifecycle controller marks the Pod as no longer ready if their Node becomes NotReady. Waiting for the pod eviction timeout (default 5 minutes) should not be necessary. I am not sure why that did not work. @ghaering maybe you can try increasing the logging of the kube-controller-manager to see how it reacts to node readiness.

I think the MetalLB behavior makes sense. It matches how kube-proxy works. MetalLB doing faster IP failover does not help much if the service endpoints are partly dead, but still in the loadbalacing pool.

If 40 seconds is too high, it's possible to tweak the node monitor periods. That increases the load on the API and etcd, though, but it's feasible. There is ongoing work to make the node heartbeats cheaper.

[mxey]

Oh, and one more idea: change the architecture of MetalLB, and make the controller responsible for selecting an announcing node for each service, instead of making the speakers do it. The controller has a more global view of the world, and it could healthcheck the speakers explicitly to help it make decisions.

This would mean you have to run multiple controllers with leader election, because otherwise if you lose the node the controller runs on, you have to wait 40s + 5m for the controller pod to be evicted.

As a bonus, it would make it much easier to have a status page on the controller that shows the full state of the cluster... Possibly.

I think that's better done by collecting the Prometheus metrics from all the speakers. I have set up a simple Grafana dashboard that shows me which node is advertising which service IP.

[ghaering]

@danderson Yes, it should per default take around 40 secs to detect unhealthy nodes. The relevant setting is https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/#options

--node-monitor-grace-period duration     Default: 40s
--
  | Amount of time which we allow running Node to be unresponsive before marking it unhealthy. Must be N times more than kubelet's nodeStatusUpdateFrequency, where N means number of retries allowed for kubelet to post node status.

That's also about the time I experienced.

[ghaering]

The MetalLB controller only runs once, and from what I saw yesterday it takes 5 minutes (!) per default until a pod in "unreachable node" state is deleted and scheduled on another node. That was the other setting I wrote above that I changed from default 5 minutes to 20 seconds (- --pod-eviction-timeout=20s).
I don't think depending on Kubernetes itself is a promising approuch. OTOH implementing leader election yourself is a PITA.Could we require that there are initially at least three nodes and borrow/steal existing leader election code,like software like etcd must have implemented?

[mxey]

The MetalLB controller only runs once, and from what I saw yesterday it takes 5 minutes (!) per default until a pod in "unreachable node" state is deleted and scheduled on another node.

It's true that pod eviction by default starts after 5 minutes. But that does not affect the MetalLB speaker failover.

[danderson]

So, this is what I don't understand: in your original report, you said it took 5min for the IP failover to happen. But, given the design of MetalLB 0.7, the failover should happen after ~40s, when the failed node becomes unhealthy. Pod eviction takes longer, but MetalLB manages failover based on service endpoint health, not pod health. When the node becomes NotReady, the endpoint should move into notReadyAddresses, and that's when failover should happen.

Can you confirm that it took >5min for the failover to happen? If so, that could indicate a bug in the leader selection code, where it's not ignoring unready pods. That would be bad :).

If the failover does happen in 40 seconds... Then it's mostly "working as intended", because we're trusting k8s to tell us if the service is healthy. But, I still think 40s is quite long, and I'd like to think about ways to make the failover faster.

It would be much easier if I also controlled the kube-proxy dataplane layer, because then I could confidently implement my own health probing and fully reprogram the entire dataplane as needed. Unfortunately, by design I'm trying to cooperate with Kubernetes... And so we end up with this problem :/

[ghaering]

Ok, I rerun the test. Here are the results.

#!/usr/bin/python -u
import requests
import time

def test():
    try:
        response = requests.get("http://10.115.195.206", timeout=0.5)
    except:
        return False
    return True

if __name__ == "__main__":
    while 1:
        result = test()
        print time.time(), result
        time.sleep(1)

I started the script, prepared fetching the logs, then shut down the node where
the load balancer IP pointed to.

Results of the HTTP test script

1534092121.95 True
...
1534092293.57 True
1534092294.68 True
1534092295.8 True
1534092296.92 True
1534092297.97 False
1534092299.48 False
...
1534092603.64 False
1534092605.14 False
1534092606.65 False
1534092607.76 True
1534092608.87 True
1534092610.08 True

metallb-test python -c "print 1534092607.76 - 1534092297.97"
309.789999962

ca. 300 seconds => 5 minutes
`

I also have the speaker logs, but the timestamps are off compared to the test script (UTC vs. local time).

The node was unhealthy in the expected time, but failover took 5 minutes.

[danderson]

Thanks for the repro. That's definitely way too long.

I've looked at the code that does leader selection, and it seems to be correct: as soon as an endpoint transitions into notReadyAddresses, it gets removed from the eligible set and failover should take place.

I'm going to have to reproduce this locally on my cluster and dig into Kubernetes's behavior some more. Something is definitely wrong somewhere.