kube-proxy iptables random routing is inefficient

[bfleming-ciena]

I've learned recently that userspace routing was replaced with iptables routing in k8s, and iptables uses random pod selection for load balancing.

This means most of the time we have pods that are idle. If it were round-robin then we would guarantee to distribute evenly.

Is this random selection the only option we have available right now?

Thanks

/sig network

[islinwb]

@stonefury Maybe you should try ipvs.

[bfleming-ciena]

Oh wow, didn't know that was in the pipeline. It's beta I see, but still, is this to replace iptables mode, or just an alternate mode? Either way, thanks for that @islinwb

[dims]

@stonefury it will be GA in 1.11 - #58442

[islinwb]

@stonefury Currently IPVS is an alternate mode but I guess someday most people will use it. It takes round-robin as the default load balancing algorithm and you can choose others. (See the ipvs docs 1, 2)

[thockin]

A) Random means "equal probability of hitting any backend". If you have idle pods it suggests you either don't have a statistically significant number of connections or you're doing client affinity or something to defeat the randomizer.

B) Round-robin becomes a distributed decision - each node chooses independently of each other node. So to your backend service it's basically random anyway.

Anyway ipvs mode is going GA in 1.11, so please feel free to try it out :)

[bfleming-ciena]

@thockin - Yes, I later learned the the dev team uses persistent connections to the microservice, and so they only do a small number of initial connections, so statistically it was not evenly distributed. I had them increase the front end so it would make more connections. So this ultimately was an implementation issue on the dev side, though having round-robin would've made this a non issue for them.

Thanks all

[Jeffwan]

@thockin

I use socket to connect to a cluster IP which has many backend pods listening on same port. I notice random is totally not guaranteed. I disable sticky session on Service level (by default) but I am wondering if iptable proxy mode has something in-built to reuse address of backend pod.

I send ~1000 request and they all go to same pod.

[zghnr1993]

i'm wondering too . why many request go to same pod .