Allow multiple (compatible) services to use the same loadBalancerIP

[danderson]

Is this a bug report or a feature request?:

Feature request.

What happened:

@miekg tried to set up his home DNS service on Kubernetes, with MetalLB. This requires a service that routes TCP+UDP 53 to a deployment.

Unfortunately, because of kubernetes/kubernetes#23880 , services are not allowed to have mixed protocols, because of poorly explained reasons involving GCP's implementation of load-balancing.

The workaround on GCP is: create 2 Services, one for each protocol, and make them use the same loadBalancerIP, by preallocating a static IP in GCP.

This doesn't work in MetalLB, because we have a 1:1 mapping of Service to IP. Annoyingly, MetalLB trivially "supports" multiprotocol services, so the correct solution would be to remove that constraint in the vanilla k8s control plane, but...

What you expected to happen:

MetalLB should have a way to force the allocation of the same IP to multiple services, as long as those services use disjoint sets of (proto, port).

How to reproduce it (as minimally and precisely as possible):

Create a service, pointing to whatever, serving ports tcp/53 and udp/53. Observe that k8s rejects the service. Create separate services, one for tcp/53 and one for udp/53. This time, k8s accepts it, but MetalLB allocates separate IPs. Try setting the same loadBalancerIP on both services, notice that MetalLB does not allow this.

[miekg]

Seems instead of using net.IP we should use net.Addr which has a network

[danderson]

net.Addr is really generic though, it could be any kind of address at all (including unix domain socket, raw socket address, ...). So it would need a bunch of extra validation to make sure the addrs make sense.

[miekg]

yep sure, but you can't ask for those from k8s :-)

[danderson]

Oh, I see. Hm, yeah, that would work. The allocator lib will need a lot of changes to make that work, I think?...

We also need to figure out how to enable/disable sharing, because sharing has potentially surprising behaviors (e.g. : IP is shared between 2 services, tcp/100 and tcp/200, but then the first service is edited to also have tcp/200... What do?). So, probably should be off by default?

[miekg]

I was (narrowly) focussing on my use case a service with the same port on 2 diff. protocols. Differrent ports opens another can of worms.

[danderson]

Hmm, yeah, same ports but different protocols has a nice property right now, that k8s does not allow the problematic configurations at all. That will change once the upstream bug is fixed though, so I would still like to think about how to allow the generic solution.

I think a prerequisite for the generic solution is to implement #2. Once we have admission control set up, we can reject unsafe config changes before they are applied.

[miekg]

hmmm, don't we also need to track if the new service identical enough to the old service? Pointing to the same things in the cluster, etc?

Bit more work than anticipated :/

[danderson]

It depends what kube-proxy programs in the dataplane on the nodes. In theory, there's no problem with pointing different service IP ports to different services, as long as kube-proxy doesn't program "default-drop" entries on the nodes.

Easy way to check this: hack up metallb (as ugly as you like) so that it sets status.loadbalancer.ingress[0].ip to the same IP for 2 different services, and then inspect iptables on the nodes to see what got programmed (iptables -L -n -v and iptables -L -n -v -t nat). Ideally, in the nat tables, we should see entries for each port of each service, which DNAT to different targets. The critical part is that there should not be any entries for the service IP, without a port selector.

[fkrauthan]

I could really use the feature that different Ports can use the same IP as my VPS provider for example just assigns one public IP per VPS and does not offer any way to receive additional IPs (based on my research there are quiet a few providers that have similar restrictions for there VPS products).

In terms of conflicts the LoadBalancer creation could just be marked as failed in case of conflicts. Including on an update.

[hclaria]

Hello,

We look forward to host a CoreDNS deployment on our cluster with a LoadBalancer service handled by MetalLB.

This service would need to expose both UDP and TCP protocols on the same external IP.

"b) it doesn't work when I deploy, i.e the -tcp service overwrites the udp one" - #122
@miekg , do you mean kube-proxy's iptables rules are being overwritten by the latest service ? In that case we would need to bump that issue to the sig network or wait for a stable IPVS implementation ?