New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support mixed protocols in service.type=loadbalancer #23880

Open
bprashanth opened this Issue Apr 5, 2016 · 20 comments

Comments

Projects
None yet
@bprashanth
Member

bprashanth commented Apr 5, 2016

It should be possible to use a single IP to direct traffic to multiple protocols with a single Service of Type=Loadbalancer. We just need to:

  1. Promote ephemeral to static ip
  2. Create another forwarding rule with the same static ip, but a different protocol/port

Unfortunately we need this dance because a single forwarding rule only supports 1 protocol.

When we do this we should make sure the firewall rules are opened up for the right protocol: https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/gce/gce.go#L927 (that just takes the first protocol for the firewall rule).

Is there a reason we didn't do this initially?

@vsimon

This comment has been minimized.

Contributor

vsimon commented Jun 6, 2016

It would be very nice to support this.

@samuraisam

This comment has been minimized.

samuraisam commented Jul 15, 2016

Does #24090 support this?

@therc

This comment has been minimized.

Contributor

therc commented Jul 15, 2016

For AWS, you can use a combination of #23495 and #26268

@samuraisam

This comment has been minimized.

samuraisam commented Jul 18, 2016

Can it be accomplished on GCE?

@thomasbarton

This comment has been minimized.

thomasbarton commented Dec 23, 2016

@samuraisam I was able to work around this limitation by creating 2 separate services of type=LoadBalancer one for tcp and one for udp. Using the parameter loadBalancerIP: XXX.XXX.XXX.XXX with the same ip address for both it works. I am posting this in case anyone else runs into the same issue.

@ensonic

This comment has been minimized.

ensonic commented Mar 15, 2017

@thomasbarton Did you run it once, check what IP it got and then hard-coded the IP?

@thomasbarton

This comment has been minimized.

thomasbarton commented Mar 15, 2017

@ensonic On GCE you can do that. But i recommend reserve the ip address first and making it static. Then create both services with the LoadBalanceIP set.

@fejta-bot

This comment has been minimized.

fejta-bot commented Dec 22, 2017

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

@ensonic

This comment has been minimized.

ensonic commented Dec 22, 2017

/remove-lifecycle stale
This would still be nice to have.

@ffledgling

This comment has been minimized.

ffledgling commented Feb 5, 2018

Bump. This would be really nice to have!

@fejta-bot

This comment has been minimized.

fejta-bot commented Jun 14, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@ffledgling

This comment has been minimized.

ffledgling commented Jun 14, 2018

/remove-lifecycle stale

@kiall

This comment has been minimized.

Contributor

kiall commented Jul 25, 2018

As a very very quick hack, I simply removed the validation code that enforces only a single protocol:

diff --git a/pkg/apis/core/validation/validation.go b/pkg/apis/core/validation/validation.go
index 7050c604e5..7747d527fc 100644
--- a/pkg/apis/core/validation/validation.go
+++ b/pkg/apis/core/validation/validation.go
@@ -3714,9 +3714,6 @@ func ValidateService(service *core.Service) field.ErrorList {
                                includeProtocols.Insert(string(service.Spec.Ports[i].Protocol))
                        }
                }
-               if includeProtocols.Len() > 1 {
-                       allErrs = append(allErrs, field.Invalid(portsPath, service.Spec.Ports, "cannot create an external load balancer with mix protocols"))
-               }
        }
 
        if service.Spec.Type == core.ServiceTypeClusterIP {

After this, I rebuilt kube-apiserver, and with MetalLB, mixed protocol load balancers now work just fine:

NAME               TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                       AGE
mixed-protocols    LoadBalancer   172.29.12.226   172.29.32.9   53:31874/TCP,53:31874/UDP     3m

I think the best course forward is to:

  1. Remove this test
  2. In all existing LB implementations, add an equivalent of this validation. Likely emitting an event against the LB, rather than failing the creation of the resource in the first place.

Any objections? I'm happy to implement when I have a little time.

Updated: After a few minutes, I spotted this event:

Type     Reason                Age              From                             Message
----     ------                ----             ----                             -------
Normal   IPAllocated           7m               metallb-controller               Assigned IP "172.29.32.9"
Warning  PortAlreadyAllocated  1m (x3 over 7m)  portallocator-repair-controller  Port 31874 was assigned to multiple services; please recreate service

Everything still works, I suspect the portallocator-repair-controller needs an update to consider port and protocol, rather than just port.

@tuminoid

This comment has been minimized.

tuminoid commented Sep 10, 2018

We're getting this complaint from portallocator-repair-controller on 1.10.4 also on NodePort service that exposes both TCP port and UDP port on the same NodePort.

@fejta-bot

This comment has been minimized.

fejta-bot commented Dec 9, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@voor

This comment has been minimized.

voor commented Dec 9, 2018

/remove-lifecycle stale

@kvaps

This comment has been minimized.

Contributor

kvaps commented Dec 10, 2018

I am also interested in solving this issue.
I think checking an annotation can be good compromise for solve that.

@thockin do you agree on allow-mixed-protocol: true annotation for override this check?

according conversation from #64471

@voor

This comment has been minimized.

voor commented Dec 10, 2018

Just for a quick workaround if you're encountering this with things like MetalLB, you can actually have two Services mapped to the same IP Address. This does require a workaround annotation, metallb.universe.tf/allow-shared-ip: "true"

apiVersion: v1
kind: Service
metadata:
  name: network-dns
  namespace: kube-system
  annotations:
    metallb.universe.tf/allow-shared-ip: "true"
  labels:
    k8s-app: kube-dns
spec:
  type: LoadBalancer
  loadBalancerIP: 192.168.2.241
  ports:
    - name: dns
      port: 53
      protocol: UDP
  selector:
    k8s-app: kube-dns
---
apiVersion: v1
kind: Service
metadata:
  name: network-dns-tcp
  namespace: kube-system
  annotations:
    metallb.universe.tf/allow-shared-ip: "true"
  labels:
    k8s-app: kube-dns
spec:
  type: LoadBalancer
  loadBalancerIP: 192.168.2.241
  ports:
    - name: dns-tcp
      port: 53
      protocol: TCP
  selector:
    k8s-app: kube-dns
@arianvp

This comment has been minimized.

arianvp commented Dec 12, 2018

Just so that it's documented clearly, (I haven't seen it yet in the ticket), I think the desired behaviour would be that the above example could be written like:

apiVersion: v1
kind: Service
metadata:
  name: network-dns-tcp
  namespace: kube-system
  labels:
    k8s-app: kube-dns
spec:
  type: LoadBalancer
  ports:
    - name: dns-tcp
      port: 53
      protocol: TCP
    - name: dns-udp
      port: 53
      protocol: UDP
  selector:
    k8s-app: kube-dns
@voor

This comment has been minimized.

voor commented Dec 12, 2018

Certainly, @arianvp except I believe you're missing the proposed solution, which is to add the following:

apiVersion: v1
kind: Service
metadata:
  name: network-dns-tcp
  namespace: kube-system
+  annotations:
+    kubernetes.io/allow-mixed-protocol: true
  labels:
    k8s-app: kube-dns
spec:
  type: LoadBalancer
  ports:
    - name: dns-tcp
      port: 53
      protocol: TCP
    - name: dns-udp
      port: 53
      protocol: UDP
  selector:
    k8s-app: kube-dns

This would then disable the validation that @kiall indicated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment