New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ingress and TCP #23291

Open
bprashanth opened this Issue Mar 21, 2016 · 43 comments

Comments

@bprashanth
Member

bprashanth commented Mar 21, 2016

This is more of an open question, should we teach the Ingress about TCP?

If we fold TCP into Ingress:

  • Everything in the Ingress is about entry into the cluster
  • Everything in the Service is intra cluster (ips, algorithms, affinity etc)
  • Type=NodePort will continue to be an escape hatch for Services

This would enable us to leverage the Ingress for things like SSL routing (#19333), and leverage Service.Type=Loadbalancer for internal lb (whatever form that takes).

How to express type=loadbalancer?

Create an Ingress like:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: no-rules-map
spec:
  backend:
    serviceName: backendSvc
    servicePort: 80

kubectl expose would do this (discussion on #17429 needs to converge).

How to express SNI?

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: rules-map
spec:
  # SSL routing
  rules:
  - host: www.foo.com
    tcp:
      backend:
        serviceName: svc1
        servicePort: 80
  - host: foo.bar.com
    tcp:
      backend:
        serviceName: svc2
        servicePort: 80

If the above spec is defined with a TLS section, the Ingress controller will terminate the connection using SNI to figure out which certs to serve. Discussion on: #19333

Do we need to express TCP frontend ports?

The challenge here is defining a frontend port that we don't really need for HTTP (:80 and :443 are implicit). One way is:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: rules-map
spec:
  rules:
  - host: www.foo.com
    tcp:
      port: 3321
      backend:
        serviceName: svc1
        servicePort: 80

Another option is to disallow port, the Ingress controller just picks one and writes it to Status (on GCE a port is not needed). All that matters is that TCP traffic reaches the backend. For legacy apps that need a port, they create multiple ingresses with a ingress.port annotation that the Ingress controller tries to honor.

@kubernetes/goog-cluster @kubernetes/rh-networking @smarterclayton @aledbf @thockin @therc

@smarterclayton

This comment has been minimized.

Contributor

smarterclayton commented Mar 22, 2016

@kokhang

This comment has been minimized.

Member

kokhang commented Apr 18, 2016

Hi bprashanth, I am looking at ingress/ingressContoller to provide access to multiple non-http(s) apps from the outside. How would you provide access to many apps which expose the same port? For example, if i want to provide access to two different mysql deployments, in which both expose 3306. How can i do this without having to use random ports for each? Are you thinking of supplying a vip for every service and have that vip map to the backend app? Thanks

@aledbf aledbf referenced this issue Apr 19, 2016

Closed

Haproxy ingress #223

@sputnik13

This comment has been minimized.

sputnik13 commented May 3, 2016

we would really like to see TCP support in Ingress, is there anything we can do to help move this forward beyond an idea/question?

@bprashanth

This comment has been minimized.

Member

bprashanth commented May 3, 2016

@sputnik13 a big step forward would be teaching the existing ingress controllers about Type=LoadBalancer behavior (and flag controlling it). It really doesn't make sense to have 2 things that do ingress tcp, but we already have type=lb. When Ingress does TCP the idea is that we'd deprecate Type=Loadbalancer (or repurpose it to mean internal lb).

So the code in service_controller needs to be moved to an Ingress controller (or the other way around, though I'd rather have just one thing that understands ingress) - if someone creates a tcp ingress or a service of type=loadbalancer, they should get the same thing. I think most people are Ok with tcp ingress as long as we don't have 2 idioms that we support forever that mean the exact same thing.

@sputnik13

This comment has been minimized.

sputnik13 commented May 4, 2016

@bprashanth given existing ingress controllers are contribs, is it necessary that they all come to a common implementation of TCP or are you saying that someone needs to create an ingress controller that supports TCP, then the community can reason about whether to accept it?

I assumed that the path to acceptance would be more along the lines of getting agreement on what the Ingress resource spec would look like (which I think you already did), then creating code for the API to accept those changes to the spec, then flowing that down to ingress controllers...

I'm still quite new to the community so just looking for pointers on how best to help this effort.

@bprashanth

This comment has been minimized.

Member

bprashanth commented May 4, 2016

So I create a TCP ingress on aws/gce etc, how will that work?

  1. It won't work - not acceptable. Then we don't have a cross platform resource. You can just use config map like the nginx controller currently does.
  2. It won't work till we've ported the code in service_controller that currently fulfills service.Type=Loadbalancer to respect both service.Type=Loadbalancer and TCP Ingress, at which point we deprecated type=loadbalancer
  3. We port the code now, then (or in parallel) create a TCP Ingress resouce in the api, so it works from the start

3 > 2
The danger of adding a resource now without doing 2 is that you end up with a messy solution. We can't have 2 ways to do TCP lb (without a clear deprecation plan), and we can't add a resource that has only 1 backend.

@sputnik13

This comment has been minimized.

sputnik13 commented May 11, 2016

@bprashanth I'm not suggesting that the feature should go out with incomplete implementations, what I'm asking for is some sort of agreement on what the Ingress resource looks like that supports TCP so that we can start implementing it.

FYI, the NGINX ingress controller already does support TCP and UDP ingress via ConfigMap.

I've started a proposal mostly based on what you posted here (https://github.com/sputnik13/kubernetes/blob/l4-ingress-proposal/docs/proposals/l4-ingress.md). Not sure whether I'm following the correct processes. If I'm not, please let me know what I can do better.

@bprashanth

This comment has been minimized.

Member

bprashanth commented May 11, 2016

@sputnik13 That sounds good, please open a pr. Ther's no format per-se, just pattern match existing proposals: https://github.com/kubernetes/kubernetes/tree/master/docs/proposals

Fyi since this is not going to make 1.3 comments will trickle through till the release, but if we have everyone on the same page at the start of 1.4 we should be able to nail it for next-release.

@pires

This comment has been minimized.

Member

pires commented May 19, 2016

This is more of an open question, should we teach the Ingress about TCP?

Yes! And @sputnik13 proposal nails it.

@bprashanth

This comment has been minimized.

Member

bprashanth commented Dec 1, 2016

@jayunit100

This comment has been minimized.

Member

jayunit100 commented Mar 28, 2017

Should we close this issue?... since the original purpose was to debate the question. It looks like

  • community agrees on tcp/l4 and ,
  • L4 proposal was successfully merged
@kargakis

This comment has been minimized.

Member

kargakis commented Mar 28, 2017

@jayunit100 can you post a link to the proposal that was merged?

@jayunit100

This comment has been minimized.

Member

jayunit100 commented Mar 28, 2017

#25821 above ......?

@kargakis

This comment has been minimized.

Member

kargakis commented Mar 28, 2017

Doesn't look like it got merged.

@jayunit100

This comment has been minimized.

Member

jayunit100 commented Mar 28, 2017

Ahhh sorry ; closed, not merged: In any case, Are we all in agreement that the proposal en route will just flesh out details but the community wants ingress to support L4? If so we still should close.

@jayunit100

This comment has been minimized.

Member

jayunit100 commented Mar 28, 2017

I.e. My motive here is to establish that we do (or don't) agree on L4 support - and we can debate the implementation in a follow on issue.

@djsly

This comment has been minimized.

Contributor

djsly commented Jun 1, 2017

@nicksardo

  1. The Load Balancer doesn't support SSL termination and we would like to combine our ssl management in the same controller.
  2. We have multiple namespaces in the same cluster exposing the same services, some Cloud provider have limits on the number of PublicIP that can be added by Load Balancer hence using a single point of entry (ingress controller' service) simplify things up a lot.
@thockin

This comment has been minimized.

Member

thockin commented Jun 1, 2017

@pires

This comment has been minimized.

Member

pires commented Jun 1, 2017

Why would there be the need to give up? For instance, ELB supports PROXY protocol as an opt-in. Doesn't GCLB L4 do something like that as well? I know apps need to accommodate that but it's a trade-off.

I wonder if a Service, which last time I checked in GKE generates a regional GCLB L4, is any different.

@thockin

This comment has been minimized.

Member

thockin commented Jun 1, 2017

@pires

This comment has been minimized.

Member

pires commented Jun 1, 2017

I mean, it's the same trade-off as in L7 with the HTTP headers being put in place. But you have a point this will need to be made clear to users.

As a note, just confirmed GCLB L4 does support PROXY v1.
screen shot 2017-06-01 at 08 09 55

@thockin

This comment has been minimized.

Member

thockin commented Jun 1, 2017

@joemiller

This comment has been minimized.

joemiller commented Jun 2, 2017

@thockin Yes, we are willing to give up source-ip. We're looking at this mainly for mTLS connections and in those cases we care much more about authenticating the client cert than the source-IP.

@thockin

This comment has been minimized.

Member

thockin commented Jun 4, 2017

@hiscal2015

This comment has been minimized.

hiscal2015 commented Aug 3, 2017

+1 It is wonderful if ingress supports TCP and SNI

@accnops

This comment has been minimized.

accnops commented Dec 14, 2017

Hi! Is this still being investigated?

I could certainly use this in my projects...

@pires

This comment has been minimized.

Member

pires commented Dec 14, 2017

There was a proposal PR but it got closed #25821

@slaskawi

This comment has been minimized.

Contributor

slaskawi commented Feb 2, 2018

I would also love to see an Ingress that supports custom TCP-based protocols. Currently in OpenShift we got it working by using TLS/SNI. However you get a performance hit that was and in some cases (like Data Grids without sensitive data) it is not desirable. Another options to look at are HTTP/1.1 Upgrade and TLS/ALPN for switching an HTTP connection to a custom protocol. We'll see how that goes in the future.

Of course I do agree with @thockin that we have a Load-Balancer Services for this but @smarterclayton also has a point that an Ingress indicates that there will be some traffic from the outside world whereas Services seem to be more internal to me.

So again, I know you can achieve this kind of behavior even today but I would love to see a TCP Ingress someday in Kube.

@bgrant0607

This comment has been minimized.

Member

bgrant0607 commented Feb 2, 2018

@ggprod

This comment has been minimized.

ggprod commented Feb 21, 2018

So in the time-being (intil the L4 Ingress feature makes it into a release). What is the best way to provide an SSL Proxy Load Balancer functionality(for non-HTTP TCP connections) for pods deployed on GKE?

@tookko

This comment has been minimized.

tookko commented Mar 23, 2018

It would be great if there was an elegant way to integrate ingress also for other non-HTTP TCP protocols. For example, an IIOP proxy would be needed for CORBA based traffic, a simple L4 proxy cannot handle it.

@git001

This comment has been minimized.

git001 commented May 30, 2018

I have described several ingress controller solutions for kubernetes in my blog post https://www.me2digital.com/blog/2018/05/session-sticky/#kubernetes-ingress all of them can handle http/https/tcp and some of them also udp.

From my point of view are there several available solutions for a generic ingress handler. Doesn't the current solutions solve the question?

@pires

This comment has been minimized.

Member

pires commented Jun 7, 2018

@git001 Ingress controllers have to resort to their own solutions to provide such functionality, e.g. HAProxy Ingress controller uses ConfigMap. The desire discussed in this thread is to evolve the Ingress object, or create a new object, for L4 load-balancing.

@MichaelSp

This comment has been minimized.

MichaelSp commented Jun 7, 2018

Here is a working solution for SNI: #19333 (comment)

@pires

This comment has been minimized.

Member

pires commented Jun 7, 2018

This is still not L4 Ingress!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment