New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Explore kube-router as ingress controller #30

Closed
bzub opened this Issue Jun 21, 2017 · 20 comments

Comments

@bzub
Collaborator

bzub commented Jun 21, 2017

From the TODO:

explore integration of an ingress controller so Kube-router will be one complete solution for both east-west and north-south traffic

@bzub

This comment has been minimized.

Show comment
Hide comment
@bzub

bzub Jun 21, 2017

Collaborator

I wonder what unique features a new ingress controller could bring to Kubernetes? I suppose since kube-router uses IPVS loadbalancing without configuration at the service proxy level that it could serve as a very simple to configure ingress controller. Others can be difficult to configure (nginx, haproxy, etc) since they are not native to kubernetes.

To pursue this I think we would need to pull in and use an HTTP library for host-based routing. Caddy and gorilla libraries come to mind, if not the Go standard library http pkg.

Also, this presentation seems relevant: https://blog.codeship.com/kernel-load-balancing-for-docker-containers-using-ipvs/

Collaborator

bzub commented Jun 21, 2017

I wonder what unique features a new ingress controller could bring to Kubernetes? I suppose since kube-router uses IPVS loadbalancing without configuration at the service proxy level that it could serve as a very simple to configure ingress controller. Others can be difficult to configure (nginx, haproxy, etc) since they are not native to kubernetes.

To pursue this I think we would need to pull in and use an HTTP library for host-based routing. Caddy and gorilla libraries come to mind, if not the Go standard library http pkg.

Also, this presentation seems relevant: https://blog.codeship.com/kernel-load-balancing-for-docker-containers-using-ipvs/

@bzub bzub self-assigned this Jun 22, 2017

@bzub bzub added the enhancement label Jun 22, 2017

@murali-reddy

This comment has been minimized.

Show comment
Hide comment
@murali-reddy

murali-reddy Jun 22, 2017

Member

"I wonder what unique features a new ingress controller could bring to Kubernetes?"

I sort of agree to your concern. There are number of exisitng dedicated ingress controllers available, functionally we may not add anything extra.

But its one more moving peice user has to understand how to deploy and operate. I kind of feel each of current Kubernetes networking solutions solve a pointed problem. Users have to deploy different solutions for pod networking, east-west traffic service load balancing/proxy, network policies and for north-south traffic have to deploy ingress controller. Its just over complicated i feel.

Reason i added in TODO, was if kube-router can just do essential L7 routing/ssl termination we might just have one cohesive solution dealing with north-south, east-west traffic requirements.

Please weigh in pros and cons to see if its worth the effort.

Member

murali-reddy commented Jun 22, 2017

"I wonder what unique features a new ingress controller could bring to Kubernetes?"

I sort of agree to your concern. There are number of exisitng dedicated ingress controllers available, functionally we may not add anything extra.

But its one more moving peice user has to understand how to deploy and operate. I kind of feel each of current Kubernetes networking solutions solve a pointed problem. Users have to deploy different solutions for pod networking, east-west traffic service load balancing/proxy, network policies and for north-south traffic have to deploy ingress controller. Its just over complicated i feel.

Reason i added in TODO, was if kube-router can just do essential L7 routing/ssl termination we might just have one cohesive solution dealing with north-south, east-west traffic requirements.

Please weigh in pros and cons to see if its worth the effort.

@bzub

This comment has been minimized.

Show comment
Hide comment
@bzub

bzub Jun 24, 2017

Collaborator

I think you're right. Many of the basic features needed to implement an Ingress Controller are already in kube-router, and it would be great if a user started out with a cluster that supported it with this one core component.

Collaborator

bzub commented Jun 24, 2017

I think you're right. Many of the basic features needed to implement an Ingress Controller are already in kube-router, and it would be great if a user started out with a cluster that supported it with this one core component.

@bzub bzub added this to Someday in Distributed Load-Balancer Jun 24, 2017

@murali-reddy

This comment has been minimized.

Show comment
Hide comment
@murali-reddy

murali-reddy Jun 24, 2017

Member

sorry if it was not clear from my earlier comment. Ingress controller is pretty involved piece of work as it requires L7 routting, SSL termination etc. As you know IPVS is just L4 load balancer it's not meant for that, so its not suited for Ingress controller. We have ingress controllers built on HaProxy, Nginx, Envoy Træfik, etc and there are services mesh like linkerd, istio etc which can ingress as well. I was suggesting "weigh in pros and cons to see if its worth the effort." so that it wont be just another ingress controller, but solve some problem (operational simplicity as all-in-one solution, was one reason i was suggesting to have ingress controller in Kube-router).

Member

murali-reddy commented Jun 24, 2017

sorry if it was not clear from my earlier comment. Ingress controller is pretty involved piece of work as it requires L7 routting, SSL termination etc. As you know IPVS is just L4 load balancer it's not meant for that, so its not suited for Ingress controller. We have ingress controllers built on HaProxy, Nginx, Envoy Træfik, etc and there are services mesh like linkerd, istio etc which can ingress as well. I was suggesting "weigh in pros and cons to see if its worth the effort." so that it wont be just another ingress controller, but solve some problem (operational simplicity as all-in-one solution, was one reason i was suggesting to have ingress controller in Kube-router).

@bzub

This comment has been minimized.

Show comment
Hide comment
@bzub

bzub Jun 24, 2017

Collaborator

True, the L7 aspect is not as simple as I made it seem. I do think with the help of a mature HTTP library and integrating kube-lego that the implementation will be less painful.

Still I think it's worth pursuing, I will give it a shot. Getting an Ingress Controller out of the box would be a huge feature and possibly on-par or superior to other controllers in terms of performance. It would also go hand-in-hand with a global load-balancer as depicted in #10.

Collaborator

bzub commented Jun 24, 2017

True, the L7 aspect is not as simple as I made it seem. I do think with the help of a mature HTTP library and integrating kube-lego that the implementation will be less painful.

Still I think it's worth pursuing, I will give it a shot. Getting an Ingress Controller out of the box would be a huge feature and possibly on-par or superior to other controllers in terms of performance. It would also go hand-in-hand with a global load-balancer as depicted in #10.

@murali-reddy

This comment has been minimized.

Show comment
Hide comment
@murali-reddy

murali-reddy Jun 24, 2017

Member

Yes, indeed its worth pursuing.

Member

murali-reddy commented Jun 24, 2017

Yes, indeed its worth pursuing.

@bzub bzub added this to the v0.2.0 milestone Jul 10, 2017

@murali-reddy murali-reddy assigned murali-reddy and unassigned bzub Jul 14, 2017

@murali-reddy

This comment has been minimized.

Show comment
Hide comment
@murali-reddy

murali-reddy Jul 14, 2017

Member

I will take stab and work towards a prototype to see how it fits in.

Member

murali-reddy commented Jul 14, 2017

I will take stab and work towards a prototype to see how it fits in.

@hwinkel

This comment has been minimized.

Show comment
Hide comment
@hwinkel

hwinkel Aug 27, 2017

And as we work on L4 with IPVS, an IPVS based ingress could act as UDP ingress too. Despite the fact the ingress API model is very HTTP centric and has never thought about other protocols. the Service API is much more rich here.

hwinkel commented Aug 27, 2017

And as we work on L4 with IPVS, an IPVS based ingress could act as UDP ingress too. Despite the fact the ingress API model is very HTTP centric and has never thought about other protocols. the Service API is much more rich here.

@murali-reddy

This comment has been minimized.

Show comment
Hide comment
@murali-reddy

murali-reddy Aug 28, 2017

Member

@hwinkel thanks for your comment. Agree current Ingress definition is very L7 centric and though there is a discussion to support L4 I suspect it will happen any time soon. We are looking for inputs on what is the best way forward w.r.t ingress in Kube-router that will be beneficial to Kuberentes users.

Member

murali-reddy commented Aug 28, 2017

@hwinkel thanks for your comment. Agree current Ingress definition is very L7 centric and though there is a discussion to support L4 I suspect it will happen any time soon. We are looking for inputs on what is the best way forward w.r.t ingress in Kube-router that will be beneficial to Kuberentes users.

@hwinkel

This comment has been minimized.

Show comment
Hide comment
@hwinkel

hwinkel Sep 6, 2017

Would be Good, in the meantime may annotation to ingress resource can help. As we are providing network services, namely Udp based or l3 based packet services, we would like to use somerhing like ingress for l3 or l4 balancing or traffic steering to pods. The steering sometimes needs a bit more clever then pure routing or round Robin, like based on source IP or given tunnel id's in the Udp protocols. That's why we think about a l3/l4 or Udp l7 ingress

hwinkel commented Sep 6, 2017

Would be Good, in the meantime may annotation to ingress resource can help. As we are providing network services, namely Udp based or l3 based packet services, we would like to use somerhing like ingress for l3 or l4 balancing or traffic steering to pods. The steering sometimes needs a bit more clever then pure routing or round Robin, like based on source IP or given tunnel id's in the Udp protocols. That's why we think about a l3/l4 or Udp l7 ingress

@murali-reddy

This comment has been minimized.

Show comment
Hide comment
@murali-reddy

murali-reddy Sep 7, 2017

Member

@hwinkel as of now, Kubernetes Ingress resource would not fit in to L4/L3 (for e.g lack of port details etc). I was thinking to add custom resource with CRD's that can provide custom ingress resource and kube-router can implement an ingress controller based on IPVS. I will give a thought and come up with a proposal.

@bzub @Thoro if you have any thoughts on L4 ingress please share.

Member

murali-reddy commented Sep 7, 2017

@hwinkel as of now, Kubernetes Ingress resource would not fit in to L4/L3 (for e.g lack of port details etc). I was thinking to add custom resource with CRD's that can provide custom ingress resource and kube-router can implement an ingress controller based on IPVS. I will give a thought and come up with a proposal.

@bzub @Thoro if you have any thoughts on L4 ingress please share.

@murali-reddy

This comment has been minimized.

Show comment
Hide comment
@hwinkel

This comment has been minimized.

Show comment
Hide comment
@hwinkel

hwinkel Sep 7, 2017

I'm Awre of the kubernetes discussions, and made the point that http is not enough. Further I assume that crds will exists if you have different ingress requirements. For me is unclear when you will leave the scope of ingress and you enter a specific 'loadbalacer' proxy or frontend application distributes the requests across backend services. But I think there is headroom to extend ingress. There is a good example about a more powerful ingress controller with a lot of options, Ill add here. (currently typing on a mobile)

hwinkel commented Sep 7, 2017

I'm Awre of the kubernetes discussions, and made the point that http is not enough. Further I assume that crds will exists if you have different ingress requirements. For me is unclear when you will leave the scope of ingress and you enter a specific 'loadbalacer' proxy or frontend application distributes the requests across backend services. But I think there is headroom to extend ingress. There is a good example about a more powerful ingress controller with a lot of options, Ill add here. (currently typing on a mobile)

@murali-reddy

This comment has been minimized.

Show comment
Hide comment
@murali-reddy

murali-reddy Sep 7, 2017

Member

sure @hwinkel plese share your thoughts

Member

murali-reddy commented Sep 7, 2017

sure @hwinkel plese share your thoughts

@hwinkel

This comment has been minimized.

Show comment
Hide comment
@hwinkel

hwinkel Sep 11, 2017

hi, back on normal keyboard. as mentioned above, what i'm thinking is to extend ingress capabilities for a UDP based ingress with annotations, the same way other more sophisticated ingress controllers like voyager are doing.

https://github.com/appscode/voyager

Hopefully the stock ingress resource will learn soon how to deal with some not so HTTP oriented scenarios like L7 UDP Protocols, L4 UDP.

Further I'm interested to discuss how even a L3 (or L2?) model could be mapped to the idea of ingress, this brings me to the idea of SDN for traffic steering, nothing more (from a resource and API perspective) is a ingress is doing.

hwinkel commented Sep 11, 2017

hi, back on normal keyboard. as mentioned above, what i'm thinking is to extend ingress capabilities for a UDP based ingress with annotations, the same way other more sophisticated ingress controllers like voyager are doing.

https://github.com/appscode/voyager

Hopefully the stock ingress resource will learn soon how to deal with some not so HTTP oriented scenarios like L7 UDP Protocols, L4 UDP.

Further I'm interested to discuss how even a L3 (or L2?) model could be mapped to the idea of ingress, this brings me to the idea of SDN for traffic steering, nothing more (from a resource and API perspective) is a ingress is doing.

@murali-reddy

This comment has been minimized.

Show comment
Hide comment
@murali-reddy

murali-reddy Sep 14, 2017

Member

I am summarizing discussion we had over gitter and some rough thoughts. Taking a holistic look at the ingress and some of the practices done by web-scale companies as described in #88 there are three parts or related issues for ingress. some of the issues are specific to on-premise clusters

  • Currently Kube-router advertises ClusterIP (and perhaps should advertise service external IP as well) or VIP of the service to peered upstream routers. Routers can do L3 ECMP based load balancing of the traffic to the cluster nodes. This serves two purpose, HA and scale-out load balancing. When a node goes down, routers can stop sending the service traffic to the node. Traffic gets load balanced to the nodes. For HA, there can be other options like VRRP.

  • Connection persistence: Since packets of a connection does not necessarily hit same node, packets belonging to the same connection should always be directed to the same service endpoint. Which require a consistent hashing so each node pick same endpoint for connection.

  • Direct server return: service endpoint can send the return traffic directly to the routers by-passing ingress node.

  • from managment plane standpoint we need a way through ingress API, annotations or CRD to express rich set of ingress traffics and requirements.

Member

murali-reddy commented Sep 14, 2017

I am summarizing discussion we had over gitter and some rough thoughts. Taking a holistic look at the ingress and some of the practices done by web-scale companies as described in #88 there are three parts or related issues for ingress. some of the issues are specific to on-premise clusters

  • Currently Kube-router advertises ClusterIP (and perhaps should advertise service external IP as well) or VIP of the service to peered upstream routers. Routers can do L3 ECMP based load balancing of the traffic to the cluster nodes. This serves two purpose, HA and scale-out load balancing. When a node goes down, routers can stop sending the service traffic to the node. Traffic gets load balanced to the nodes. For HA, there can be other options like VRRP.

  • Connection persistence: Since packets of a connection does not necessarily hit same node, packets belonging to the same connection should always be directed to the same service endpoint. Which require a consistent hashing so each node pick same endpoint for connection.

  • Direct server return: service endpoint can send the return traffic directly to the routers by-passing ingress node.

  • from managment plane standpoint we need a way through ingress API, annotations or CRD to express rich set of ingress traffics and requirements.

@SEJeff

This comment has been minimized.

Show comment
Hide comment
@SEJeff

SEJeff Sep 18, 2017

Contributor

For L7 http routing, may I suggest httprouter which is fast due to using a Trie for routing O(1) instead of a list of regular expressions O(n) complexity in lookup time.

Alternatively, look up chi which is what cloudflare / heroku use for their production url routing (it also uses a Trie).

Contributor

SEJeff commented Sep 18, 2017

For L7 http routing, may I suggest httprouter which is fast due to using a Trie for routing O(1) instead of a list of regular expressions O(n) complexity in lookup time.

Alternatively, look up chi which is what cloudflare / heroku use for their production url routing (it also uses a Trie).

@SEJeff

This comment has been minimized.

Show comment
Hide comment
@SEJeff

SEJeff Oct 5, 2017

Contributor

Additionally, one feature critical to any serious ingress controller would be integrated opentracing support. Several of the more commonly used ingress controllers support this. The nginx ingress just added support for it in the most recent beta release

zipkin go client libs
jaeger go client libs

Having an instrumented distributed application really helps application owners troubleshoot things. Having this as part of kube-router natively (in addition to the prometheus stats) would be incredible.

Contributor

SEJeff commented Oct 5, 2017

Additionally, one feature critical to any serious ingress controller would be integrated opentracing support. Several of the more commonly used ingress controllers support this. The nginx ingress just added support for it in the most recent beta release

zipkin go client libs
jaeger go client libs

Having an instrumented distributed application really helps application owners troubleshoot things. Having this as part of kube-router natively (in addition to the prometheus stats) would be incredible.

@murali-reddy

This comment has been minimized.

Show comment
Hide comment
@murali-reddy

murali-reddy Oct 6, 2017

Member

@SEJeff thats a nice feedback. i have opened seperate issue #181

Member

murali-reddy commented Oct 6, 2017

@SEJeff thats a nice feedback. i have opened seperate issue #181

@murali-reddy

This comment has been minimized.

Show comment
Hide comment
@murali-reddy

murali-reddy Dec 25, 2017

Member

Given the momentum and possible hook into istio, it does make sense to integrate Envoy. Closing this issue in favour of #130

Member

murali-reddy commented Dec 25, 2017

Given the momentum and possible hook into istio, it does make sense to integrate Envoy. Closing this issue in favour of #130

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment