New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AWS, ELB: Allow LoadBalancer for service with sessionAffinity: ClientIP #13892

Closed
mcbain opened this Issue Sep 12, 2015 · 33 comments

Comments

@mcbain
Copy link

mcbain commented Sep 12, 2015

failed to create external load balancer for service pro/xyz: unsupported load balancer affinity: ClientIP

As in understood from http://kubernetes.io/v1.0/docs/user-guide/services.html the service setting 'sessionAffinity' is for the kube-proxy on each minion and not for the external load-balancer.
If both supporting the same sessionAffinity, one may also configure it on the external load-balancer, but it's just an optimization to already route the traffic to the target minion.

@justinsb

This comment has been minimized.

Copy link
Member

justinsb commented Sep 12, 2015

Hi - we use ELB on AWS for the load-balancer, in TCP mode. My understanding is the ELB only supports sticky sessions when used in HTTP mode. Even then it uses a cookie, not the clientIP.

What are you trying to do here? It's not trivial, but there are options... we could add support for cookie-based ELB HTTP load balancers, we could change kube-proxy to do ClientIP based load-balancing. There's also some work going on for Layer 7 HTTP load balancing, so it might be that it gets incorporated into that. So any more details on what you'd like to achieve would be helpful!

@mcbain

This comment has been minimized.

Copy link

mcbain commented Sep 12, 2015

Maybe I misunderstood the kube proxy load balancer. My idea of setting the sessionAfinity to clientIP was that all traffic to the service would be managed by kube proxy and routed to the correct target pod e.g. by client ip, even when the pod is on an other minion. The ELB would then just need to route the external traffic to one of the minions. Maybe this is completely wrong?!

@thockin

This comment has been minimized.

Copy link
Member

thockin commented Sep 15, 2015

When coming through an ELB traffic frist has to arrive at a node (chosen by
ELB) and then get directed to a pod (chosen by kube-proxy). Because
kube-proxies are not connected, and ELB doesn't do sticky, you might end up
at two different nodes, which make independent decisions about which
backend to give you. :(

On Sat, Sep 12, 2015 at 10:56 AM, mcbain notifications@github.com wrote:

Maybe I misunderstood the kube proxy load balancer. My idea of setting the
sessionAfinity to clientIP was that all traffic to the service would be
managed by kube proxy and routed to the correct target pod e.g. by client
ip, even when the pod is on an other minion. The ELB would then just need
to route the external traffic to one of the minions. Maybe this is
completely wrong?!


Reply to this email directly or view it on GitHub
#13892 (comment)
.

@justinsb

This comment has been minimized.

Copy link
Member

justinsb commented Sep 16, 2015

@thockin Do you think consistent hashing based on client IP would work though (not 100% sticky, but close)? Although to capture the client IP we would also need to turn on ELB's PROXY protocol, so it definitely would be a lot of work.

@mcbain is this primary for sticky HTTP sessions? If so I think it would be much simpler to implement that with cookies in a Layer 7 load balancer (nginx/haproxy) running inside k8s, without requiring the (complicated) changes above. Is HTTP the primary use case?

@thockin

This comment has been minimized.

Copy link
Member

thockin commented Sep 16, 2015

@justinsb that sounds closer to correct than just ignoring the field, but
if it is a ton of work, it's probably not urgent.

On Tue, Sep 15, 2015 at 7:45 PM, Justin Santa Barbara <
notifications@github.com> wrote:

@thockin https://github.com/thockin Do you think consistent hashing
based on client IP would work though (not 100% sticky, but close)? Although
to capture the client IP we would also need to turn on ELB's PROXY
protocol, so it definitely would be a lot of work.

@mcbain https://github.com/mcbain is this primary for sticky HTTP
sessions? If so I think it would be much simpler to implement that with
cookies in a Layer 7 load balancer (nginx/haproxy) running inside k8s,
without requiring the (complicated) changes above. Is HTTP the primary use
case?


Reply to this email directly or view it on GitHub
#13892 (comment)
.

@mcbain

This comment has been minimized.

Copy link

mcbain commented Sep 16, 2015

@justinsb we want to serve our meteor apps with websockets support :)

@justinsb

This comment has been minimized.

Copy link
Member

justinsb commented Sep 16, 2015

Well that sounds familiar :-)

I think the best bet here is going to be to use an HTTP L7 load balancer (nginx or haproxy) running inside k8s, and use the sticky session support that it offers (probably cookie based). k8s does have a layer 7 load balancer integration that I think/hope will be merged in 1.1; I don't know if it will support sticky sessions or websockets but we should try to make sure it does.

The big problem with something based on client-ip hashing is that ELB masks the client IP, so we'd have to enable the PROXY protocol, and then I don't think that would work with an iptables-backed kube-proxy (which is I believe where we're going).

@rroopreddy

This comment has been minimized.

Copy link

rroopreddy commented Oct 9, 2015

Same issue with MQTT traffic through load balancer to message broker in the pods. Similar to Websockets, TCP connections have to be sticky between load balancer <-> Services <-> Pods

@markoshust

This comment has been minimized.

Copy link

markoshust commented Mar 1, 2016

Is my issue (http://stackoverflow.com/questions/35322432/running-meteor-app-with-nginx-ssl-proxy-on-kubernetes) caused because the ClientIP session affinity isn't being properly set in k8s? I thought I can use the network-layer load balancing vs doing it within nginx config.

@justinsb

This comment has been minimized.

Copy link
Member

justinsb commented Mar 4, 2016

Maybe ... but I think a 10 second delay sounds more like mongodb polling? How about if we collaborate on putting an example into the examples/ dir: If we can get that working, you'll know the problem isn't k8s! Hopefully we'll find the same problems you're hitting. Would you mind opening a new issue and tagging me?

@markoshust

This comment has been minimized.

Copy link

markoshust commented Mar 4, 2016

@justinsb thank you, it sounded like polling to me, however sometimes when it connects, there is no 10 sec lag -- it's instant. i'll put together a quick example using the images i'm using, will try to do so by next week. cheers!

@guoshimin

This comment has been minimized.

Copy link
Contributor

guoshimin commented Mar 22, 2016

@justinsb What is the L7 load balancer you were talking about? Is it Ingress?

@paralin

This comment has been minimized.

Copy link
Contributor

paralin commented Apr 8, 2016

+1 to enabling PROXY protocol on the ELBs.

@bjoernwenzel-tommapps

This comment has been minimized.

Copy link

bjoernwenzel-tommapps commented Apr 29, 2016

+1

@ApsOps

This comment has been minimized.

Copy link
Contributor

ApsOps commented May 19, 2016

I'm trying to run a socket.io application in k8s and facing this issue as well. I set sessionAffinity: ClientIP but it seems it won't work since ELB doesn't send client IPs by default. I tried enabling proxy protocol in ELB but then it just stops serving the requests.

I think it would be nice to have support for proxy protocol since this would be the case in most (all?) the websocket applications.

@justinsb

This comment has been minimized.

Copy link
Member

justinsb commented Jun 4, 2016

Support for PROXY protocol just landed. With that, you should be able to run an ingress controller (e.g. nginx) that can see client IPs and add them as X-Forwarded-For (or similar), and do sticky sessions based on IP or cookies.

I think technically we haven't delivered the request feature (clientIP with Type=LoadBalancer), but we have implemented an alternative way of doing it. Assigning to a not-immediate-future milestone.

@justinsb justinsb added this to the next-candidate milestone Jun 4, 2016

@devonbarrett

This comment has been minimized.

Copy link

devonbarrett commented Jul 19, 2016

Similar use case with Meteor+websockets, would be super useful to have this.

How about if we collaborate on putting an example into the examples/ dir

@justinsb can't see anything in the examples directory, did anything happen with that?

@njuicsgz

This comment has been minimized.

Copy link
Contributor

njuicsgz commented Nov 21, 2016

+1 to enabling PROXY protocol on the ELBs.

@MichaelJCole

This comment has been minimized.

Copy link

MichaelJCole commented Nov 22, 2016

@justinsb

Support for PROXY protocol just landed. With that, you should be able to run an ingress controller (e.g. nginx) that can see client IPs and add them as X-Forwarded-For (or similar), and do sticky sessions based on IP or cookies.

Do you have an example for sticky sessions? I got the impression it wasn't implemented in Ingress. I'm also trying to get a Meteor service up on k8s.

@abourget

This comment has been minimized.

Copy link

abourget commented Jan 16, 2017

Also looking for an example use of this, is proxy protocol enabled by default now?

@MichaelJCole

This comment has been minimized.

Copy link

MichaelJCole commented Jan 17, 2017

FWIW, I used the nginx ingress on AWS and GCE. I don't understand the platforms deeply, but this got me what I wanted: https://github.com/kubernetes/contrib/tree/master/ingress/controllers/nginx

@andyp1per

This comment has been minimized.

Copy link

andyp1per commented May 25, 2017

The application load balancer seems like the way to go now?
https://aws.amazon.com/blogs/aws/new-aws-application-load-balancer/

@renewooller

This comment has been minimized.

Copy link

renewooller commented Jun 14, 2017

To those using the ingress controller instead of AWS ELB - are you placing all your nodes in a public subnet, and then adding all those nodes to a cname or something? I'm having trouble seeing how we'd achieve this from a private subnet.

update: I can see that enabling PROXY protocol will provide the source information http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/enable-proxy-protocol.html which would give you the means to make it work.

@fejta-bot

This comment has been minimized.

Copy link

fejta-bot commented Dec 27, 2017

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

@fejta-bot

This comment has been minimized.

Copy link

fejta-bot commented Jan 26, 2018

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

@fejta-bot

This comment has been minimized.

Copy link

fejta-bot commented Feb 25, 2018

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@wrstlrlp13

This comment has been minimized.

Copy link

wrstlrlp13 commented Apr 16, 2018

/reopen

Is there anything indicating that support for sessionAffinity: clientIP is coming to load balancers for aws anytime soon?

Or is the nginx approach just what I should be going with?

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Apr 16, 2018

@wrstlrlp13: you can't re-open an issue/PR unless you authored it or you are assigned to it.

In response to this:

/reopen

Is there anything indicating that support for sessionAffinity: clientIP is coming to load balancers for aws anytime soon?

Or is the nginx approach just what I should be going with?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@MichaelInfoworks

This comment has been minimized.

Copy link

MichaelInfoworks commented Apr 16, 2018

LOL, google has time to automate crappy bots, but not fix bugs. You got played @wrstlrlp13. Do it again robot!

/reopen

Sucka!

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Apr 16, 2018

@MichaelInfoworks: you can't re-open an issue/PR unless you authored it or you are assigned to it.

In response to this:

LOL, google has time to automate crappy bots, but not fix bugs. You got played @wrstlrlp13. Do it again robot!

/reopen

Sucka!

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@stonefury

This comment has been minimized.

Copy link

stonefury commented May 10, 2018

So am I right that ELB with a service ClientIP sessionaffinity was never supported then? Is the only solution to put a ingress controller in between and use the proxy protocol?

Thanks

@chirangaalwis

This comment has been minimized.

Copy link

chirangaalwis commented Jun 2, 2018

I am facing the same issue. I moved into using service type LoadBalancer recently due to kubernetes/ingress-nginx#2522.

@justinsb @thockin
as @stonefury has questioned, will ELB with a service ClientIP session affinity ever be supported? Is using Ingress the only solution?

@rileytj

This comment has been minimized.

Copy link

rileytj commented Dec 19, 2018

Why not support the X-FORWARDED-FOR HTTP Header?
https://en.wikipedia.org/wiki/X-Forwarded-For

sessionAffinity: ClientIP
sessionAffinityConfig:
    clientIP:
        xForwardedFor: true  <propose>
        timeout: 10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment