AWS Feature: Support for application load balancer

[ProTip]

ELBv2 supports layer 7 load balancing. Surprisingly it's already GA and integrated with ECS. It would be great if kubernetes supported hooking up services to a specific ELB instance with path rules.

[mattjonesorg]

Would this be an Ingress Controller implementation for the Application Load Balancer?

[crizCraig]

Would this supercede #23483 as ALB's will correctly detect if one service goes down, rather than taking the whole instance out?

[ProTip]

IMHO I would say yes to both.

AWS is already referring to the ELB as "legacy"; Application Load Balancer looks like the future.

I would assume that this would be implemented as an ingress controller?

[tazjin]

I think an Ingress controller is the best way to go about this, similar to the GCP L7 load balancers. Would probably be part of k8s/contrib rather than this repo.

[jchv]

Would it not make sense to use ALBs in place of ELBs always, including in the case of Service type LoadBalancer? ELB still lacks support for WebSockets and HTTP/2. Therefore, as long as ALBs can be used as drop-in replacements for ELBs without any major consequences, it seems very desirable to do so.

[danopia]

In the CloudFormation API, ALBs are actually called ElasticLoadBalancingV2. There's not much chance of ELBV1 getting new features. ALBs are a little cheaper too

[tazjin]

Changing the LoadBalancer type services to use ALBs could probably be switched based on an annotation. Any thoughts about that?

[jchv]

The more I think about this, the less sure I am about how it should work. ALBs are decidedly Layer 7 LBs and only support HTTP (and /2 and websockets.) This is actually a good reason to not use them in place of ELB classic in the default case.

I took a look at the API. It looks like go-aws-sdk already supports ELBv2, which is nice. ELB classic instances and ALBs live in different namespaces.

Annotations to support ALB for Load Balancers may make sense as its still useful - but I think the idea to implement an ingress controller atop ELBv2 is probably the most sound idea, since ALBs do provide routing support. (I haven't tried using the Ingress resource, but looking at ALB rules vs Ingress rules, they definitely seem similar.)

[jokeyrhyme]

Is Kubernetes aware that certain tasks specifically offer HTTP services? Or does it only care about ports? Perhaps ELB v2 only makes sense in cases where we've specified (or it has been automatically detected)?

[hjacobs]

Having support for Application Load Balancer (ALB) could also solve #35187:

• register every pod in the ALB (own node port for each) --- this is not possible with Classic ELB
• forward the pod's node port to the pod (i.e. one iptables rule per pod)

=> We avoid one extra hop, preserve the source IP and avoid non-free cross-AZ data transfer (0.01 USD per GB)

[rajivm]

I don't think ELBv2/ALB could implement all the rules of an Ingress controller since it (still) does not support Header / Host based routing. ELBv2 only supports path based routing.

Currently, I am manually using ELBv2 in front of an Nginx Ingress controller to get the full functionality I need (HTTP 2 + Host routing to different services). It would be great to have direct support for ELBv2 as a LB option for all-http cases.

[sporkmonger]

And now ALB supports the AWS WAF, while ELB does not. Seems likely that this pattern of new feature releases targeting ALB primarily is likely to continue.

[hjacobs]

@rajivm yes, we have a similar approach using Skipper as HTTP router and another component to orchestrate ALBs (for SSL termination): zalando-incubator/kubernetes-on-aws#169

[cemo]

This would be a great improvement because ALB supports AWS WAF as well whereas ELB does not.

[metral]

@justinsb Are there any updates to support the integration of AWS ALB's in k8s? Specifically, is the IngressController implementation bubbling up as the desired approach for this?

Thanks!

[hjacobs]

@metral what IngressController implementation are you referring to? We build our own small ALB manager (we use it together with Skipper which has direct Kubernetes support): https://github.com/zalando-incubator/kube-ingress-aws-controller

ALBs don't yet directly support the host-based routing Ingress exposes, so you need to plug in something extra like nginx or Skipper.

[metral]

@hjacobs No implementation specifically. Was more wondering if ALB's were going to be conceptually tied to IngressController's in the future in some regard, given their close resemblance. Given the lack of header / host-based routing in ALB's, this seems to be a blocker for IngressController integration.

In other words, are there plans to integrate some of the custom work required to enable ALB's in k8s directly, as in your current solution, or is an implementation like yours the only route forward until ALB's and IngressControllers further align with one another?

[hrach]

Personally, I would suggest integrating ALB not as ingress controller, but only as the entry point to the cluster - for now. It supports:

• HTTP 2 (ELB not)
• WAF (ELB not)
• is less expensive than ELB ;-)

[jchv]

The problem is that it only supports HTTP, whereas LoadBalancer is by-default Level 3, and you can forward any TCP service on AWS right now.

[vendrov]

@johnwchadwick I'm not sure what is "Level 3", ELB does support TCP which is OSI 4.
Take a note that ALB does support WebSocket (which are blocked in HTTP in ELB).
While IMO ALB should be used as the default LB, It will be great If we could add annotation to the K8S-service specifying which type of LB to use.

[jchv]

Sorry, meant layer 4. I agree with allowing ALB through an annotation, but I don't think it makes sense to assume services are HTTP. That'd break most of my Kubernetes LoadBalancer services, which are largely not Websockets or any form of HTTP. If the LoadBalancer says "protocol": "TCP" it would be only reasonable to assume that it's a TCP proxy and not necessarily an HTTP one.

To be clear: I want ALBs in LoadBalancer too. However, I don't see how this could logically be a default setting.

[dpratt]

This would be fantastic - frankly I’d prefer to keep core ALB support to just LoadBalancer services. The new TCP ALB would be even better due to the source IP preservation.

[97turbotalon]

I'd also be very interested in the ability to configure ALB.

[jchv]

A TCP ALB is called a Network Load Balancer or NLB and as far as I know that is in Kubernetes 1.9 today.

[cdenneen]

@jchv yeah I think that eventually the "Classic ELB" will be deprecated and maybe if the default type=LoadBalancer can switch from elb -> nlb under the covers then we are forward compatible. Obviously elbv2 (ALB) is different so probably wouldn't be straight forward replacement here as NLB probably could be.

[hjacobs]

@cdenneen I totally agree, the Application Load Balancer (ALB) is a L-7 router and therefore clearly is suited for Ingress instead of service type=LoadBalancer. Use https://github.com/zalando-incubator/kube-ingress-aws-controller or https://github.com/coreos/alb-ingress-controller and be done with it 😄

[greg-jaunt]

NLBs do not support SSL termination and are therefore not a suitable replacement for ELBs for Kubernetes Services of type LoadBalancer.

[zparnold]

Correct, @greg-jaunt I would also say an NLB is a challenge too because you have to know in advance either the ENI ID's you're binding your TCP ALB (NLB) to, or know the IP address list in advance. Our cluster size is very dynamic and is only used to serve HTTP(s) traffic so the ALB is really the only thing I'd be working on. Let me know though if you think this'd be a wasted effort. 😄

[jchv]

If I'm not mistaking NLB is supported today with the service.beta.kubernetes.io/aws-load-balancer-type annotation. I'm also pretty sure it uses target groups the same way ALBs do, I'm not sure I see why it would be different.

The lack of SSL termination is a loss, but on one hand I wonder if that's even a problem. Is it mandatory for a Kubernetes LoadBalancer implementation to support SSL termination? Also, it seems possible that SSL termination at the cost of losing source IP preservation could be added into a future NLB update.

It seems really like the Ingress resource is what you want for ALB. You could implement ALB on the service level, it's no less unreasonable than implementing the HTTP/HTTPS protocol modes in ELB which Kubernetes does do as well. But, the Ingress resource will give you a lot richer functionality, and it works today. Is there a serious downside to using Ingress controllers for this?

[cdenneen]

The lack of SSL on NLB is actually a deal breaker for me. So for that reason alone I would like to see type=LoadBalancer for AWS actually support ALB under the coverts even if for L4 basic functionality with SSL. ALB features can be utilized further for L7 if needed with alb-ingress-controller but the basic type=LoadBalancer functionality for services would continue to work as a step towards migrating away from Classic ELB.

[jchv]

I'm unsure if I understand. Application Load Balancers do not support TCP load balancing and cannot function as L4 load Balancers at all. This is made clear in the documentation for ALBs:

An Application Load Balancer functions at the application layer, the seventh layer of the Open Systems Interconnection (OSI) model.

There is a load balancer type that runs at L4. It's called a Network Load Balancer in AWS parlance:

A Network Load Balancer functions at the fourth layer of the Open Systems Interconnection (OSI) model.

In the control panel, the three options (HTTP, TCP, Classic) simply map to ALB, NLB and ELB. Unless I'm horribly mistaken, there is no ALB mode that turns it into a TCP load balancer, and NLB is the only non-classic option for doing TCP load balancing.

[zparnold]

@jchv Looks like NLB is the only accepted option at present.

// ServiceAnnotationLoadBalancerType is the annotation used on the service
// to indicate what type of Load Balancer we want. Right now, the only accepted
// value is "nlb"
const ServiceAnnotationLoadBalancerType = "service.beta.kubernetes.io/aws-load-balancer-type"

Since someone else when through the trouble of figuring out wiring of AWS target groups to LB config (the new way of configuring Load Balancers in AWS,) I'll just try to hack together http/https with it. 😄 My usage of the cluster (at least for inbound traffic) is almost exclusively HTTP(s).

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[hjacobs]

I think it does not make sense to keep the ALB/ELBv2 open here as the existing Ingress controllers such as https://github.com/zalando-incubator/kube-ingress-aws-controller/ solve it.

/close

[BrianChristie]

FYI for reference:
SIG-AWS has taken over the CoreOS aws-alb-ingress-controller as an official SIG-AWS Subproject as of June 1st.

• Subproject Proposal: https://docs.google.com/document/d/1h6nfuE2DWTLsnJCHdJ7jq5lYVLW_tWOlDtUVNL5OGs8/view
• Subproject Repo: https://github.com/kubernetes-sigs/aws-alb-ingress-controller

[meringu]

I don't think the ALB ingress controller addresses all use cases for ALB support in Kubernetes.

• We currently use nginx-ingress (alb-ingress can not be used, as it has a different annotation API).
• Currently use an ELB with ACM SSL termination (NLB cannot be used to terminate SSL)
• We require http2 and WebSocket support (Cannot use ELB with proxy protocol on SSL backends https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/using-elb-listenerconfig-quickref.html. As this isn't supported, backend services can not detect the client scheme when only using TCP backends)

Seemingly the only way to support ACM, nginx-ingress, WebSockets, and http2 is to use an ALB in front of the nginx-ingress Kubernetes service.

Although ALBs can do Layer 7 specific routing, they don't have to. We can configure the ALB to send all traffic to the default backend. The ALB would them act like and ELB, but with http2 and WebSocket support, leaving nginx-ingress controller to do all the awesome traffic management.

I don't think we should change the annotation default to 'alb', but add it as an option much like 'nlb'.

[meringu]

AWS has just announced TLS termination with NLBs: https://aws.amazon.com/blogs/aws/new-tls-termination-for-network-load-balancers/

If this was supported in Kubernetes I would have no need for ALB support.

[ITBlogger]

Would like to see ALB support added for LoadBalancer type service as well

[GiladJudes]

+1 Would like to see ALB support added for LoadBalancer type service as well

[awsfanatic]

+1

[jamroks]

+1 for ALB support added for LoadBalancer type service, through some annotation like the one for nlb

service.beta.kubernetes.io/aws-load-balancer-type: alb

[jdomag]

+1

[zeroooing]

+1

[krystan]

I don't think ELBv2/ALB could implement all the rules of an Ingress controller since it (still) does not support Header / Host based routing. ELBv2 only supports path based routing.

Currently, I am manually using ELBv2 in front of an Nginx Ingress controller to get the full functionality I need (HTTP 2 + Host routing to different services). It would be great to have direct support for ELBv2 as a LB option for all-http cases.

This is no longer the case: https://aws.amazon.com/blogs/aws/new-advanced-request-routing-for-aws-application-load-balancers/

[Buffer0x7cd]

I think another use case can be, where cluster are already running with some other ingress controllers like Nginx or Kong, but want to utilize WAF, which is only available on ALBs. Having the support of choosing an ALB for LoadBalancer Type ( with support of some annotations) will simplify the setup

[callum-p]

👍 I've been wanting this for a while so I can use AWS WAF. I use the NGINX ingress controller. I don't want to swap to the ALB ingress controller that doesn't have the same feature set.

[minusv]

Any updates on this? I've been trying to use ingress controllers other than alb-ingress-controller but none of the controllers provides support for alb.
Annotation for service.beta.kubernetes.io/aws-load-balancer-type: alb will definitely be a huge help.