[ECS,Fargate] Multiple target groups for a service

[zaitsman]

Summary

I want to be able to route traffic from two different Path rules in my ALB to the same ECS instance and have ECS automatically register targets in both groups.

Description

Pretty much what the summary says. I can manually add targets to the second group, but if the service is restarted, then the second group loses the targets

Expected Behavior

Unable to have 2 targets for the same ECS service

Observed Behavior

N/A

Environment Details

Supporting Log Snippets

[dastbe]

hey @zaitsman, ECS service team member here.

We don't currently support multiple target groups per service today. However, if you're able to use the same port on the same ALB you can re-use the same target group for multiple rules on a single loadbalancer.

[alexzaytsev-newsroomly]

@dastbe I do use that, actually, but my use case is that i want the same container to respond to two different TG's for different ALB's. I understand that is a feature request.
One way to solve it would be if we could get a Cloudwatch alarm each time ECS registers something, we could then lambda those additional registrations.

[adnxn]

One way to solve it would be if we could get a Cloudwatch alarm each time ECS registers something

@alexzaytsev-newsroomly, have you looked at the ECS event stream? There are container instance registration/deregistration that are surfaced through CloudWatch events. Let us know if this captures your specific use case.

[hridyeshpant]

@dastbe does we have Multiple target groups for ECS service support in Fargate mode?

[bataras]

Is multiple target groups per ECS service feature going to happen? On the roadmap?

[DenisBY]

I need this feature too. Our application inside a container listens on 3 different ports for different purposes but I'm able to (automatically) register container only in one target group

[Raizex]

We could also really use this as well. We've reached the maximum number of certificates per load balancer and need a way to add the service to additional load balancers without creating a duplicate service.

[sergedomk]

I could use this as well. We would like to put up a single Fargate ECS service with several containers each providing a different branch of the code running on a different port for our product and QA teams to use.

[mcmar]

This is a blocking issue for me.
I have one primary service and one secondary service. The primary receives traffic for /management route. The primary and secondary both receive traffic for all other routes *.
This pattern is impossible to implement without the ability to register 2 target groups for the primary instance.

This is the pattern currently used by Prisma as described here:
prisma/prisma1#2850 (comment)

[zymtx5g79k]

I solved an issue by pointing current instances in stack, but it will not works with auto scaling. The feature will be useful.

[mcmar]

@zymtx5g79k Can you clarify what you mean by pointing current instances in stack? I'd love to get a solution to this and I'm not currently using autoscaling. Does your solution work with ECS on Fargate?

[zymtx5g79k]

@mcmar, hello:

EC2 -> Target groups -> Create target group -> Target type (instance) -> Create.
Select just created target group -> Actions -> Register and deregister targets -> Check all Instances that work in your stack -> Add to registered -> Save.

After some time the target group will detect which an instance has your service and will mark it as healthy. Load balancer will redirect traffic on the healthy instance.

In my case: it slanger service. Port 4567 was assigned in ECS service and port 8080 (websocket) is listen from all instances.

Tip: Target group need some time to correct healthy status after ECS service migration across EC2 instances. You can get connections refused until new healthy instances was found.

In my case: websocket connections will be repeated. It is regular case for frontend. I use EC2 stack "because I can"! (:

[theladyjaye]

I could use this too.

In my use case, we have one legacy application running behind an Application load balancer but rather than rebuild the whole application from scratch, we can go down the route of rebuilding pieces and just forwarding from the first Application Load Balancer to a new target group (listener rules). Unfortunately when using Fargate/ECS it owns the registrations for the Target Group and a target group can only be associated with 1 Load Balancer (which Fargate/ECS owns). So we are left with trying to figure out a way to mirror that Target Group when we get new ECS Registrations.

[m0n5t3r]

another use case would be something that exposes multiple ports (like, say, rabbitmq's 5672 (AMQP) + 1567 (management UI)); since network load balancers seem to force traffic to the target group port instead of keeping the listener port, one needs to have a target group for each port, but only one of them ever gets updated by ECS

[hajdukd]

This is a blocker - we require to have an option to register more than single port for our services.

[theladyjaye]

@hajdukd I hate to see you blocked by this. Per one of the comments above:

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_cwe_events.html

Using ECS Event Streams via CloudWatch events and triggering a Lambda definitely is a solution that works. It’s not as convenient if they offered it automatically, but you can add and remove from as many target groups as you want. So for a small bit of extra work (1 Lambda function and some event wiring) you will be unblocked.

[renuez]

Nathan Peck from ECS product team has recommended a few very useful options: https://gist.github.com/nathanpeck/a8537e21627b3a0f3e735eecbe6f6384

[coultn]

Thanks everyone for the feedback on this issue. I wanted to let you know that we on the ECS team are aware of it, and are actively considering solutions. +1's and additional details on use cases are always appreciated and will help inform our work moving forward.

[cnelson]

+1

[davidfti]

+1

[outofcoffee]

@coultn we are trying to model multiple ingress routes to a shared service. In particular, some traffic comes into a load balancer in a private subnet with one set of path routing rules, whilst other traffic comes via a load balancer in a public subnet with a different set of path rules.

Using separate security groups to logically represent the rulesets for the different ingress routes can be somewhat helpful here, but with the default max of 5 SGs per load balancer this can be a squeeze. It also doesn't provide the routing benefits of separate load balancers in public/private subnets.

Isolation at the task level is not required, and it would be inefficient to run multiple service instances, each with multiple task instances for redundancy, simply for the purposes of attaching multiple load balancers.

Essentially the pattern we're looking to emulate is how Pods can have multiple Services pointing to them in Kubernetes. This is one of the main drivers pushing us away from ECS.

[dodwmd]

+1

[senthilkumar-s]

+1

[sampsasaarela]

+1

[broglep-work]

We did a PoC with Cloud Watch Events & Lambda function. You can find the CF stack at https://gist.github.com/broglep-koubachi/0ceed52c509f75a6c26afc43b807a0eb
(it uses an ecs service and its load balancer config as basis to create targets for the twin target group)

[tkgregory]

+1 My use case is a Gitlab task that needs to expose ports 443 and 22. I'm currently using an ALB for the HTTPS, and would need another load balancer and target group for the SSH.

[sandeepboyapati]

We did a PoC with Cloud Watch Events & Lambda function. You can find the CF stack at https://gist.github.com/broglep-koubachi/0ceed52c509f75a6c26afc43b807a0eb
(it uses an ecs service and its load balancer config as basis to create targets for the twin target group)

This works perfectly.....Thanks @broglep-koubachi

[kokuryuha]

+1 as a feature request as well. Currently looking at deployments and realized this was going to be an issue immediately.

[joemoore13]

+1 the case of a single application listening on multiple ports (web and tcp) has multiple cases for many technologies and this makes working with them in ECS more difficult.

[abby-fuller]

moving this over to the containers roadmap since this is an ECS feature request, rather than an agent issue.

[ashayshub]

@coultn Is this going to be released only for Fargate? That will be a shame really cause I bet most of the users using it are asking it for EC2 as well!

[coultn]

@coultn Is this going to be released only for Fargate? That will be a shame really cause I bet most of the users using it are asking it for EC2 as well!

When this feature is launched it will support ECS on EC2 and ECS on Fargate (commonly known as just 'Fargate').

[chrobotm]

@coultn will this get released with cloudformation support?

[ledbit]

+1

[coultn]

@coultn will this get released with cloudformation support?

Yes, that is the plan.

[akshayram-wolverine]

Hi everyone! I am on the product management team on AWS Container Services. This feature is now in developer preview.

If you are interested in participating please see here: https://github.com/aws/containers-roadmap/tree/master/preview-programs/ecs-multipletg-service In the interest of your time, the link has examples to set up an ECS service with multiple target groups in both the EC2 and Fargate launch type. Please email me at akshram [at] amazon.com with your AWS account ID(s) and AWS region(s) to get started.

We would love to hear feedback!

[whereisaaron]

Thanks for the update @akshayram-wolverine, I read the examples and both the use cases there focus on using multiple ALBs for one ECS/Fargate service. The misses one common use case, i.e. imagine you have an ECS/Fargate service expose ports 80 and 443, and you want to expose both ports with one ALB or ELBv2. Will this new feature be able to support that? Can we have port 80 and 443 on the same Load Balancer going to the same ECS/Fargate service?

[akshayram-wolverine]

Hey @whereisaaron. Yes your use case should be supported as well! You should just create two target groups one for each port and attach it to your ECS service.

[anshul0915zinnia]

Thanks for the update @akshayram-wolverine . can it support the use like if have one container running in service can we link it to 2 target groups which will help us to resolve issue to host our application on internal ELB and external ELB?

…

On Tue, Jul 9, 2019 at 10:18 AM akshayram-wolverine < ***@***.***> wrote: Hey @whereisaaron <https://github.com/whereisaaron>. Yes your use case should be supported as well! You should just create two target groups one for each port and attach it to your ECS service. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#104?email_source=notifications&email_token=AB4BMPJLGHPELHZ5RI57GC3P6QKBLA5CNFSM4GPJVEQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZPCGUY#issuecomment-509485907>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB4BMPNIGHZRJU5XM4WPZG3P6QKBLANCNFSM4GPJVEQQ> .

-- Regard, Anshul Agarwal Mob No:+91-7830892922

[akshayram-wolverine]

Hey @anshul0915 Yes!

[pitkley]

Hi @akshayram-wolverine. These are great news, thank you for implementing this feature-request!

One question I have is whether or not target-groups of instance-type ip are supported with this change? Right now assigning a network load balancer target-group whose instance-type is set to ip, rather than the default instance, fails with a message noting that only instance-mode is supported.

Our use-case is the ability to run TCP-based services within an ECS-cluster, making these services available through a network load balancer, and have those services be reachable by other services within the same ECS node! The last point is crucial: if the request is originating from the same instance the NLB tries to route to, i.e. the source instance-ID and destination instance-ID are identical, the request will not be able to propagate through the NLB (I can't find the documentation for this right now, unfortunately).

Should this be a concious decision or technical limitation, is there a "workaround" or alternative solution to this problem? (I.e. should one use the awsvpc networking in this case?)

(This question might be hijacking this issue, but I was unable to find a more fitting one. Please let me know if I should open a new issue for this!)

[whereisaaron]

@pitkley: Loopback timeout

[pitkley]

@whereisaaron thank you, that is what I was unable to find. If I understand the points made by the documentation correctly, running the containers with the awsvpc networking-mode should fix my issue. I'll test this as soon as I get around to it and report back.

[pmoghaddam]

Will there be a way to distinguish which ALB the request is coming from?

In the case of internal vs. external, if you cannot tell the difference, it'll be challenging to effectively use this.

[coultn]

Launched! https://aws.amazon.com/about-aws/whats-new/2019/07/amazon-ecs-services-now-support-multiple-load-balancer-target-groups/

[GoeranEnvirio]

Is it possilbe to configure a current fargate service this way:
aws ecs update-service --service my-http-service --cli-input-json loadbalancers.json

where I have put the loadbalancers like this is in the loadbalancers.json file:

"loadBalancers":[
{
"targetGroupArn":"arn:aws:elasticloadbalancing:region:123456789012:targetgroup/target_group_name_1/1234567890123456",
"containerName":"jenkins",
"containerPort":8080
},
{
"targetGroupArn":"arn:aws:elasticloadbalancing:region:123456789012:targetgroup/target_group_name_2/6543210987654321",
"containerName":"jenkins",
"containerPort":50000
}
]

[GoeranEnvirio]

It did not work got:

Error parsing parameter 'cli-input-json': Invalid JSON: No JSON object could be decoded
JSON received: loadbalancers.json

[Istador]

Is it possilbe to configure a current fargate service this way:
aws ecs update-service --service my-http-service --cli-input-json loadbalancers.json

where I have put the loadbalancers like this is in the loadbalancers.json file:

[...]

If I remember correctly, the loadBalancers settings can't be changed once the service is created. Neither with the aws ecs update-service command nor with the aws console.

You should be able to create a new service with aws ecs create-service that is using multiple loadBalancers, and then delete the old one.

It did not work got:

Error parsing parameter 'cli-input-json': Invalid JSON: No JSON object could be decoded
JSON received: loadbalancers.json

The example in the documentation of create-service is using the following syntax with file://:

--cli-input-json file://ecs-simple-service-elb.json

---

BTW another use case that is working thanks to this feature:
Having two target groups for the same ecs service, with one target group being sticky and the other being non-sticky. The public load balancer decides which paths needs to be sticky or not, by forwarding to the designated target group.

[GoeranEnvirio]

@Istador thanks for the info. It's really irritating when it's almost undocumented and you need the feature.

[GoeranEnvirio]

I tried the following:
aws ecs create-service
--cluster TestDevEC2Cluster
--service-name ecs-multiport-service-nlb
--launch-type EC2
--cli-input-json file://ecs-multiport-service-nlb.json

An error occurred (InvalidParameterException) when calling the CreateService operation: Not support Classic Load Balancer in multiple load balancers.

I have configured an NLB! So what could the issue be?

If I add a role: "AWSServiceRoleForECS"

It will complain: An error occurred (InvalidParameterException) when calling the CreateService operation: You cannot specify an IAM role for services that require a service linked role.

The file:

{
"serviceName": "ecs-multiport-service-nlb",
"taskDefinition": "test-ec2task:2",
"loadBalancers": [
{
"loadBalancerName": "NewTestNLB",
"containerName": "test",
"containerPort": 80,
"targetGroupArn": "xxxxx"
},
{
"loadBalancerName": "NewTestNLB",
"containerName": "test",
"containerPort": 2022,
"targetGroupArn": "xxxxx"
}
],
"desiredCount": 1
}

So how can I create it without an error?

SUCCEDED!

There was a lot of rabbit holes and hidden ramifications. Probably the reason the examples do not work. The only way to fix it is to read a lot.

[quolpr]

@GoeranEnvirio the issue is that you specified loadBalancerName in loadBalancers. Only targetGroupArn, containerName, containerPort should be specified

The only way to fix it is to read a lot.

But the better way - describe how did you fix your issue, so that will save a lot of time of other guys ;)

[stnor]

I have a use case where I would need a service to use two target groups. One is an NLB which forwards requests (SSL client cert) to one port on the service, and another is used on an ALB.

I'm unable to configure this in the web console. and according to support (Case ID - 6524068251), this is not supported. Please consider this use case in the future, thanks. As it is right now, I need to run two services that does the same thing.

Thanks!

[wedneyyuri]

@stnor it's really strange because your usecase is already supported but you will need to recreate your ECS service.

[stnor]

@wedneyyuri According to AWS Support, a mix of NLB and ALB target groups isn't supported.

[kstro21]

According to the documentation here

The service must use either an Application Load Balancer or Network Load Balancer.

We can't use both for the same service. I would benefit from being able to use both. Please, consider this use case.