[ECS] : ECS Support for IPv6

[vibhav-ag]

Hi all,

Looking to get input on your IPv6 requirements as we develop our strategy here. IPv6 for ECS is a broad area and your feedback will help us prioritize our feature delivery to support the most important use cases.

Some topics that would be especially useful to get clarity on:

• ECS already supports dual stack (IPv4+IPv6) for Host and AWSVPC Networking modes. Do you need this to be extended to Bridge Networking mode?
• Do you need IPv6-only (IPv4 disallowed) access?
• What type of VPC resources do you want to access over IPv6?
• Are you planning to use IPv6 only within your VPC(s), or are you also planning to connect your Tasks to IPv6 internet?
• Do you require image pulls from ECR over IPv6?
• Do you plan to use ECS Service Discovery and/or AWS App Mesh with your ECS workloads with IPv6?

Anything else that is important to you!

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Which service(s) is this request for?
ECS, Fargate

[jfrlb]

1. Bridge mode would be extremely interesting, as we are trying to run many application instances on "own" (and separate) IPv6 addresses within the same EC2 instance. We have rather small containers that need something like 0.25 vCPU and 200MB RAM, so we would like to run several of those on small t3-instances, but that does not work with awsvpc mode due to the ENI limits (and not trunking on t-instances). Currently we are working around it with docker host mode, but I'd rather say that's really just a work around with associated disadvantages (e.g. container has to run as root).
2. We are absolutely lacking the ability to "fix" the IPv6 addresses used by the containers. Our (client) application needs to contact our backend behind static IPv6 addresses but currently ECS only allows to specify the /64 prefix; but we'd rather need something like an "elastic IP" for our backend service. It would be a great starting point already, if a "secondary" IPv6 could be configured to the EC2's ENI. Through docker host mode we could use that as well in our containers. It is already possible to configure this on the linux machine, but it looks like the additional address is not considered in the VPC routing and the VPC thus does not forward any packets to such addresses today.
3. Network load balancer does not support UDP + IPv6 combination!! And no IPv6 for internal load balancers at all.
4. We would need IPv6 multicast support within VPC networking (we are receiving traffic into the VPC through a VPN endpoint and would like to forward received multicast packets to a number of ECS containers who are listening for these)

[jfrlb]

5. Especially for development and test stages it would be beneficial if AWS/VPC would also support ULA addressing (RFC4193) for VPC traffic. This would allow to create setups with specified, static and independent IPv6 addresses that you can rely on.

[zoltanmajo]

For our setup currently running in production we'd need IPv6 support for tasks launched by ECS services. That way we could access services outside of the VPC using IPv6.

So from the points mentioned in the description the following matches our requirements the best:
"Are you planning to use IPv6 only within your VPC(s), or are you also planning to connect your Tasks to IPv6 internet?"

The answer is yes. Many thanks in advance.

[leezu]

Do you need IPv6-only (IPv4 disallowed) access?

Definitely. That's because AWS has bad support for IPv4 addresses on multi-network interface instance types such as p4d.
Multi-NIC instances do not support association of public IPv4 addresses at startup (they only support association of private IPv4 addresses and IPv6 addresses). Thus multi-NIC EC2 instances currently can't join the ECS clusters until an EIP is manually assigned or a NAT Gateway is configured in the subnet. NAT Gateway adds extra costs to customers and automatic EIP assignment is hard to configure for customers (requires implementation of Lambda functions based on SNS events of p4 instance startup).

Once ECS supports IPv6-only (IPv4 disallowed) access, ECS will work out-of-the box for multi-NIC instances.

[mogadanez]

ECS already supports dual stack (IPv4+IPv6) for Host and AWSVPC Networking modes.

but there is a lack of documentation on how to properly connect it to ALB.

ECS works with instance type of target group,
but in loadbalancer setting I cant create instance type with ipv6 selection

[tlaverdure]

Do you require image pulls from ECR over IPv6?

Yes, this would great for IPv6 egress only workloads.

[martin-pronestor]

I was experimenting setting up a VPC using dualstack and having two types of subnets (two public and one private) for a simple ECS hosted web application with an elastic load balancer in front. The images for the web application were hosted on ECR and I wanted to see if I could avoid having a NAT gateway (and elastic IPv4) associated to the private subnet and only use IPv6 and an egress-only gateway.

I was a little disappointed to find out that images couldn't be pulled from ECR to a running ECS with IPv6. It didn't even work if I used a different image repository.

Do you require image pulls from ECR over IPv6?

Yes, please. Hopefully this could fix our issue and we could save the cost associated with using a NAT gateway and Elastic IP.

[CGarces]

It didn't even work if I used a different image repository.

I don't get it. If you use and external repo like Docker Hub, egress-only gateway with IP6 not works?

we could save the cost associated with using a NAT gateway and Elastic IP.

Why you need and Elastic IP?.
Based on your requirements maybe VPC endpoint is cheaper than NAT gateway

[martin-pronestor]

I don't get it. If you use and external repo like Docker Hub, egress-only gateway with IP6 not works?

From the error message that I received, when the ECS task was provisioning, it said it got a TCP timeout using the IPv4 address of the repository, so I assume the agent responsible for pulling images doesn't know how to use the IPv6 version of the repository endpoint. In my tests I tried pulling from ECR and from GitHub private container registry (https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry).

Why you need and Elastic IP?. Based on your requirements maybe VPC endpoint is cheaper than NAT gateway

From what I've been told you have to have an elastic IP in order to use a managed NAT gateway. (https://stackoverflow.com/questions/43094786/why-does-a-aws-nat-gateway-require-an-elasticip)
I tried setting up VPC endpoint but ultimately found that it ends up being almost as expensive as the NAT gateway because you need to add S3, ECR-DKR, ECR-API, and possibly CloudWatch (https://docs.aws.amazon.com/AmazonECR/latest/userguide/vpc-endpoints.html). It is also way harder to set up and debug :(

[gregwebs]

Can we create separate issues to support an IPv6 egress only gateway? A lot of the comments here are in support of this and I think there are too many other things being asked about on this issue.

Personally I would really like IPv6 egress only because it is a more secure architecture to be able to block ingress at the gateway level.

[relgames]

Hi, it would be nice to support IPv6 addresses to register ECS tasks in ALB target group (awsvpc mode).
Right now, when I tried to enable IPv6 in the target group, ECS is trying to use ipv4 to register in the ipv6 target group:

service test-EcsService-fYJDJS3mLw5J failed to register targets in target-group test-tg  with (error The IP address '172.17.2.196' is not a valid IPv6 address)

The task itself has IPv6:

Private IP
172.17.2.196

IPv6 address
2a05:d018:1cee:b902:bd1e:854e:44a4:1f78

[julienp-b2wise]

Hi @vibhav-ag, as of today it's possible to launch fargate tasks in a dual stack subnet (ipv4 + ipv6), are there any plans to support ipv6-only subnets in the future?

[mbtran1]

Hi @vibhav-ag, as of today it's possible to launch fargate tasks in a dual stack subnet (ipv4 + ipv6), are there any plans to support ipv6-only subnets in the future?

hi @julienp-b2wise and team, good timing of your inquiry. We (ECS) have plans for expanded IPv6 support, where ECS will deliver IPv6 support for its tasks and services on all compute platforms in dual stack (IPv4+IPv6) and IPv6-only subnets, with load balancer integration, image pulls, Domain Name System (DNS), and Task Metadata Service (TDMS) over IPv6.

Couple questions -- Please share your requirements in terms of ECR public IPv6 endpoint support and ECR PrivateLink IPv6 endpoint support for production deployments. Is there a hard requirement for ECR public IPv6 endpoint support for production deployments in 2023? Thanks.

[ngoyal16]

Hi, it would be nice to support IPv6 addresses to register ECS tasks in ALB target group (awsvpc mode). Right now, when I tried to enable IPv6 in the target group, ECS is trying to use ipv4 to register in the ipv6 target group:

service test-EcsService-fYJDJS3mLw5J failed to register targets in target-group test-tg  with (error The IP address '172.17.2.196' is not a valid IPv6 address)

The task itself has IPv6:

Private IP
172.17.2.196

IPv6 address
2a05:d018:1cee:b902:bd1e:854e:44a4:1f78

We are also looking solution for same. @relgames have you found any alternative.

[julienp-b2wise]

Hi @mbtran1, thx for the feedback. As of our requirements we would just need to have ECS/Fargate tasks ipv6-only with the ability to pull from ECR of course, but we wouldn't need ecr ipv6 public endpoint support nor privatelink support at least for now.

[grahamschuckman]

Adding in the federal perspective here. Our agency client is responding to the White House OMB Mandate requiring that federal agencies begin operating in IPv6-only modes. See Page 3, Section 4a: https://www.whitehouse.gov/wp-content/uploads/2020/11/M-21-07.pdf

As such, we're looking to migrate our ECS workloads into an IPv6-only mode of operation.

[CGarces]

If we can't connect to ECR with IP6, there will be some impact con bills in 2024.
Run big Fargate clusters (each one with a public IPv4) will be more expensive.
https://aws.amazon.com/blogs/aws/new-aws-public-ipv4-address-charge-public-ip-insights/

[IllarionovDimitri]

Ran today onto the same issue. "IPv6 only" configuration is not yet supported by AWS, so the public IPv4 must be assigned to the tasks running in public subnets regardless of IPv6. If there is no solution until Feb. 2024, even attaching IPv6 will not prevent the costs increase for public IPv4

[tophercullen]

Hi, it would be nice to support IPv6 addresses to register ECS tasks in ALB target group (awsvpc mode). Right now, when I tried to enable IPv6 in the target group, ECS is trying to use ipv4 to register in the ipv6 target group:

service test-EcsService-fYJDJS3mLw5J failed to register targets in target-group test-tg  with (error The IP address '172.17.2.196' is not a valid IPv6 address)

The task itself has IPv6:

Private IP
172.17.2.196

IPv6 address
2a05:d018:1cee:b902:bd1e:854e:44a4:1f78

We are also looking solution for same. @relgames have you found any alternative.

@ngoyal16 same issue here with running Fargate tasks. It had the added benefit of causing all sorts of internal breakage with AWS. Tasks and services stuck in various defunct states for hours despite deleting.

[ngoyal16]

@tophercullen what we have did is we have added a event bridge listener for task creation and invoke lammda and adding the ipv6 as target to nlb... That what we are doing as of now..but if AWS team can this own our behalf would be much better solution

[majkel-hyggio]

Guys, isn't that the anwser https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-nat64-dns64.html ?
If I get it right, that switch should enable EC2 ipv6 only instances to connect to ECR.

[tophercullen]

@majkel-hyggio You will incur additional costs in both NAT gateway hours and data transfer. NAT data transfer costs are not cheap, and if you don't setup a NAT per subnet (multiply NAT gateway hour charges), you'll also incur regional data transfer costs. Depending on the various elements (task durations, counts, image sizes etc.) it may actually be cheaper to just to incur the costs of public ipv4 addresses.

In either case, you're simply paying more for AWS ECR just not having an IPv6 endpoint available.

[jnawk]

Given there's a charge for public IPv4 addresses coming in February, it's pretty important that there's a status-quo (ie free) alternative available, lest the change be viewed as purely an opportunistic money grab under the guise of IP address exhaustion.

[leonardlin]

I was caught off-guard by this too.

In preparation to ipv6 migration push (charge on public ipv4), I tried to move as much as possible to ipv6.

Turns out ec2 instances without public ipv4 can't join ecs-clusters.

ECS do not support IPv6 EC2 instances, even if they have a private IPv4

See also here: https://stackoverflow.com/questions/38599701/register-ec2-instance-to-ecs-cluster-without-public-ip

[ajpauwels]

Hi, it would be nice to support IPv6 addresses to register ECS tasks in ALB target group (awsvpc mode). Right now, when I tried to enable IPv6 in the target group, ECS is trying to use ipv4 to register in the ipv6 target group:

service test-EcsService-fYJDJS3mLw5J failed to register targets in target-group test-tg  with (error The IP address '172.17.2.196' is not a valid IPv6 address)

The task itself has IPv6:

Private IP
172.17.2.196

IPv6 address
2a05:d018:1cee:b902:bd1e:854e:44a4:1f78

@relgames Were you able to resolve this? Hitting it right now, and I can't tell if it was just never implemented, or there's a switch I need to flip.

[relgames]

No, seems like it's still not supported.

[timzuiddam]

And now it's February and we are hit with this, where's the customer-centric approach here? Still no update?

[gregwebs]

There are many AWS services lacking in IPV6 support: https://docs.aws.amazon.com/vpc/latest/userguide/aws-ipv6-support.html
AWS is making an estimated billion dollars a year (revenue, but there's certainly some profit there) charging for IPV4 addresses. That money could be used to fund a roadmap with commitments for full IPV6 support for various services.

[maxgruebneraeroqual]

We're also being reamed by AWS switching to charge for IPv4 addresses, but not providing IPv6 functionality required to move away from IPv4.

Pretty transparent cash grab from AWS, considering they'll make back the cost of the IPv4 addresses in under two years.

It's lovely running into situations like:

I need public addresses for ECS containers, because AWS Secrets Manager requires public IP addresses (for some reason?).
I can't make my ECS containers IPv6 only (for some reason?), so all my containers now cost me an extra $50 a year, or $150 if I have a load balancer in front of them (which are also only dualstack and must have at least two IPv4 addresses themselves).

Allegedly this can be worked around by using PrivateLink, so we get to waste a bunch of time trying to get that working (and hoping that there's no extortionate charges involved in that too).

[jnawk]

I need public addresses for ECS containers, because AWS Secrets Manager requires public IP addresses (for some reason?). I can't make my ECS containers IPv6 only (for some reason?), so all my containers now cost me an extra $50 a year, or $150 if I have a load balancer in front of them (which are also only dualstack and must have at least two IPv4 addresses themselves).

Allegedly this can be worked around by using PrivateLink, so we get to waste a bunch of time trying to get that working (and hoping that there's no extortionate charges involved in that too).

While this is a pretty shit move by AWS, it does sound like you can save a bit of money.
If you've got a load balancer, then you don't need your containers to have public IP addresses - chuck them in private subnets. You'll need a NAT gateway or NAT instance to provide them with outbound internet access to access the secrets manager endpoint, OR, you can create an interface endpoint for secrets manager in a private subnet, which will give your containers access to secrets manager without needing outbound internet access - there's a per hour charge associated, naturally - both with any NAT solution you use, and for the interface endpoint, but run the numbers, it might come to less than what you're otherwise paying.

Of note, by running your containers in private subnets, you'll need to make sure they can access your container registry - if that's ECR, then you'll need an ECR interface endpoint, and a S3 gateway endpoint. Same story with any other things your containers access.

Yeah, it might cost more to sort it out than you'd otherwise pay to AWS for IP addresses, so you might be better just paying that racket.

[Spacefish]

A first step would be to add Ipv6 support to ECR / dkr.ecr..
If i look at my ECR Registry, it´s CNAME pointing to a public network load balancer: nlb3-21115c3760ec6ec9.elb.ap-east-1.amazonaws.com
but this hostname only has an A record but no AAAA record.

Same thing for the layer download endpoint as well, prod-ap-east-1-starport-layer-bucket.s3.ap-east-1.amazonaws.com is a CNAME to s3-r-w.ap-east-1.amazonaws.com which has no AAAA record.

[klevo]

C'mon AWS this is embarrassing. Let's get that ECR IPv6 support in at least.

[maxgruebneraeroqual]

Yeah, it might cost more to sort it out than you'd otherwise pay to AWS for IP addresses, so you might be better just paying that racket.

With such transparent pricing claims as:

"It is important to note that although the NAT gateway does not charge for data transfer in, there will still be a per-GB data processing charge on data that flows through the NAT gateway, regardless of direction. The data transfer out charges are in addition to this charge. NAT gateway pricing can be found on the
Amazon VPC pricing page."

What could possibly go wrong?

But yes, it is more expensive to add all these workarounds and use 20 additional AWS services to compensate, so we just have to take it.

[mocknen]

I also tried, but still failed to specify ECS configured with IPv6 and Fargate as the target group for ALB, with the following error message.

service <SERVICE_NAME> failed to register targets in target-group <TARGET_GROUP_NAME> with (error The IP address '<ENI_IPV4_ADDRESS>' is not a valid IPv6 address)

I am certain that the ENI configured for the task has an valid IPv6 address.
I have also tried configuring my account for dualStackIPv6 as described in the documentation, but nothing has changed.

[TeoTN]

The whole issue could have been easily solved by AWS for low-cost tinkerers by offering up to 2-3 public ipv4 addresses in Free Tier, assignable to ALB, just like they offer one for an EC2 instance.

Given that barely anything at AWS at the moment supports ipv6-only mode, it's a ludicrous move. It's no customer centricity just purely squeezing as much money from people as possible, for no value added.

[gauravgoyal591]

Due to the AWS team charging for IPv4 and not being ready for IPv6 connectivity, please refrain from charging for IPv4 until your architecture is ready.

[TJNII]

Is this expected?

$ dig AAAA ecs.us-east-2.amazonaws.com

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.amzn2.13.5 <<>> AAAA ecs.us-east-2.amazonaws.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4379
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;ecs.us-east-2.amazonaws.com.   IN      AAAA

;; AUTHORITY SECTION:
ecs.us-east-2.amazonaws.com. 59 IN      SOA     ns-1999.awsdns-57.co.uk. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 60

;; Query time: 0 msec
;; SERVER: 10.192.0.2#53(10.192.0.2)
;; WHEN: Thu Feb 29 02:11:06 UTC 2024
;; MSG SIZE  rcvd: 170

IPv6 in my VPC is working, I was able to install dig into the ec2 instance running amazon/amzn2-ami-ecs-hvm-2.0.20240109-x86_64-ebs over IPv6. When I search for "ECS IPv6" the top hit is a press release proudly proclaiming "Amazon ECS now supports Internet Protocol Version 6 (IPv6) in awsvpc networking mode" so I'd expect this to work.

[gauravgoyal591]

No it's not working from the IPV6 servers Connect timeout on endpoint URL: "https://api.ecr.us-east-1.amazonaws.com/" [image: birdeye-icon] <https://birdeye.com/?utm_source=product_email&utm_medium=customers&utm_campaign=default_signature&utm_term=birdeye_icon> *Gaurav Kumar Goyal* Software DevOps Engineer (797) 620-2418 birdeye.com <https://birdeye.com/?utm_source=product_email&utm_medium=customers&utm_campaign=default_signature&utm_term=birdeye_dot_com>

…

On Thu, Feb 29, 2024 at 7:48 AM Tom Noonan II ***@***.***> wrote: Is this expected? $ dig AAAA ecs.us-east-2.amazonaws.com ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.amzn2.13.5 <<>> AAAA ecs.us-east-2.amazonaws.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4379 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;ecs.us-east-2.amazonaws.com. IN AAAA ;; AUTHORITY SECTION:ecs.us-east-2.amazonaws.com. 59 IN SOA ns-1999.awsdns-57.co.uk. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 60 ;; Query time: 0 msec ;; SERVER: 10.192.0.2#53(10.192.0.2) ;; WHEN: Thu Feb 29 02:11:06 UTC 2024 ;; MSG SIZE rcvd: 170 IPv6 in my VPC is working, I was able to install dig into the ec2 instance running amazon/amzn2-ami-ecs-hvm-2.0.20240109-x86_64-ebs over IPv6. When I search for "ECS IPv6" the top hit is a press release proudly proclaiming "Amazon ECS now supports Internet Protocol Version 6 (IPv6) in awsvpc networking mode" so I'd expect this to work. — Reply to this email directly, view it on GitHub <#1340 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATNDOEMIVVHJM24WW3R373DYV7QQVAVCNFSM424AH2SKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJXGAZDONRSGAZQ> . You are receiving this because you commented.Message ID: ***@***.***>

[CGarces-Apser]

Is this expected?

$ dig AAAA ecs.us-east-2.amazonaws.com

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.amzn2.13.5 <<>> AAAA ecs.us-east-2.amazonaws.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4379
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;ecs.us-east-2.amazonaws.com.   IN      AAAA

;; AUTHORITY SECTION:
ecs.us-east-2.amazonaws.com. 59 IN      SOA     ns-1999.awsdns-57.co.uk. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 60

;; Query time: 0 msec
;; SERVER: 10.192.0.2#53(10.192.0.2)
;; WHEN: Thu Feb 29 02:11:06 UTC 2024
;; MSG SIZE  rcvd: 170

IPv6 in my VPC is working, I was able to install dig into the ec2 instance running amazon/amzn2-ami-ecs-hvm-2.0.20240109-x86_64-ebs over IPv6. When I search for "ECS IPv6" the top hit is a press release proudly proclaiming "Amazon ECS now supports Internet Protocol Version 6 (IPv6) in awsvpc networking mode" so I'd expect this to work.

ECS partially support IPv6 but ECR not, ECR is the big issue
https://docs.aws.amazon.com/vpc/latest/userguide/aws-ipv6-support.html

[pawelfraczyk]

Hi, it would be nice to support IPv6 addresses to register ECS tasks in ALB target group (awsvpc mode). Right now, when I tried to enable IPv6 in the target group, ECS is trying to use ipv4 to register in the ipv6 target group:

service test-EcsService-fYJDJS3mLw5J failed to register targets in target-group test-tg  with (error The IP address '172.17.2.196' is not a valid IPv6 address)

The task itself has IPv6:

Private IP
172.17.2.196

IPv6 address
2a05:d018:1cee:b902:bd1e:854e:44a4:1f78

End of March getting the same error.

VPC enabled ipv6, all subnets with routing tables.
ALB in dual-stack mode
ECS enabled in region for whole account to use dual-stack mode
Target group with ipv6

ECS Fargate Task has both ipv4 & ipv6 and can't register as a target in TG because it tries ipv4....

Any solutions?

[Laurenfrost]

Hello! Any udpates here?

I got exactly the same error when I tried to register a ecs fargate task to an ipv6 target group.

service `test-service` failed to register targets in target-group `test-tg-tbt-v6` with (error The IP address '192.168.162.68' is not a valid IPv6 address)

[ajpauwels]

@pawelfraczyk @Laurenfrost

There is unfortunately no solution here, it requires AWS changing things on the back-end to detect if a target group is IPv6-only and then registering IPv6 targets rather than IPv4 targets.

[victorvhpg]

Any update ? @tabern