[EKS] [request]: Remove requirement of public IPs on EKS managed worker nodes

[atheiman]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
Remove requirement of public IPs on EKS managed worker nodes. If worker nodes have egress access to the apiserver and the apiserver can reach the worker nodes in the same vpc by private ip, I dont think public IP should be required. Assigning public ips to k8s worker nodes (or any ec2 instances) is a security vulnerability some organizations wont accept.

Which service(s) is this request for?
EKS

update 4/22
This change is now made, see the details here

---

update 4/17
We're planning to make the change to managed node groups to stop assigning public IPs to nodes on April 22, 2020.

If you are launching nodes into public subnets, you'll need to change your subnet settings to set the mapPublicIpOnLaunch to TRUE so that IPs are assigned and the nodes can connect to the public cluster endpoint. If you are not using public subnets, starting April 20, you can create a new node group and public IPs will no longer be assigned.

We'll update on this thread when the change is live.

We wrote a blog announcing this change and how to check the public IP assignment settings for your VPC.

We also wrote a deep dive blog on node networking that explains the various options for subnet and cluster endpoint configuration.

[rtripat]

We are evaluating an API opt-in to disable setting public IP on managed nodes. However, for some context on why Managed node groups get public IP in assigned in all subnets. From our documentation

"Amazon EKS managed node groups can be launched in both public and private subnets. The only requirement is for the subnets to have outbound internet access. Amazon EKS automatically associates a public IP to the instances started as part of a managed node group to ensure that these instances can successfully join a cluster.

This ensures compatibility with existing VPCs created using eksctl or the Amazon EKS-vended AWS CloudFormation templates. The public subnets in these VPCs do not have MapPublicIpOnLaunch set to true, so by default instances launched into these subnets are not assigned a public IP address. Creating a public IP address on these instances launched by managed node groups ensures that they have outbound internet access and are able to join the cluster."

[atheiman]

Yea, thats the part in your documentation I was surprised by. Seems like an unnecessary security hole going a long way to ensure nodes have outbound internet access. Public IPs on EC2s is a security violation in lots of organizations.

I really think opting out of public IPs on worker nodes will be a requirement for many organizations to even try out managed worker nodes.

[whereisaaron]

How about defaulting to what the user asked for 😄

If a users launches instances in a subnet they have configured with MapPublicIpOnLaunch disabled, the nodes should not get a public IP assigned by default - isn't that the whole point of the setting? It seems unhelpful and counter-intuitive to override users' express request not to add public IP addresses.

Could you instead warn/inform the user in AWS console or eksctl output that the nodes are launching in a subnet with public IP disabled? And point them to the documentation to fix that if they made mistake by disabling public IPs?

Please make this subnet override an optional setting.

[Siva-R]

In which version of EKS we can expect this to be fixed?

[omerfsen]

+1

[davidalger]

We are evaluating an API opt-in to disable setting public IP on managed nodes. However, for some context on why Managed node groups get public IP in assigned in all subnets. From our documentation…

While the quoted documentation does mention eksctl or CF template related reasoning, it seems a rather contrived reason. Require these networks be upgraded to utilize managed node groups and provide an upgrade path. The requirement is that "subnets have outbound internet access" which can be accomplished by having a NAT Gateway on the subnet. I.e. fulfilling the stated requirement to allow the worker node access to the control plane doesn't even need to require a Public IP be associated to the worker node.

AWS documentation on Cluster VPC considerations states the following:

We recommend a network architecture that uses private subnets for your worker nodes and public subnets for Kubernetes to create internet-facing load balancers within.

To me it seems rather bizarre that EKS best practices would outline creating both public and private subnets (with the public ones being used for things like ELB creation) yet a major managed feature addition to EKS was launched with a Public IP magically added to the EC2 instances created in a "private" subnet.

IMO an "API opt-in to disable setting public IP on managed nodes" is the wrong way to go about it. It should be opt-in to have public IP addressing applied to instances created in subnets where public addressing is turned off.

[bchav]

Hey folks, dropping in to provide some clarification around this to help with some of the security concerns here. This is indeed confusing, and we're working on product changes to make this less so, but it doesn't place your managed nodes in private subnets at additional risk.

Due to the way VPC networking works, managed nodes with a public IP in a private subnet do not behave any differently than regular instances in private subnets. All egress to the internet would require traversal of a NAT Gateway or NAT instance, and the public IP address will not be used. Public IPs require a route to the internet via an Internet Gateway (IGW). As such, the public IP of these instances are not accessible from the internet. The public IP address is effectively a no-op. The only access to nodes in private subnets comes from other resources that share the "Cluster Security Group".

More about the cluster security group: https://docs.aws.amazon.com/eks/latest/userguide/sec-group-reqs.html#cluster-sg

[adonley]

Even though the above reasoning makes absolute sense, my org requires that we document and port scan any "public ip" for compliance. We could write a tool that looks for nodes coming and going from node groups and listens to node group creation but it would make life easier not to deal with these issues (also it's less confusing for the security team / auditors ;) )

[prashant-wipro]

Any ETA on when this fix will be made available.