Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DNS resolution for EKS Private Endpoints #221

Open
tabern opened this issue Mar 26, 2019 · 25 comments

Comments

@tabern
Copy link
Contributor

commented Mar 26, 2019

Automatic DNS resolution for EKS private endpoint private hosted zone within VPC. This allows fully direct connect functionality from on-prem computers to Kubernetes cluster API server that is only accessible within a VPC.

Follow-up feature from #22

@tabern tabern created this issue from a note in containers-roadmap (We're Working On It) Mar 26, 2019

@tabern tabern added the EKS label Mar 26, 2019

@rajarajanpsj

This comment has been minimized.

Copy link

commented Mar 26, 2019

@tabern Thanks for creating a separate issue for this. We want to see if there is a temporary way to tackle this issue assuming we definitely want to use the setting of disable public access and enable private access. At this point, even if we have to setup some extra hacks, would something like below work? Do you see any issues?

  1. Have a lambda within the VPC where EKS cluster resides that will pick up the cluster endpoint hostname/IP (Is this possible? I dont see the PHZ that EKS manages, so where do we get the IP) and update a common Route53 resolver running a common VPC (assuming our datacenter DNS is setup up to forward to this common Route53 resolver which will give back the private IP of the EKS endpoint).

  2. Will the above logic work? Do we have like a private ip from the VPC per subnet mapped to EKS point DNS name? This would be the case if it internally uses the private link technology right? Or is this an incorrect assumption?

  3. Does the EKS private endpoint itself use NLB at the background? So i can assume that the if I map the private IP to the EKS DNS name once, then it is guaranteed not to change? or it can change anytime and we have to constantly check for that from our lamdba?

@nurus

This comment has been minimized.

Copy link

commented Mar 29, 2019

I wanted to add that it might be beneficial for the EKS cluster api endpoints to appear as a target for Route53 Alias records. We are able to resolve a Route53 private hosted zone over VPN but using a CNAME created in this private zone doesn't help us resolve the shadow eks.amazonaws.com domain created for the cluster. An ALIAS record would return A records to the endpoints and solve our problem.

@ikester

This comment has been minimized.

Copy link

commented Mar 29, 2019

@nurus did you try creating ALIAS records for the EKS endpoint on the Route53 private hosted zone?

@nurus

This comment has been minimized.

Copy link

commented Mar 29, 2019

@ikester, I tried to create it manually through the console but was unable to but this probably because there is no hosted zone provided for the EKS endpoint. Looking at doing this with the aws cli or terraform shows that this is a required parameter.

@rajarajanpsj

This comment has been minimized.

Copy link

commented Apr 1, 2019

@nurus Did it work, it will be great if you give more information on how you made it work.

@nurus

This comment has been minimized.

Copy link

commented Apr 2, 2019

@rajarajanpsj, unfortunately not. AWS does not provide the hosted zone of the EKS cluster api endpoint which is a required parameter to create an ALIAS record.

@kivagant-ba

This comment has been minimized.

Copy link

commented Apr 2, 2019

Hello. Overall the change is very nice.
What about a peered VPC? For example if a CI/CD system installed in another VPC how to access the private EKS API endpoint?

@joshkurz

This comment has been minimized.

Copy link

commented Apr 2, 2019

what's the concern for just making the DNS entry public? I think that would solve all these issues.

@pwdebruin1

This comment has been minimized.

Copy link

commented Apr 3, 2019

I agree with @joshkurz it worked for RDS, if your RDS instance is privately exposed your RDS endpoint DNS name is resolvable publicly to a private IP within your VPC.

@wico

This comment has been minimized.

Copy link

commented Apr 11, 2019

One way to workaround this issue is a HTTPS/CONNECT proxy (e.g. tinyproxy) running inside the VPC, which allows connecting to the kubernetes private endpoint but without asking the VPC DNS from the client side. But thats not a nice solution.

It would be really really good if the authorative DNS-servers for sk1.eu-west-1.eks.amazonaws.com would also have a knowledge about the private endpoint records and not only the VPC internal DNS.

@cansong

This comment has been minimized.

Copy link

commented Apr 11, 2019

We managed to solve this by using Route53 Resolver. Basically, you want an Inbound Endpoint in your EKS cluster VPC, an Outbound Endpoint in your peered VPC, and a rule associated to your peered VPC that forwards your cluster domain name requests to the Inbound Endpoint ip addresses.

Don't forget to allow UDP 53 on your cluster security group for your peered VPC outbound endpoint ip addresses, and to check your on-going Network ACL rules.

@pwdebruin1

This comment has been minimized.

Copy link

commented Apr 11, 2019

@cansong genius! Would be cool to get more native support but best workaround I've seen so far!

@ikester

This comment has been minimized.

Copy link

commented Apr 11, 2019

@cansong That's a good solution if you can have peered VPCs like that. We've been documenting that approach and will be releasing it soon to help customers for whom the multi-VPC solution works.

@wico

This comment has been minimized.

Copy link

commented Apr 12, 2019

For all the others who cannot do the the solution mentioned by @cansong, we really need the zones being public (or having an option to make the zone public or private), as written by #221 (comment) :)

@IuryAlves

This comment has been minimized.

Copy link

commented Apr 16, 2019

For all the others who cannot do the the solution mentioned by @cansong, we really need the zones being public (or having an option to make the zone public or private), as written by #221 (comment) :)

@wico Genius!!

@robertgates55

This comment has been minimized.

Copy link

commented Apr 25, 2019

We're using a transit gateway to route between our VPCs. Is there a workaround using resolvers (or any workaround at all!) that we could try out?

@tabern

This comment has been minimized.

Copy link
Contributor Author

commented Apr 25, 2019

All - wanted to let you know we published a blog that details how to enable resolution for private endpoints when using a peered VPC or Direct Connect: https://aws.amazon.com/blogs/compute/enabling-dns-resolution-for-amazon-eks-cluster-endpoints/

This has been developed and test by AWS solution architects and the EKS team. Let us know if it works or doesn't work for you.

We'll be keeping this issue open as we consider delivery of this feature to be when resolution works automatically for EKS clusters and the team is continuing to develop a integrated solution for private endpoint DNS resolution.

@wico

This comment has been minimized.

Copy link

commented Apr 26, 2019

Unfortunately (and as mentioned), the above solution eventually does not help if people/pods/... still query e.g. 8.8.8.8 (and if you don't intercept this traffic).

An example:
In EKS, the builtin-dns resolution (coredns) comes per default with a setting to proxy (forward) DNS-querys to the VPC dns and as a fallback to 8.8.8.8:

The coredns-pod has this config:

/ # cat /etc/resolv.conf
nameserver <VPC_DNS>
nameserver 8.8.8.8

That means: Even with a solution as mentioned by @tabern and even if you are inside the VPC - if you have e.g. 2 EKS clusters (cluster 1 and cluster 2) and you do operations from cluster 1 against the API of the cluster 2, you could end up with not being able to resolve the API endpoint of that cluster 2. This happens if your in-cluster coredns of cluster 1 forwards the request to 8.8.8.8 to resolve the API of the cluster 2.

Sure, the settigs for coredns could be changed to not use (remove) 8.8.8.8 (and also, the resolv.conf has options timeout:2 attempts:5) - but I just wanted to illustrate the problem with a specific example. And I hope I did not miss anything. :)

Note: The above applies to EKS kubernetes 1.11. I have not testet with 1.12 - maybe the default coredns-config is different there.

@shinka81

This comment has been minimized.

Copy link

commented Apr 30, 2019

Creating a conditional forwarding rule for each private API endpoint seems very burdensome, hopefully what @joshkurz suggested is being considered as that seems like the most hands off solution to me. in our case there are some controls around usage of the appliance (infoblox) where DNS is managed on prem where these forwarding rules would be programmatically created so it is difficult for us to automate that after creating these clusters.

@mvillumsen

This comment has been minimized.

Copy link

commented May 2, 2019

@tabern I just used the blog post to setup dns resolution for EKS from our on-premise network using a VPN connection and it works fine. So thanks for the guide. However, as previously mentioned it would be a lot easier if you were just able to choose whether or not you want the DNS zones to be public or private available and control the access using a security group.

@hobti01

This comment has been minimized.

Copy link

commented May 20, 2019

@tabern the proposed solution is good when it is allowed to modify the DNS resolution. Unfortunately we cannot do this due to security policies.

It would be great to have the ability to assign a custom hostname/FQDN for the API endpoint which could be managed simply via Route53. Of course the server certificate should include the custom name(s).

@chrisferry

This comment has been minimized.

Copy link

commented May 21, 2019

@tabern this doesn't help for transit gateways. When do we expect to see support for them?

@MithilShah

This comment has been minimized.

Copy link

commented May 29, 2019

One interim solution is to modify the /etc/hosts file on the developer machine to locally resolve the endpoint dns. Here's a blog that shows a shell script that can do that automatically.
http://www.studytrails.com/devops/kubernetes/local-dns-resolution-for-eks-with-private-endpoint/

@ttercero

This comment has been minimized.

Copy link

commented May 29, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.