Is there any documentation for making bottlerocket work without the internet access to the instances?

[soura49]

• We are using Bottlerocket AMI for EKS-managed Node groups
• Right Now, We have Egress for Internet open from Node Security Group
• But when we Remove that it fails to join the cluster and load the kernel modules etc.
• Is there a list of Internet Calls that Bottlerocket AMI does for starting up?

[jpculp]

I got this working in the past with AWS PrivateLink by adding the following regional VPC endpoints:

ECR (even if you use a different registry it's necessary for host containers and pause container)

• com.amazonaws.us-west-2.s3 (gateway) # required to pull the ECR image layers
• com.amazonaws.us-west-2.api
• com.amazonaws.us-west-2.ecr.dkr

EKS

• com.amazonaws.us-west-2.eks
• com.amazonaws.us-west-2.sts # required when using IAM roles for service accounts

IMDS (necessary for populating settings on first boot)

• com.amazonaws.us-west-2.ec2
• com.amazonaws.us-west-2.ec2messages

Bottlerocket control container

• com.amazonaws.us-west-2.ssm
• com.amazonaws.us-west-2.ssmmessages

[soura49]

Thank you will make these changes and will test it out.

[soura49]

• Thank you @jpculp
• I made progress on the things.
• Only thing that is failing right now is that it is in NotReady State foreever.
• Are there any other API calls that are required but missing?

[jpculp]

Are these fresh nodes from after the endpoints were created? If not they might be missing some settings that they couldn't populate on first boot. I would try spinning down the cluster and then spinning back up with new nodes.

[soura49]

• These are Fresh Nodes i created a new managed nodegroup after making the changes and they are still in NotReady State.
• I found this document https://repost.aws/knowledge-center/eks-cluster-node-group-private-network which mentions creating the VPC endpoint for STS and cloud watch logs along with the ones that you have mentioned above.
• I am concerned only about the STS do we need the STS VPCendpoint as well?

[larvacea]

EKS private clusters says that an STS endpoint is required if your pod uses IAM Roles for service accounts.

[soura49]

• The exact issue was aws-vpc-cni was not healthy because of the IAM role for Service Account that is making it NotReady.
• Once the STS Vpc endpoint is made it is healthy.

[larvacea]

Thanks for the update! It's great to hear what unblocked you.

[jpculp]

Awesome find! I added it to the list above.