Create EKS Fully-Private cluster with bottlerocket

[joshthebeast]

What were you trying to accomplish?

I would like to create a EKS Fully-Private cluster with bottlerocket using eksctl. However, it seems the bottlerocket worker nodes are not able to join the cluster.

What happened?

I can see the error message Error: timed out (after 25m0s) waiting for at least 2 nodes to join the cluster and become ready in "self" after the worker node stack been created :

2021-05-18 10:31:43 [ℹ]  building nodegroup stack "eksctl-test-nodegroup-self"
2021-05-18 10:31:43 [ℹ]  --nodes-min=2 was set automatically for nodegroup self
2021-05-18 10:31:43 [ℹ]  --nodes-max=2 was set automatically for nodegroup self
2021-05-18 10:31:46 [ℹ]  deploying stack "eksctl-test-nodegroup-self"
2021-05-18 10:31:46 [ℹ]  waiting for CloudFormation stack "eksctl-test-nodegroup-self"
2021-05-18 10:32:04 [ℹ]  waiting for CloudFormation stack "eksctl-test-nodegroup-self"
2021-05-18 10:32:22 [ℹ]  waiting for CloudFormation stack "eksctl-test-nodegroup-self"
2021-05-18 10:32:39 [ℹ]  waiting for CloudFormation stack "eksctl-test-nodegroup-self"
2021-05-18 10:32:59 [ℹ]  waiting for CloudFormation stack "eksctl-test-nodegroup-self"
2021-05-18 10:33:16 [ℹ]  waiting for CloudFormation stack "eksctl-test-nodegroup-self"
2021-05-18 10:33:34 [ℹ]  waiting for CloudFormation stack "eksctl-test-nodegroup-self"
2021-05-18 10:33:54 [ℹ]  waiting for CloudFormation stack "eksctl-test-nodegroup-self"
2021-05-18 10:34:10 [ℹ]  waiting for CloudFormation stack "eksctl-test-nodegroup-self"
2021-05-18 10:34:30 [ℹ]  waiting for CloudFormation stack "eksctl-test-nodegroup-self"
2021-05-18 10:34:46 [ℹ]  waiting for CloudFormation stack "eksctl-test-nodegroup-self"
2021-05-18 10:35:04 [ℹ]  waiting for CloudFormation stack "eksctl-test-nodegroup-self"
2021-05-18 10:35:23 [ℹ]  waiting for CloudFormation stack "eksctl-test-nodegroup-self"
2021-05-18 10:35:42 [ℹ]  waiting for CloudFormation stack "eksctl-test-nodegroup-self"
2021-05-18 10:35:43 [ℹ]  waiting for the control plane availability...
2021-05-18 10:35:43 [✔]  saved kubeconfig as "/Users/josh/.kube/config"
2021-05-18 10:35:43 [ℹ]  no tasks
2021-05-18 10:35:43 [✔]  all EKS cluster resources for "test" have been created
2021-05-18 10:35:44 [ℹ]  adding identity "arn:aws:iam::<Account>:role/eksctl-test-nodegroup-sel-NodeInstanceRole-181GOSNF3D564" to auth ConfigMap
2021-05-18 10:35:45 [ℹ]  nodegroup "self" has 0 node(s)
2021-05-18 10:35:45 [ℹ]  waiting for at least 2 node(s) to become ready in "self"
Error: timed out (after 25m0s) waiting for at least 2 nodes to join the cluster and become ready in "self"

How to reproduce it?

1. I am using the following template from the example and adding privateCluster section:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: test
  region: us-west-2

privateCluster:
  enabled: true
  additionalEndpointServices:
  - 'autoscaling'
  - 'logs'

nodeGroups:
  - name: self
    instanceType: m5.xlarge
    privateNetworking: true
    desiredCapacity: 1
    amiFamily: Bottlerocket
    labels:
      "network-locality.example.com/public": "true"
    bottlerocket:
      enableAdminContainer: true
      settings:
        motd: "Hello, eksctl!"

2. using the following command to create the EKS cluster :

$ eksctl create cluster -f cluster_config.yml

Versions

$ eksctl version
0.49.0
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:23:52Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.15-eks-cf75c9", GitCommit:"cf75c9601ec63a63ba54c1c47caa9945af1f3a81", GitTreeState:"clean", BuildDate:"2021-02-13T00:14:16Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

May I know if it is the expected behavior ?
I can create the EKS cluster and worker node successfully without adding privateCluster.

[joshthebeast]

I found an existing issue [EKS] [request]: VPC endpoint support for EKS API #2984 . Not sure if they are related. Does bottlerocket need to run bootstrap.sh to connect themselves to the cluster ?

[yysu]

@cPu1 I can see the PR #3737 was merged. Does this PR also fix this issue ?

[cPu1]

@cPu1 I can see the PR #3737 was merged. Does this PR also fix this issue ?

@yysu I misread the title of the issue and have removed my earlier comment. That PR fixes the issue for Amazon Linux 2 and Ubuntu nodegroups but not for Bottlerocket. We'll be working on a fix for Bottlerocket soon.

[yysu]

@cPu1 Thanks for checking this issue. It's great to hear that !

[vranystepan]

I believe it's caused by metadata.settings.kubernetes.cluster-dns-ip.setting-generator that connects to EKS API via pluto.
Perhaps pluto can be customized a bit here https://github.com/bottlerocket-os/bottlerocket/blob/0fe7e8f5ac3d5a6d2994504ff1f29c8e661659cd/sources/api/pluto/src/main.rs#L273 to allow standalone call of get_cluster_dns_from_imds_mac 🤷

It seems that I was able to accomplish the connection with the following:

[settings.kubernetes]
"cluster-dns-ip"  = "172.20.0.10"

But it's not the universal solution since the IP might differ.