Ubuntu2004 managed nodes fail to come up on k8s 1.20

[aclevername]

started failing consistently:

https://github.com/weaveworks/eksctl-ci/runs/4389011807?check_suite_focus=true#step:7:42981
https://github.com/weaveworks/eksctl-ci/runs/4394682147?check_suite_focus=true#step:7:42813

[15] 2021-12-02 12:59:19 [✖]  unexpected status "ROLLBACK_IN_PROGRESS" while waiting for CloudFormation stack "eksctl-it-managed-wonderful-creature-1638447107-nodegroup-ubuntu"
[15] 2021-12-02 12:59:19 [ℹ]  fetching stack events in attempt to troubleshoot the root cause of the failure
[15] 2021-12-02 12:59:20 [▶]  AWS::CloudFormation::Stack/eksctl-it-managed-wonderful-creature-1638447107-nodegroup-ubuntu: ROLLBACK_IN_PROGRESS – "The following resource(s) failed to create: [ManagedNodeGroup]. Rollback requested by user."
[15] 2021-12-02 12:59:20 [✖]  AWS::EKS::Nodegroup/ManagedNodeGroup: CREATE_FAILED – "Nodegroup ubuntu failed to stabilize: [{Code: NodeCreationFailure,Message: Instances failed to join the kubernetes cluster,ResourceIds: [i-08ce0c00104cedf0d, i-0a20429db29afca17]}]"
[15] 2021-12-02 12:59:20 [▶]  AWS::EKS::Nodegroup/ManagedNodeGroup: CREATE_IN_PROGRESS – "Resource creation Initiated"
[15] 2021-12-02 12:59:20 [▶]  AWS::EKS::Nodegroup/ManagedNodeGroup: CREATE_IN_PROGRESS
[15] 2021-12-02 12:59:20 [▶]  AWS::IAM::Role/NodeInstanceRole: CREATE_COMPLETE
[15] 2021-12-02 12:59:20 [▶]  AWS::EC2::LaunchTemplate/LaunchTemplate: CREATE_COMPLETE
[15] 2021-12-02 12:59:20 [▶]  AWS::EC2::LaunchTemplate/LaunchTemplate: CREATE_IN_PROGRESS – "Resource creation Initiated"
[15] 2021-12-02 12:59:20 [▶]  AWS::IAM::Role/NodeInstanceRole: CREATE_IN_PROGRESS – "Resource creation Initiated"
[15] 2021-12-02 12:59:20 [▶]  AWS::IAM::Role/NodeInstanceRole: CREATE_IN_PROGRESS
[15] 2021-12-02 12:59:20 [▶]  AWS::EC2::LaunchTemplate/LaunchTemplate: CREATE_IN_PROGRESS
[15] 2021-12-02 12:59:20 [▶]  AWS::CloudFormation::Stack/eksctl-it-managed-wonderful-creature-1638447107-nodegroup-ubuntu: CREATE_IN_PROGRESS – "User Initiated"
[15] Error: failed to create nodegroups for cluster "it-managed-wonderful-creature-1638447107"
[15] 2021-12-02 12:59:20 [▶]  failed task: create managed nodegroup "ubuntu" (will continue until other parallel tasks are completed)
[15] 2021-12-02 12:59:20 [▶]  failed task: 1 task: { create managed nodegroup "ubuntu" } (will continue until other parallel tasks are completed)
[15] 2021-12-02 12:59:20 [▶]  failed task: 1 task: { 1 task: { create managed nodegroup "ubuntu" } } (will not run other sequential tasks)
[15] 2021-12-02 12:59:20 [ℹ]  1 error(s) occurred and nodegroups haven't been created properly, you may wish to check CloudFormation console
[15] 2021-12-02 12:59:20 [ℹ]  to cleanup resources, run 'eksctl delete nodegroup --region=us-west-2 --cluster=it-managed-wonderful-creature-1638447107 --name=<name>' for each of the failed nodegroup
[15] 2021-12-02 12:59:20 [✖]  waiting for CloudFormation stack "eksctl-it-managed-wonderful-creature-1638447107-nodegroup-ubuntu": RequestCanceled: waiter context canceled
[15] caused by: context deadline exceeded
[15] 
[15] • Failure [1507.707 seconds]
[15] (Integration) Create Managed Nodegroups
[15] /__w/eksctl-ci/eksctl-ci/eksctl/integration/tests/managed/managed_nodegroup_test.go:58
[15]   Bottlerocket and Ubuntu support
[15]   /__w/eksctl-ci/eksctl-ci/eksctl/integration/tests/managed/managed_nodegroup_test.go:94
[15]     Ubuntu [It]
[15]     /__w/eksctl-ci/eksctl-ci/eksctl/integration/tests/managed/managed_nodegroup_test.go:123
[15] 
[15]     Expected '../../../eksctl "create" "nodegroup" "--config-file" "-" "--verbose" "4"' to succeed, got return code 1
[15] 
[15]     /__w/eksctl-ci/eksctl-ci/eksctl/integration/tests/managed/managed_nodegroup_test.go:106
[15] ------------------------------

[Skarlso]

I wouldn't call this flaky. It consistently fails. :D

[nikimanoledaki]

Integration tests #130
1536:2021-12-03T10:54:26.7075082Z [16] 2021-12-03 10:54:26 [✖] AWS::EKS::Nodegroup/ManagedNodeGroup: CREATE_FAILED – "Nodegroup ubuntu failed to stabilize: [{Code: NodeCreationFailure,Message: Instances failed to join the kubernetes cluster,ResourceIds: [i-0850e58c5992fd67c, i-0dcb589ebafc92381]}]"

This suite started failing with this error in Integration tests #127.

[Skarlso]

Another run https://github.com/weaveworks/eksctl-ci/runs/4406500167?check_suite_focus=true

[aclevername]

I've ssh'd onto the nodegroup while its failing:

no kubelet logs:

ls -lha /var/log/
total 376K
drwxr-xr-x  10 root root            4.0K Dec  6 10:02 .
drwxr-xr-x  13 root root            4.0K Dec  1 06:20 ..
-rw-r--r--   1 root root             16K Dec  1 06:31 alternatives.log
drwx------   3 root root            4.0K Dec  6 10:02 amazon
drwxr-xr-x   2 root root            4.0K Dec  1 06:31 apt
-rw-rw----   1 root utmp               0 Dec  1 06:20 btmp
-rw-r-----   1 root adm             5.4K Dec  6 10:02 cloud-init-output.log
-rw-r--r--   1 root adm             140K Dec  6 10:02 cloud-init.log
drwxr-xr-x   2 root root            4.0K Dec  6 10:02 containers
drwxr-xr-x   2 root root            4.0K Aug  4 01:00 dist-upgrade
-rw-r--r--   1 root root            154K Dec  1 06:31 dpkg.log
-rw-r--r--   1 root root            2.5K Dec  1 06:31 fontconfig.log
drwxr-sr-x+  3 root systemd-journal 4.0K Dec  6 10:01 journal
-rw-rw-r--   1 root utmp            286K Dec  6 10:03 lastlog
drwxr-xr-x   2 root root            4.0K Dec  6 10:02 pods
drwx------   2 root root            4.0K Dec  6 10:01 private
drwxr-x---   2 root adm             4.0K Dec  6 10:02 unattended-upgrades
-rw-rw-r--   1 root utmp            2.7K Dec  6 10:03 wtmp

The kubelet binary does exist:

ls -lha /snap/bin/
total 8.0K
drwxr-xr-x 2 root root 4.0K Dec  6 10:02 .
drwxr-xr-x 8 root root 4.0K Dec  1 06:31 ..
lrwxrwxrwx 1 root root   13 Dec  1 06:31 amazon-ssm-agent.ssm-cli -> /usr/bin/snap
lrwxrwxrwx 1 root root   19 Dec  6 10:02 kubectl -> kubectl-eks.kubectl
lrwxrwxrwx 1 root root   13 Dec  1 06:31 kubectl-eks.kubectl -> /usr/bin/snap
lrwxrwxrwx 1 root root   19 Dec  6 10:02 kubelet -> kubelet-eks.kubelet
lrwxrwxrwx 1 root root   13 Dec  1 06:31 kubelet-eks.kubelet -> /usr/bin/snap
lrwxrwxrwx 1 root root   24 Dec  1 06:31 ssm-cli -> amazon-ssm-agent.ssm-cli

Kubectl is correctly authenticated and can successfully talk to the control plane:

kubectl get pods
No resources found in default namespace.
ubuntu@ip-192-168-30-94:~$ kubectl get pods -A
NAMESPACE     NAME                       READY   STATUS    RESTARTS   AGE
kube-system   coredns-86d9946576-9gnjl   0/1     Pending   0          36m
kube-system   coredns-86d9946576-t27lx   0/1     Pending   0          36m

docker is running fine:

docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

It appears to me the kubelet is just not running:

ps aux | grep kub
ubuntu      7798  0.0  0.0   5196   732 pts/0    S+   10:24   0:00 grep --color=auto kub

[aclevername]

bootstrap script failing:

cat /var/log/cloud-init-output.log
...

eksctl: running /etc/eks/bootstrap
Aliasing EKS k8s snap commands
Added:
  - kubelet-eks.kubelet as kubelet
Added:
  - kubectl-eks.kubectl as kubectl
Stopping k8s daemons until configured
Stopped.
Cluster "kubernetes" set.
The connection to the server localhost:8080 was refused - did you specify the right host or port?
Exited with error on line 483

bootstrap script:

  480
   481  # We want to find out kubelet version, but kubectl version is easier to parse
   482  # and it is installed at the same version as kubelet from snap in live build.
   483  kubernetes_version=$(/snap/bin/kubectl version -o json | jq -r '.clientVersion | (.major + "." + .minor)')
   484

Looks like the bootstrap script fails to run kubectl version -o json, even though now it appears the nodegroup can communicate to the cluster:

/snap/bin/kubectl version -o json | jq -r '.clientVersion | (.major + "." + .minor)'
1.20

When the bootstrap script was run it must not have had a working connection.

this is only for 1.20. The AMI being used is: ami-0a350c3585b5a9356