Nodegroup not deployed with no errors if stack is in ROLLBACK_COMPLETE

[hryamzik]

What were you trying to accomplish?

Redeploy nodegroup that failed to deploy earlier, cloud formation is in ROLLBACK_COMPLETE state.

How to reproduce it?

1. Create node group with restricted labels using conf:

  - name: test
    instanceType: m5.large
    minSize: 1
    maxSize: 2
    desiredCapacity: 2
    labels:
      role: test
      kubernetes.io/test: ok

2. It should fail with expected message: AWS::EKS::Nodegroup/ManagedNodeGroup: CREATE_FAILED – "Label cannot contain reserved prefixes [kubernetes.io/, k8s.io/, eks.amazonaws.com/]

3. Update config to fix labels:

  - name: test
    instanceType: m5.large
    minSize: 1
    maxSize: 2
    desiredCapacity: 2
    labels:
      role: test
      node-role.kubernetes.io/test: ok

What happened?

Node group wasn't updated. No errors thrown, see output below.

Logs

2021-07-22 13:00:13 [ℹ]  eksctl version 0.57.0
2021-07-22 13:00:13 [ℹ]  using region us-east-1
2021-07-22 13:00:24 [ℹ]  nodegroup "services" will use "" [AmazonLinux2/1.18]
2021-07-22 13:00:24 [ℹ]  nodegroup "test" will use "" [AmazonLinux2/1.18]
2021-07-22 13:00:24 [ℹ]  nodegroup "workers" will use "" [AmazonLinux2/1.18]
2021-07-22 13:00:31 [ℹ]  3 existing nodegroup(s) (services,test,workers) will be excluded
2021-07-22 13:00:33 [ℹ]  2 sequential tasks: { fix cluster compatibility, no tasks }
2021-07-22 13:00:33 [ℹ]  checking cluster stack for missing resources
2021-07-22 13:00:36 [ℹ]  cluster stack has all required resources
2021-07-22 13:00:36 [ℹ]  no tasks
2021-07-22 13:00:36 [✔]  created 0 nodegroup(s) in cluster "testcluster"
2021-07-22 13:00:36 [✔]  created 0 managed nodegroup(s) in cluster "testcluster"
2021-07-22 13:00:41 [ℹ]  checking security group configuration for all nodegroups
2021-07-22 13:00:41 [ℹ]  all nodegroups have up-to-date configuration

Anything else we need to know?

Versions

$ eksctl info
eksctl version: 0.57.0
kubectl version: v1.21.2
OS: darwin

[Callisto13]

Thanks for reporting @hryamzik 👍

Eksctl does not support updating nodegroup config, so from its perspective you did not ask for anything to be created: nodegroup test already existed, so eksctl did nothing. You can see this from the log line 3 existing nodegroup(s) (services,test,workers) will be excluded

You need to delete nodegroup test and create it again with the new config.

[hryamzik]

Thanks for response @Callisto13

Node group in is not listed in ROLLBACK_COMPLETE is not listed in eksctl get nodegroups. Cloudformation stack shows all resources are rolled back (destroyed) and the only thing that exists is the stack itself.

P.S.: actually if I can still use this nodegroup fo free there's another bug in kubectl not showing it and in EKS not allowing to create pods on it ;-)

[Callisto13]

Node group in is not listed in ROLLBACK_COMPLETE is not listed in eksctl get nodegroups. Cloudformation stack shows all resources are rolled back (destroyed) and the only thing that exists is the stack itself.

using eksctl to delete the nodegroup should remove that stack. eksctl looks at stacks to see what exists, so yeh it is annoying that technically nothing is there, but eksctl thinks there is. maybe there is something we can do to improve this 🤔

P.S.: actually if I can still use this nodegroup fo free there's another bug in kubectl not showing it and in EKS not allowing to create pods on it ;-)

🤔

[hryamzik]

I can confirm that eksctl is able to delete NG while it can't list it. So not listing it seem to be an issue to me as well. Can we reopen this issue?

[Callisto13]

Those calls are looking at different things, the create and delete are concerned with stacks and hit the CF api, the get looks at whatever is registered with the cluster.

I am away from now until week after next, but I will leave it with the team and they will think about what or if we can do anything here 👍

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.