Solutions for "expect exactly one securityGroup tagged with kubernetes.io/cluster/<NAME>" don't work as expected

[yannrouillard]

Description

The recent addition in the FAQ about the expect exactly one securityGroup tagged with kubernetes.io/cluster/<NAME> is very much welcomed but it seems some of the solutions don't work properly.

Solution 2 & 3 don't work because aws-load-balancer-controller seems to not care about the actual value of the tag.
Whether it is set to empty string or something else, aws-load-balancer-controller still complains with expect exactly one securityGroup tagged with kubernetes.io/cluster/...
Also Solution 3 might not work in the long run as EKS might put back the tag upon platform update.

Solution 1 doesn't work out of the box as creating a new cluster with the given setup results in the following errors:

Error: One of ['cidr_blocks', 'ipv6_cidr_blocks', 'self', 'source_security_group_id', 'prefix_list_ids'] must be set to create an AWS Security Group Rule

This is caused by cluster_security_group_rules referencing the node_security_group being always added to the cluster security group despite node_security_group_id being set to null.

This can be solved by either providing our own node_security_group_id or disabling the creation or the cluster_security_group, (create_cluster_security_group = false) but it causes the replacement of the EKS cluster for an existing one (and a priori, a custom cluster security group should be provided in that case)

The solution 4. works fine.

• ✋ I have searched the open/closed issues and my issue is not listed.
  (It's in fact linked to Multiple security groups are created with the kubernetes.io/cluster tag #1986 which is already closed)

Expected behavior

The documentation should probably be updated to remove 2 & 3 solutions.

For solution 1. it is unclear to me if the module should be fixed to support having no node_security_group set, or if the doc should be updated to indicate that a custom node security group should be created.

[ogusak]

I stumbled over this as well when using solution 1, though the error in my case was a bit different:

│ Error: waiting for Security Group (sg-<>) Rule (sgrule-<>) create: couldn't find resource
│ 
│   with module.eks[0].aws_security_group_rule.cluster["egress_nodes_443"],
│   on .terraform/modules/eks/main.tf line 174, in resource "aws_security_group_rule" "cluster":
│  174: resource "aws_security_group_rule" "cluster" {

However, solution 2 worked for me with a slight modification: use null as the tag's value instead of "" (as documented in FAQ) i.e.,:

  node_security_group_tags = {
    "kubernetes.io/cluster/${local.name}" = null
  }

This was also suggested in the related issue

[DodgeCamaro]

This solution doesn't work. After migration from v17 to v18. I've got this error
Error: One of ['cidr_blocks', 'ipv6_cidr_blocks', 'self', 'source_security_group_id', 'prefix_list_ids'] must be set to create an AWS Security Group Rule

I've used the first solution
create_node_security_group = false # default is true
attach_cluster_primary_security_group = true # default is false

[antonbabenko]

This issue has been resolved in version 18.30.2 🎉

[fatoyeus]

This issue has been resolved in version 18.30.2 🎉

Using version 18.30.2 still actively producing this error

[govardha]

This causes a scenario where the module goes into a never ending loop. When I revert back to 18.30, I am able to create a EKS managed node group without an issue.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.