New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Multiple security groups are created with the kubernetes.io/cluster tag #1986
Comments
hi @blakepettersson - the EKS managed node group sub-module security group does not have these tags currently |
@bryantbiggs so it seems 🤦. The issue in the case of EKS managed node groups seems to be that EKS autogenerates a security group that already has the |
Here's a minimal repro: locals {
vpc_id = "a vpc id"
aws_region = data.aws_region.current.name
current_account = data.aws_caller_identity.current.account_id
}
provider "aws" {
region = "eu-north-1"
}
data "aws_region" "current" {}
data "aws_caller_identity" "current" {}
data "aws_subnets" "all" {
filter {
name = "vpc-id"
values = [local.vpc_id]
}
}
module "eks" {
source = "terraform-aws-modules/eks/aws"
version = "18.10.1"
cluster_name = "test"
cluster_version = "1.21"
cluster_endpoint_private_access = true
cluster_endpoint_public_access = true
vpc_id = local.vpc_id
subnet_ids = data.aws_subnets.all.ids
eks_managed_node_groups = {
default = {
medium = {
instance_types = ["t3.medium"]
desired_size = 3
max_size = 10
min_size = 3
disk_type = "gp3"
}
}
}
enable_irsa = true
} Running Both of these security groups have the tag |
Modified from examples/complete removing all external modules and keeping 1 inline managed node group only. nodegroup configuration module "eks" {
source = "terraform-aws-modules/eks/aws"
version = "~> 18.20.5"
.
.
.
# EKS Managed Node Group(s)
eks_managed_node_group_defaults = {
ami_type = "AL2_x86_64"
disk_size = 10
instance_types = ["t2.small", "t3.small", "t3a.small"]
attach_cluster_primary_security_group = true
vpc_security_group_ids = [aws_security_group.additional.id]
}
eks_managed_node_groups = {
green = {
min_size = 1
max_size = 10
desired_size = 1
instance_types = ["t3.small"]
capacity_type = "SPOT"
labels = {
Environment = "test"
GithubRepo = "terraform-aws-eks"
GithubOrg = "terraform-aws-modules"
}
taints = {
# dedicated = {
# key = "dedicated"
# value = "gpuGroup"
# effect = "NO_SCHEDULE"
# }
}
update_config = {
max_unavailable_percentage = 50 # or set `max_unavailable`
}
tags = {
ExtraTag = "example"
}
}
} kubectl describe service
This workaround solves the problem. Pass it to the module "eks" # Temp workaround for bug : double owned tag
# https://github.com/terraform-aws-modules/terraform-aws-eks/issues/1810
node_security_group_tags = {
"kubernetes.io/cluster/${local.name}" = null
} Notice that taint configuration is also disabled since it caused rejection in scheduler with core-dns deployment.
|
terraform-aws-eks/node_groups.tf Line 160 in dc8a6ee
|
I'm also experiencing an issue with this after installing the aws-load-balancer-controller. I get the following error from there.
After checking the console. i can see i have 2 sgs with the same tag Im also using eks managed node groups. |
@kaykhancheckpoint check the last couple posts for the workaround from leiarenee |
This workaround solved it for me. (#1810) (Repeating for brevity) Module 'eks' node_security_group_tags = {
"kubernetes.io/cluster/${local.name}" = null
} |
@diginc @leiarenee yup, thank you |
This issue has been automatically marked as stale because it has been open 30 days |
This workaround did not work for us. |
|
apologies - does that mean that it does work for your case or not? |
It means my ops engineer told me he did it and then realized that you would actually posted a slightly different command, So we never actually tried it. We are used to different work around though (In this case we need to set a kernel parameter, so are using a start up container that’s privileged to do it instead.)
… On Jun 3, 2022, at 20:22, Bryant Biggs ***@***.***> wrote:
I can confirm we've tried that. It did not work.
apologies - does that mean that it does work for your case or not?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.
|
In my Karpenter Example, this doesn't work. And these stop create security group doesn't work so: create_node_security_group = false
create_cluster_security_group = false
I am still trying.... |
In this offical doc: AWS says : And in my |
After digging in to this, here is what I found: If you use node_security_group_tags = {
"kubernetes.io/cluster/${local.name}" = null
} Why - the primary security group created by the EKS service automatically tags the cluster's primary security group. I tried to use cluster_tags = {
"kubernetes.io/cluster/${local.name}" = null
} To "hide" this tag and remove the conflict, but this route does not work. Its safe to say, if you want to attach the cluster's primary security group to your nodes, you need to disable the tag on the node shared security group. Its not a matter of just having multiple security groups with the tag key:value combo, its only a concern when multiple security groups with the tag key:value combo are ATTACHED to your node(s). You can refer to this example which did not encounter any issues provisioning the ALB controller chart https://github.com/clowdhaus/eks-reference-architecture/tree/main/karpenter/us-east-1 Let me know if there is anything else I missed or any other questions pertaining to this issue. |
If I understand you correctly @bryantbiggs , EKS automatically tags the clusters primary security group with
does not work. An alternative would be to set Do I understand this correctly? |
No - the security group and tag in question only come into play when they are attached to nodes. So you can have 40+ security groups in your account with this tag without issue, but as soon as you attach more than one of those security groups to node(s) in your cluster, then its a conflict |
That is very interesting. |
Hi, @kaykhancheckpoint how did you actually deploy and enabled the "load balancer controller"? |
Running into what appears to be a similar issue, I think I've been able to work around it by using The problem appears to be that there are more than 1 security group that are tagged with This may be a clue when running an apply:
Note that existing Karpenter instances need to be replaced for this tag change to take effect. |
This issue has been automatically marked as stale because it has been open 30 days |
This issue was automatically closed because of stale in 10 days |
I think this issue has probably been closed prematurely; I just hit it as well with this template. |
+1 Stepped over this as well with 18.26.6, using Results was the inability to have the ELB working, having two security groups with Using |
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further. |
Description
The issue is exactly the same as #1810, but for EKS managed node groups, with the same workaround (and presumably the same fix)
Versions
18.10.1
Terraform version:
Reproduction Code [Required]
Steps to reproduce the behaviour:
Possible workaround:
The text was updated successfully, but these errors were encountered: