[FEATURE] Provide example for additional security group to managed node group

[schwichti]

Is your feature request related to a problem? Please describe

I am struggeling for weeks to add security groups to a managed node group. I need firewall rules that allow incoming traffic on two ports. This is a follow-up of #219. Looking into the code the interplay of create_worker_security_group, create_launch_template and worker_additional_security_group_ids inside the managed_node_groups definition seems to be relevant, but it is not clear to me.

Describe the solution you'd like

Provide a working example to add a security group to managed nodes groups that allows traffic on certain ports.

[vara-bonthu]

Hi @schwichti

worker_additional_security_group_ids can be used only with Launch templates for Managed and Self-Managed Node groups. Try this example and let me know if you face any issues.

I used your old example with some modifications

# Your example code
resource "aws_security_group" "worker_security_group" {
  name        = "allow dns and minio"
  description = ""
  vpc_id      = module.vpc.vpc_id
}

# Your example code
resource "aws_security_group_rule" "allow_dns" {
  description              = "Allow pods DNS"
  type                     = "ingress"
  from_port                = 53
  to_port                  = 53
  protocol                 = 17
  security_group_id        = aws_security_group.worker_security_group.id
  cidr_blocks              = ["0.0.0.0/0"]
}

# Your example code
resource "aws_security_group_rule" "allow_minio" {
  description              = "Allow access to minio"
  type                     = "ingress"
  from_port                = 9000
  to_port                  = 9000
  protocol                 = "tcp"
  security_group_id        = aws_security_group.worker_security_group.id
  cidr_blocks              = ["0.0.0.0/0"]
}

# I added this additionally
module "aws-eks-accelerator-for-terraform" {
  source = "../../"

  tenant            = local.tenant
  environment       = local.environment
  zone              = local.zone
  terraform_version = local.terraform_version

  # EKS Cluster VPC and Subnet mandatory config
  vpc_id             = module.aws_vpc.vpc_id
  private_subnet_ids = module.aws_vpc.private_subnets

  # Attach additional security group ids to Worker Security group ID
  # worker_additional_security_group_ids enabled only when `create_launch_template  = true` for Mnaaged Nodegorups
  worker_additional_security_group_ids = [resource.aws_security_group.worker_security_group.id] 

  # EKS CONTROL PLANE VARIABLES
  kubernetes_version = local.kubernetes_version

  # EKS MANAGED NODE GROUPS with minimum config
  managed_node_groups = {
    mg_5 = {
      # 1> Node Group configuration - Part1
      node_group_name = "mg5"                     # Max 40 characters for node group name

      # Launch template configuration
      create_launch_template  = true              # false will use the default launch template
      launch_template_os      = "amazonlinux2eks" # amazonlinux2eks or bottlerocket

      # 2> Node Group scaling configuration
      desired_size    = 2
      max_size        = 2
      min_size        = 2
      max_unavailable = 1 

      # 3> Node Group compute configuration
      ami_type        = "AL2_x86_64" # AL2_x86_64, AL2_x86_64_GPU, AL2_ARM_64, CUSTOM
      capacity_type   = "ON_DEMAND"  # ON_DEMAND or SPOT
      instance_types  = ["m5.large"] # List of instances used only for SPOT type
      disk_size       = 50

      # 4> Node Group network configuration
      subnet_type = "private" # public or private - Default to Private
      subnet_ids  = []        # Defaults to private subnet-ids used by EKS Controle plane. Define your private/public subnets list with comma separated subnet_ids  = ['subnet1','subnet2','subnet3']
    }
  }
}

[askulkarni2]

Hi @schwichti, did ☝🏽 work for you?

[schwichti]

I will check as soon as possible.

[Xkynar]

@askulkarni2 I was also having this same issue and your code does add the security group but the InstanceGroup fails to join the cluster (the instance does join though). After searching around, I found this comment showing a faulty behavior when AMI is not defined:

When creating a Managed node group with a launch template, the behavior differs based on whether an AMI has been specified in the launch template or not.

When no AMI is present in the launch template (as is the case for you, if I'm reading your gist correctly), EKS will merge in a section of MIME multi-part user data to the user data contents you've passed in. The part EKS merges in will attempt to bootstrap your worker node as well. Since MIME multiparts are executed in order, this means your bootstrapping happens first and the EKS bootstrapping becomes a no-op.

As a result, your worker nodes don't have the required labels for EKS to associate them with a node group.

Changing:
ami_type = "AL2_x86_64"
to

ami_type        = "CUSTOM"
custom_ami_id   = data.aws_ami.lin_ami.id

fixed it for me. Just letting you know. Cheers!

[askulkarni2]

@Xkynar thanks so much for reporting this and for the suggested fix as well. @schwichti please give this a try when you get a chance and we will try this as well and document this.

[bryantbiggs]

closing for now with the provided information above - please feel free to comment back here if we need to re-open, thanks!