Self-Managed Node Groups Not Joining EKS Cluster (CoreDNS 'DEGRADE' Error)

[eravindar12]

Description

Iam attempting to create a self managed node groups to launch EC2 instances using Amazon Linux 2023 EKS optimized AMI. However. I am encountering an issue where the node groups are not joining the cluster, which is resulting in a 'DEGRADE' error for CoreDNS.

When I use the same Terraform code and eks module to create an EKS cluster with managed node groups, it works perfectly, with no issues related to node joining or CoreDNS.

This appears to be a bug. Is there a workaround to resolve this problem by modifying the Terraform code? Any suggestions or advice would be greatly appreciated.

Error: waiting for EKS Add-On (ecp-ppp-prod:coredns) create: timeout while waiting for state to become 'ACTIVE' (last state: 'DEGRADED', timeout: 20m0s)
│
│ with module.eks.aws_eks_addon.this["coredns"],
│ on .terraform/modules/eks/main.tf line 498, in resource "aws_eks_addon" "this":
│ 498: resource "aws_eks_addon" "this" {
│

Here is the Terraform modules and Reproduction Code

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.13"

  cluster_name                   = local.name
  cluster_version                = local.cluster_version 

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets  
  control_plane_subnet_ids = module.vpc.intra_subnets

  cluster_endpoint_private_access = true
  cluster_endpoint_public_access  = true

  enable_irsa = true

  enable_cluster_creator_admin_permissions = true

  # This will set the cluster authentication use API and CONFIG MAP, EKS will automatically create an access entry for the IAM role(s) used by managed nodegroup(s)
  authentication_mode = "API_AND_CONFIG_MAP" 


  # EKS Addons
  cluster_addons = {
    coredns    = {
      most_recent = true
    }

    eks-pod-identity-agent = {
      most_recent = true
    }

    kube-proxy = {
      most_recent = true
    }
    vpc-cni = {
      # Specify the VPC CNI addon should be deployed before compute to ensure
      # the addon is configured before data plane compute resources are created
      # See README for further details
      before_compute = true
      most_recent    = true # To ensure access to the latest settings provided
      configuration_values = jsonencode({
        env = {
          # Reference docs https://docs.aws.amazon.com/eks/latest/userguide/cni-increase-ip-addresses.html
          ENABLE_PREFIX_DELEGATION = "true"
          WARM_PREFIX_TARGET       = "1"
        }
      })
    }
  } 
 

  self_managed_node_groups = {
    # AL2023 node group utilizing new user data format which utilizes nodeadm
    # to join nodes to the cluster (instead of /etc/eks/bootstrap.sh)
    al2023_nodeadm = {

      name = "cis-self-mng"
      use_name_prefix = true
      launch_template_description     = "Self managed node group example launch template"

      # ebs_optimized     = true
      enable_monitoring = true

      subnet_ids = module.vpc.public_subnets

      min_size         = 1
      max_size         = 3
      desired_capacity = 1

      instance_type = "m6i.large"

      enable_bootstrap_user_data = true
      is_eks_managed_node_group = false

      ami_id = data.aws_ami.image_cis_eks.id 
 

      launch_template_name            = "amazon-eks-al2023-node-1.30"
      launch_template_use_name_prefix = true 
      launch_template_description     = "amazon-eks-al2023-node-1.30"

   
      // The following variables are necessary if you decide to use the module outside of the parent EKS module context.
      // Without it, the security groups of the nodes are empty and thus won't join the cluster.
      vpc_security_group_ids = [
        module.eks.cluster_primary_security_group_id,
        module.eks.cluster_security_group_id,
      ]
 
      # AL2023 node group utilizing new user data format which utilizes nodeadm
      # to join nodes to the cluster (instead of /etc/eks/bootstrap.sh)
      cloudinit_pre_nodeadm = [
        {
          content_type = "application/node.eks.aws"
          content      = <<-EOT
            ---
            apiVersion: node.eks.aws/v1alpha1
            kind: NodeConfig
            spec:
              featureGates: 
                InstanceIdNodeName: true
              cluster: ecp-ppp-prod
              apiServerEndpoint: https://xxxx.us-east-1.eks.amazonaws.com
              certificateAuthority: xxxxx
              cidr: 1xx.xx.0.0/16
              kubelet:
                config:
                  shutdownGracePeriod: 30s
                  featureGates:
                    DisableKubeletCloudCredentialProviders: true 
              config: |
              [plugins."io.containerd.grpc.v1.cri".containerd]
              discard_unpacked_layers = false            
          EOT
        }
      ]
    } 
  } 

  tags = local.tags 
} 

And i even tried with self managed module however getting the same issue.

module "self_managed_node_group" {
  source = "terraform-aws-modules/eks/aws//modules/self-managed-node-group"
  version = "20.13.1"

  name                = "cis-self-mng"
  cluster_name        = "xxx-ppp-prod"
  cluster_version     = "1.30"
  cluster_endpoint    = "https://xxx.gr7.us-east-1.eks.amazonaws.com"
  cluster_auth_base64 = "xxx"
  cluster_ip_family    = "ipv4"
  cluster_service_cidr = "xx.xx.0.10"

  subnet_ids = module.vpc.private_subnets


  ami_id   = data.aws_ami.image_cis_eks.id

  user_data_template_path = "${path.module}/modules/user_data/templates/al2023_custom.tpl"

  cloudinit_pre_nodeadm = [{
    content      = <<-EOT
      ---
      apiVersion: node.eks.aws/v1alpha1
      kind: NodeConfig
      spec:
        kubelet:
          config:
            shutdownGracePeriod: 30s
            featureGates:
              DisableKubeletCloudCredentialProviders: true
    EOT
    content_type = "application/node.eks.aws"
  }]

  cloudinit_post_nodeadm = [{
    content      = <<-EOT
      echo "All done"
    EOT
    content_type = "text/x-shellscript; charset=\"us-ascii\""
  }]

  // The following variables are necessary if you decide to use the module outside of the parent EKS module context.
  // Without it, the security groups of the nodes are empty and thus won't join the cluster.
  vpc_security_group_ids = [
    module.eks.cluster_primary_security_group_id,
    module.eks.cluster_security_group_id,
  ]

  min_size     = 1
  max_size     = 4
  desired_size = 1

  launch_template_name   = "cis-self-mng"
  instance_type          = "m5.2xlarge"

  tags = {
    Environment = "xxx-ppp-prod"
    Terraform   = "true"
  }
}

This is my VPC supporting TF module

################################################################################
# VPC supportings 
################################################################################

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0.0"

  name = local.name
  cidr = local.vpc_cidr

  azs             = ["us-east-1a", "us-east-1b", "us-east-1c"]
  private_subnets = ["10.xx.xx.0/24", "10.xx.xx.0/24", "10.xx.xx.0/24"]
  public_subnets  = ["10.xx.16.xx/26", "10.xx.xx.128/26", "10.xx.xx.1xx/26"]

  enable_nat_gateway     = true
  create_igw             = true

  single_nat_gateway     = false
  one_nat_gateway_per_az = false

  enable_dns_hostnames   = true
  enable_dns_support     = true

  enable_flow_log                      = true
  create_flow_log_cloudwatch_iam_role  = true
  create_flow_log_cloudwatch_log_group = true


  public_subnet_tags = {
    "kubernetes.io/role/elb"                        = 1
    "kubernetes.io/cluster/${var.environment_name}" = "owned"
  }

  private_subnet_tags = {
    "kubernetes.io/role/internal-elb" = 1
    # Tags subnets for Karpenter auto-discovery
    # "karpenter.sh/discovery" = local.name

    "kubernetes.io/cluster/${var.environment_name}" = "owned"

  }

  tags = local.tags

}

• Module version [Required]:

• Terraform version:
  < terraform {
  required_version = ">= 1.3"
  } >

• Provider version(s):
  <Execute: terraform providers -version
  required_providers {
  aws = {
  source = "hashicorp/aws"
  version = ">= 5.52"
  }>

Terminal Output Screenshot(s)

Additional context

[bryantbiggs]

This appears to be a bug

I would disagree and state this is a user configuration error

[eravindar12]

I would disagree and state this is a user configuration error

@bryantbiggs - I understand your perspective. Could you please provide more details on what specific configuration errors you believe might be causing this issue? I have reviewed all the VPC and EKS cluster network configurations, but any further suggestions or insights would be greatly appreciated.

[zack-is-cool]

I was running into this over the past few days trying bottlerocket nodes via self-managed-node-groups. It feels like platform is already deprecated. You need to specify ami_type the way the logic is written (also here and here) in the self-managed-node-group module - it will always pick AL2_x86_64 by default if not explicitly set - unless I'm misinterpreting this.

TLDR;
try setting ami_type to AL2023_x86_64_STANDARD in your node group map

[mossad-zika]

I would disagree and state this is a user configuration error

The default example from official documentation does not create a working EKS cluster, so agree or disagree, in order to provide a working module documentation must be updated regardless

[bryantbiggs]

I would disagree and state this is a user configuration error

The default example from official documentation does not create a working EKS cluster, so agree or disagree, in order to provide a working module documentation must be updated regardless

I'm sorry, what?

[mossad-zika]

I'm sorry, what?

Don't be, try to use module's documentation as is and you won't achieve creating of a working EKS cluster

[eravindar12]

I am encountering an error that I suspect is preventing nodes from joining the cluster. I would like to resolve this issue using the TF EKS module. Could you please advise if there are any specific input values I need to add explicitly to fix it?

[bryantbiggs]

I mean, thats just an example how how you could use the module - it has made up values so it won't work if you try to deploy it as is. For example, you would need to use your own values for these:

  vpc_id                   = "vpc-1234556abcdef"
  subnet_ids               = ["subnet-abcde012", "subnet-bcde012a", "subnet-fghi345a"]
  control_plane_subnet_ids = ["subnet-xyzde987", "subnet-slkjf456", "subnet-qeiru789"]

Also, you are referring to an EKS managed node group implementation and @eravindar12 is referring to use of self-managed node group. So I don't know that your comments are valid nor providing value in this context

[mossad-zika]

I mean, thats just an example how how you could use the module - it has made up values so it won't work if you try to deploy it as is.

This is obvious, I did create VPC and subnets

vpc.tf

resource "aws_vpc" "my_vpc" {
  cidr_block = "10.0.0.0/16"

  tags = {
    Name = "study-vpc"
  }
}

resource "aws_subnet" "my_subnet_1" {
  availability_zone = "us-east-1a"
  vpc_id            = aws_vpc.my_vpc.id
  cidr_block        = "10.0.1.0/24"

  tags = {
    Name = "study-subnet-1"
  }
}

resource "aws_subnet" "my_subnet_2" {
  availability_zone = "us-east-1b"
  vpc_id            = aws_vpc.my_vpc.id
  cidr_block        = "10.0.2.0/24"

  tags = {
    Name = "study-subnet-2"
  }
}

cluster.tf

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "20.13.1"

  cluster_name    = "study-cluster"
  cluster_version = "1.29"

  authentication_mode = "API"

  cluster_endpoint_public_access = true

  cluster_addons = {
    coredns = {
      most_recent = true
    }
    kube-proxy = {
      most_recent = true
    }
    vpc-cni = {
      most_recent = true
    }
  }

  vpc_id     = aws_vpc.my_vpc.id
#   control_plane_subnet_ids = [aws_subnet.my_subnet_1.id, aws_subnet.my_subnet_2.id]
  subnet_ids = [aws_subnet.my_subnet_1.id, aws_subnet.my_subnet_2.id]

  # EKS Managed Node Group(s)
  eks_managed_node_groups = {
    small = {
      subnet_ids = [aws_subnet.my_subnet_1.id, aws_subnet.my_subnet_2.id]
      min_size     = 1
      max_size     = 1
      desired_size = 1

      instance_types = ["t3.small"]
      capacity_type  = "SPOT"
    }
  }

  # Cluster access entry
  # To add the current caller identity as an administrator
  enable_cluster_creator_admin_permissions = true
}

node is failing to join the cluster

I don't believe that module has a value if I have to investigate why even basic example doesn't work

[zack-is-cool]

@bryantbiggs I believe if you were to create a cluster with just self-managed-node-groups using non AL2 AMIs you would be able to recreate the main issue of this thread.. I don't believe eks_managed_node_groups would have the same issue, but I haven't tried personally because ami_type is properly null'd. Basically when no nodes join the cluster the TF apply gets stuck when deploying the marketplace-addons because they never get healthy.

see my comment above for more details

If you wanted to recreate the issue that I was facing you could use our example root module here (from this specific sha) - we've since fixed this by specifying ami_type = "BOTTLEROCKET_x86_64" in our self managed node group settings.

replication -

note: you might want to set cluster_endpoint_public_access = true in fixtures.secure.tfvars to poke around

tofu init
tofu apply --var-file fixtures.common.tfvars --var-file fixtures.secure.tfvars --auto-approve

[mossad-zika]

anyway sorry guys I missed that the topic was about self-managed node group, I was trying to create aws-managed node group

just I think bad documentation is a common issue for this specific module

other Anton's modules I did use worked as a charm and have awesome docs

[bryantbiggs]

@mossad-zika that is not suitable VPC

In terms of self-managed node group with AL2023, it does work https://github.com/clowdhaus/eks-reference-architecture/blob/main/self-managed-node-group/eks_default.tf

Just validated it myself

[zack-is-cool]

@mossad-zika that is not suitable VPC

In terms of self-managed node group with AL2023, it does work clowdhaus/eks-reference-architecture@main/self-managed-node-group/eks_default.tf

Just validated it myself

Right, you're using ami_type = "AL2023_x86_64_STANDARD" - if you tried without setting ami_type and used platform instead, I think you'd run into this issue where your nodes wouldn't join the cluster.

It seems like ami_type is already required, whereas in most of the code (comments) it reads like platform should still work

[zack-is-cool]

From the PR
#3030 (comment)

The platform functionality is still preserved for backwards compatibility until its removed in the next major version release

I don't believe this to be the case, because ami_type always wins, and ami_type defaults to Amazon Linux 2 due to default vars. We just ran into this using bottlerocket nodes and only specifying platform