Terraform & Helm & aws-load-balancer-controller : ALB won't be created

[tal-rofe]

I try to deploy a complete cluster using EKS. When I run terraform apply the command completes successfully. I can see that my configured Pods, Deployments, Services were created. I also see the required software for aws-load-balancer-controller was created.
I have a full code for my deployment here to check:
https://github.com/tal-rofe/terraform-issue

Expected outcome
Amazon ALB should be created

Environment

AWS Load Balancer controller version "1.4.7"
Kubernetes version "2.16.1"
Using EKS (yes/no), if so version? 1.24

Additional Context:
When I go to EC2 load balancers the list is empty. ALB was not created..

All created pods seem fine:

Deployments

Services

Service Account

So everything seems fine, except for the ALB not created.

I guess it has issues with permissions for creating load balancer? But I provided my load balancer permissions:

# https://docs.aws.amazon.com/eks/latest/userguide/aws-load-balancer-controller.html
# https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.4.4/docs/install/iam_policy.json
resource "aws_iam_policy" "alb_controller_policy" {
  name        = "alb-controller-policy"
  description = "Policy for ALB controller"
  policy      = file("./resources/node-ingress-policy.json")
  path        = "/"

  tags = merge(
    var.common_tags,
    {
      Stack = "backend"
      Name  = "${var.project}-alb-controller-policy",
    }
  )
}

data "aws_iam_policy_document" "irsa_alb_controller_trust_policy_doc" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRoleWithWebIdentity"]
    principals {
      type        = "Federated"
      identifiers = [data.aws_iam_openid_connect_provider.main.arn]
    }
    condition {
      test     = "StringEquals"
      variable = "${data.aws_iam_openid_connect_provider.main.url}:sub"
      values = [
        "system:serviceaccount:kube-system:aws-load-balancer-controller"
      ]
    }
    condition {
      test     = "StringEquals"
      variable = "${data.aws_iam_openid_connect_provider.main.url}:aud"
      values = [
        "sts.amazonaws.com"
      ]
    }
  }
}

resource "aws_iam_role" "alb_controller_role" {
  name               = "alb-controller-role"
  assume_role_policy = data.aws_iam_policy_document.irsa_alb_controller_trust_policy_doc.json
  path               = "/"

  tags = merge(
    var.common_tags,
    {
      Stack = "backend"
      Name  = "${var.project}-alb-controller-role",
    }
  )
}

resource "aws_iam_role_policy_attachment" "alb_policy_attachment" {
  role       = aws_iam_role.alb_controller_role.name
  policy_arn = aws_iam_policy.alb_controller_policy.arn
}

If I run this command: kubectl logs -n kube-system deployment.apps/aws-load-balancer-controller I get errors in the logs:

{"level":"error","ts":1674494743.514425,"logger":"controller.ingress","msg":"Reconciler error","name":"api-ingress","namespace":"default","error":"ingress: default/api-ingress: AccessDeniedException: User: arn:aws:sts::lll:assumed-role/first-eks-node-group-llllllll/i-llllll is not authorized to perform: acm:ListCertificates because no identity-based policy allows the acm:ListCertificates action"}

But I did provide this permissions in the policy json: ./terraform/k8s/resources/node-ingress-policy.json

[kishorj]

@tal-rofe, the error indicates IRSA is not setup properly for your controller pods. Could you ensure?

• the sa has appropriate role annotation eks.amazonaws.com/role-arn
• serviceaccount name matches the one specified in the controller deployment
• controller pod has the aws-iam-token volume and the corresponding mount?

[tal-rofe]

@kishorj

thanks for response. Issue was - missing serviceAccount.name in helm release for ALB controller