no identity-based policy allows the elasticloadbalancing:AddTags

[dumlutimuralp]

Description

The current IAM policy published in the AWS Doc and AWS Load Balancer Controller GitHub page has incompatible definitions in it. This causes AWS Load Balancer Controller failing in provisioining an ELB due to some Addtag errors. Seems identical to this previous GitHub issue.

Versions

• Module Version [Required]:

• AWS Load Balancer controller 2.4.4 (also tested with 2.4.4)

• EKS v1.24

Reproduction Code [Required]

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: frontendnlb
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: external #AWS Load Balancer Controller fulfills services that has this annotation 
    service.beta.kubernetes.io/aws-load-balancer-name : mynlb #User defined name given to AWS Network Load Balancer
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing #Places the load balancer on public subnets
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip #The Pod IPs should be used as the target IPs (rather than the node IPs as was the case with Network Load Balancer in the previous section)
  labels:
    app: frontendnlb
spec:
  type: LoadBalancer #The type of service
  ports:
    - port: 80 #The port which the service is running on
      targetPort: 9000 #The port on the pod which is backing this service. If not specified, it is assumed to be the same as the service port.
      name: http
  selector:
    app: frontend #The service identifies all the pods which have this label and then automatically configures the IP addresses of those pods as endpoints of this service.
EOF

Expected behaviour

For AWS Load Balancer Controller to provision an NLB.

Actual behaviour

No NLBs are provisioned. And AWS Load Balancer Controller generates the following error.

{"level":"error","ts":1676040723.3378003,"logger":"controller.service","msg":"Reconciler error","name":"frontendnlb","namespace":"workshop","error":"AccessDenied: User: arn:aws:sts::xxxxxxxxxxxx:assumed-role/eksctl-mycluster-addon-iamserviceaccount-kub-Role1-ALZQNS3HQWFQ/1676040699752964147 is not authorized to perform: elasticloadbalancing:AddTags on resource: arn:aws:elasticloadbalancing:eu-west-2:578334270511:targetgroup/k8s-workshop-frontend-8a6a890206/* because no identity-based policy allows the elasticloadbalancing:AddTags action\n\tstatus code: 403, request id: 4ee09784-239c-4afb-94f1-958b50134d8c"}

[dmegyesi]

We just faced the exact same issue today, also on v2.4.4.

By removing this condition, the problem seems to be resolved: https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/v2.4.4/docs/install/iam_policy.json#L160-L165

[abierbaum]

I am running into this issue using AWS CDK to deploy an EKS stack with

albController: {
   version: eks.AlbControllerVersion.V2_4_1,
},

[sjones512]

duplicate of #2692?

[M00nF1sh]

it's a duplicate of #2692, which is caused by a IAM change in ELB for tag resource on creation. ELB team have already rollback their change.

We'll sync with ELB team to update the IAM permission needed and update our default IAM policy.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[YikaiHu]

For anyone who has encountered the same issue, we have found a solution in CDK.

Here is the solution,

The key to solving the problem is to explicitly declare the policy of the ALB controller and assign it when creating the cluster.

#2692 (comment)

[armujahid]

/remove-lifecycle rotten

I am facing this issue with aws-load-balancer-controller helm chart 1.4.8 v2.4.7 provisioned using v4.32.1 of https://github.com/aws-ia/terraform-aws-eks-blueprints

Ingress nginx failed to provision NLB because of this issue.

[justinlarose]

This is fixed with this PR - #3046

[johngmyers]

/close

[k8s-ci-robot]

@johngmyers: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.