AWSManagedControlPlane can only install one EKS Addon at a time

[Jacobious52]

/kind bug

It looks like the Add-on manager has a bug where it will only install the last add-on you specific in the config spec of a AWSManagedControlPlane. This means you cannot use more than one EKS Addon at a time currently.

What steps did you take and what happened:

• Create an AWSManagedControlPlane plan with all 3 EKS addons

  addons:
  - name: "kube-proxy"
    version: "v1.20.4-eksbuild.2"
    conflictResolution: "overwrite"
  - name: "vpc-cni"
    version: "v1.8.0-eksbuild.1"
    conflictResolution: "overwrite"
  - name: "coredns"
    version: "v1.8.3-eksbuild.1"
    conflictResolution: "overwrite"

• Wait for reconcile. Check EKS console for add ons

- Note how only one is installed.

• Reorder the same add ons such that a different one is at the bottom. In this case vpc-cni

  - name: "kube-proxy"
    version: "v1.20.4-eksbuild.2"
    conflictResolution: "overwrite"
  - name: "coredns"
    version: "v1.8.3-eksbuild.1"
    conflictResolution: "overwrite"
  - name: "vpc-cni"
    version: "v1.8.0-eksbuild.1"
    conflictResolution: "overwrite"

• Check the console again to find that coredns has been uninstalled, and vpc-cni has replaced it

• You can repeat this in any configuration and get the same result for all add ons.

If you check the spec in cluster, all are in the spec, only the last one is in the status as active, and the EKSAddonsConfigured condition is complete.

  Addons:
    Conflict Resolution:    overwrite
    Name:                   kube-proxy
    Version:                v1.20.4-eksbuild.2
    Conflict Resolution:    overwrite
    Name:                   coredns
    Version:                v1.8.3-eksbuild.1
    Conflict Resolution:    overwrite
    Name:                   vpc-cni
    Version:                v1.8.0-eksbuild.1

Status:
  Addons:
    Arn:          <arn>/clusters_lab1-control-plane/vpc-cni/debe74d5-c83d-5712-016e-74c85d10eb05
    Created At:   2021-11-04T06:38:27Z
    Modified At:  2021-11-04T06:38:58Z
    Name:         vpc-cni
    Status:       ACTIVE
    Version:      v1.8.0-eksbuild.1
  Conditions:
    Type:                          EKSAddonsConfigured
    Last Transition Time:          2021-11-03T05:41:17Z

What did you expect to happen:

To be able to install all 3 EKS add ons.

Anything else you would like to add:

I suspect there is a nasty loop/iterator bug somewhere in

cluster-api-provider-aws/pkg/eks/addons/plan.go

Line 48 in 4274a5a

func (a *plan) Create(ctx context.Context) ([]planner.Procedure, error) {

As looking at the logs with -v=3 it shows that going into the function it expects 3 desired add ons, and 1 installed. But the number of procs that come out is 0.

I1104 06:30:13.657611       1 addons.go:34]  "msg"="Reconciling EKS addons"  
I1104 06:30:13.657736       1 addons.go:174]  "msg"="getting list of eks addons"  
I1104 06:30:13.892422       1 addons.go:46]  "msg"="getting installed eks addons"  "cluster"="clusters_lab1-control-plane"
I1104 06:30:13.892494       1 addons.go:101]  "msg"="getting eks addons installed"  
I1104 06:30:14.124798       1 addons.go:122]  "msg"="describe output"  "output"={"AddonArn":"<arn>/coredns/34be74d1-c9d9-57fc-40d6-bf76af231405","AddonName":"coredns","AddonVersion":"v1.8.3-eksbuild.1","ClusterName":"clusters_lab1-control-plane","CreatedAt":"2021-11-04T06:29:44.175Z","Health":{"Issues":[]},"ModifiedAt":"2021-11-04T06:29:50.635Z","ServiceAccountRoleArn":null,"Status":"ACTIVE","Tags":{}}
I1104 06:30:14.127780       1 addons.go:62]  "msg"="creating eks addons plan"  "cluster"="clusters_lab1-control-plane" "numdesired"=3 "numinstalled"=1
I1104 06:30:14.130360       1 addons.go:69]  "msg"="computed EKS addons plan"  "numprocs"=0
I1104 06:30:14.133097       1 addons.go:83]  "msg"="getting installed eks addons to update status"  "cluster"="clusters_lab1-control-plane"
I1104 06:30:14.135049       1 addons.go:143]  "msg"="getting eks addons installed to create state"  
I1104 06:30:14.386266       1 addons.go:164]  "msg"="describe output"  "output"={"AddonArn":"<arn>/clusters_lab1-control-plane/coredns/34be74d1-c9d9-57fc-40d6-bf76af231405","AddonName":"coredns","AddonVersion":"v1.8.3-eksbuild.1","ClusterName":"clusters_lab1-control-plane","CreatedAt":"2021-11-04T06:29:44.175Z","Health":{"Issues":[]},"ModifiedAt":"2021-11-04T06:29:50.635Z","ServiceAccountRoleArn":null,"Status":"ACTIVE","Tags":{}}

Environment:

• Cluster-api-provider-aws version: 1.0.0
• Kubernetes version: (use kubectl version): N/A
• OS (e.g. from /etc/os-release): N/A

[richardcase]

Thanks @Jacobious52. Yeah i think you are right this is probably a loop bug.....may a implicit memory aliasing in a loop bug.

[richardcase]

/triage accepted

[richardcase]

/priority critical-urgent
/area provider/eks
/assign
/lifecycle active

[richardcase]

And looking back over the code all the test assume just installing 1 addon.....at the time there was only 1 addon available. I am updating the e2e as part of this as well.

[richardcase]

Going to cherrypick this back to 0.6 and 0.7.

[richardcase]

This is fixed in the 0.7 and 1.x release series. So closing

/close