Conditions error on deleting AWSManagedControlPlane

[richardcase]

/kind bug
/area provider/eks
/help

What steps did you take and what happened:
When an EKS cluster (AWSManagedControlPlane) with managed node group (AWSManagedMachinePool) is deleted in the e2e tests the following error is seen:

E1229 22:54:05.485310       1 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="error deleting network for AWSManagedControlPlane eks-nodes-hxozyw/cluster-aoqabq-control-plane: error patching conditions: The condition \"ClusterSecurityGroupsReady\" was modified by a different process and this caused a merge/ChangeCondition conflict:   \u0026v1alpha3.Condition{\n  \tType:               \"ClusterSecurityGroupsReady\",\n  \tStatus:             \"False\",\n  \tSeverity:           \"Info\",\n- \tLastTransitionTime: v1.Time{Time: s\"2020-12-29 22:54:03 +0000 UTC\"},\n+ \tLastTransitionTime: v1.Time{Time: s\"2020-12-29 22:54:05 +0000 UTC\"},\n- \tReason:             \"Deleting\",\n+ \tReason:             \"Deleted\",\n  \tMessage:            \"\",\n  }\n" "controller"="awsmanagedcontrolplane" "name"="cluster-aoqabq-control-plane" "namespace"="eks-nodes-hxozyw"

This implies that 2 different manager processes are modifying the ClusterSecurityGroupsReady condition on the AWSManagedControlPlane instance.

What did you expect to happen:
I expect that this error not occur when deleting. If we are changing the ClusterSecurityGroupsReady condition on the AWSManagedControlPlane in 2 different managers then this needs to be changed. The only managed allowed to modify the AWSManagedControlPlane is the EKS control plane manager.

Logs from a e2e run with the error are here

Environment:

• Cluster-api-provider-aws version: 0.6.3

[k8s-ci-robot]

@richardcase:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/kind bug
/area provider/eks
/help

What steps did you take and what happened:
When an EKS cluster (AWSManagedControlPlane) with managed node group (AWSManagedMachinePool) is deleted in the e2e tests the following error is seen:

E1229 22:54:05.485310       1 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="error deleting network for AWSManagedControlPlane eks-nodes-hxozyw/cluster-aoqabq-control-plane: error patching conditions: The condition \"ClusterSecurityGroupsReady\" was modified by a different process and this caused a merge/ChangeCondition conflict:   \u0026v1alpha3.Condition{\n  \tType:               \"ClusterSecurityGroupsReady\",\n  \tStatus:             \"False\",\n  \tSeverity:           \"Info\",\n- \tLastTransitionTime: v1.Time{Time: s\"2020-12-29 22:54:03 +0000 UTC\"},\n+ \tLastTransitionTime: v1.Time{Time: s\"2020-12-29 22:54:05 +0000 UTC\"},\n- \tReason:             \"Deleting\",\n+ \tReason:             \"Deleted\",\n  \tMessage:            \"\",\n  }\n" "controller"="awsmanagedcontrolplane" "name"="cluster-aoqabq-control-plane" "namespace"="eks-nodes-hxozyw"

This implies that 2 different manager processes are modifying the ClusterSecurityGroupsReady condition on the AWSManagedControlPlane instance.

What did you expect to happen:
I expect that this error not occur when deleting. If we are changing the ClusterSecurityGroupsReady condition on the AWSManagedControlPlane in 2 different managers then this needs to be changed. The only managed allowed to modify the AWSManagedControlPlane is the EKS control plane manager.

Environment:

• Cluster-api-provider-aws version: 0.6.3

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[richardcase]

/milestone v0.7.x

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[richardcase]

/lifecycle frozen

[randomvariable]

Is this error still a thing?

[richardchen331]

This is still happening consistently when I looked at recent e2e runs at https://prow.k8s.io/job-history/gs/kubernetes-jenkins/pr-logs/directory/pull-cluster-api-provider-aws-e2e-eks

Example: https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api-provider-aws/3095/pull-cluster-api-provider-aws-e2e-eks/1486250029256544256/artifacts/clusters/bootstrap/controllers/capa-controller-manager/capa-controller-manager-66874f8d88-cvvnm/manager.log

I think I found the root cause of it and I'll submit a PR to propose a fix.

[richardcase]

We had this issue with the eks e2e in release-0.6 as well. Depending on the fix it may be worth us backporting this.

[richardchen331]

Here's the proposed fix #3157