Disassociate secondary CIDR after subnets are deleted

[Ankitasw]

What type of PR is this?
/kind bug

What this PR does / why we need it:
This PR fixes following:

• Order of disassociation of secondary CIDR post subnet deletion, otherwise the CIDR wont get deleted if subnet is still in use.
• Adds ec2:AssociateVpcCidrBlock and ec2:DisassociateVpcCidrBlock permissions to EKS controller policy.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #3277
Fixes #3273

Checklist:

• squashed commits
• includes documentation
• adds unit tests
• adds or updates e2e tests

Release note:

- Disassociate secondary CIDR after subnets are deleted.
- Adds ec2:AssociateVpcCidrBlock and ec2:DisassociateVpcCidrBlock permissions to EKS controller policy.

[k8s-ci-robot]

@Ankitasw: This issue is currently awaiting triage.

If CAPA/CAPI contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Ankitasw]

/test pull-cluster-api-provider-aws-e2e-eks

[sedefsavas]

LGTM.

Wondering if you confirmed without this fix, the new e2e changes fail? So that we will know that the test is actually catching the current bug.

[Ankitasw]

Wondering if you confirmed without this fix, the new e2e changes fail? So that we will know that the test is actually catching the current bug.

I will confirm it locally.

[Ankitasw]

/test pull-cluster-api-provider-aws-e2e-eks

[Ankitasw]

/hold
until tests passes

[Ankitasw]

@sedefsavas confirmed below error with existing code if secondary CIDR block is provided:

Reconciler error" "error"="InvalidCidrBlock.InUse: The vpc vpc-0232ce4e5a6e2c91a currently has a subnet within CIDR block 100.64.0.0/16\n\tstatus code: 400, request id: 3daf9588-4c1c-490e-8ec6-15a0d700b5fd" "name"="cluster-onrwy3-control-plane" "namespace"="eks-nodes-bmzl58" "reconciler group"="controlplane.cluster.x-k8s.io" "reconciler kind"="AWSManagedControlPlane"

Also added necessary permissions to controllers policy.

[Ankitasw]

/unhold

[Ankitasw]

This PR is ready for review.

[sedefsavas]

Since Secondary CIDR is only supported in EKS, we should not add the permissions for associate/disassociate VPC CIDRs when EKS is disabled in the clusterawsadm config file.

[Ankitasw]

@sedefsavas will make the desired change.

[Ankitasw]

/test pull-cluster-api-provider-aws-e2e-eks

[Ankitasw]

Since Secondary CIDR is only supported in EKS, we should not add the permissions for associate/disassociate VPC CIDRs when EKS is disabled in the clusterawsadm config file.

Made the changes, ready for review again.

[sedefsavas]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sedefsavas

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [sedefsavas]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Ankitasw]

/retest

[Ankitasw]

@sedefsavas the test was failing as new fixture with_s3_bucket.yaml was added in main branch and didn't have the vpc cidr permissions, added those permissions now.

[sedefsavas]

/lgtm