Migrate to AWS SDKv2

[torredil]

Is this a bug fix or adding new feature?

Dependency migration

What is this PR about? / Why do we need it?

In alignment with our SDKs and Tools Maintenance Policy, AWS SDK for Go (v1) will enter maintenance mode on July 31, 2024 and reach end-of-support on July 31, 2025. This PR implements the necessary changes to migrate to AWS SDK for Go v2.

Fixes #1684

Additional Information

https://aws.github.io/aws-sdk-go-v2/docs/migrating/

[torredil]

/hold

[github-actions]

Code Coverage Diff

File	Old Coverage	New Coverage	Delta
github.com/kubernetes-sigs/aws-ebs-csi-driver/pkg/cloud/cloud.go	83.3%	84.4%	1.1
github.com/kubernetes-sigs/aws-ebs-csi-driver/pkg/cloud/metadata.go	86.4%	96.7%	10.3
github.com/kubernetes-sigs/aws-ebs-csi-driver/pkg/cloud/metadata_ec2.go	84.1%	94.4%	10.4
github.com/kubernetes-sigs/aws-ebs-csi-driver/pkg/cloud/metadata_k8s.go	77.1%	80.0%	2.9
github.com/kubernetes-sigs/aws-ebs-csi-driver/pkg/driver/controller.go	83.5%	83.6%	0.1
github.com/kubernetes-sigs/aws-ebs-csi-driver/pkg/driver/node.go	80.8%	80.6%	-0.2
github.com/kubernetes-sigs/aws-ebs-csi-driver/pkg/util/util.go	86.4%	80.0%	-6.4

[torredil]

/retest

[AndrewSirenko]

Overall thank you so much for doing a great job on this herculean migration!

I still need to review a few more files, and run some more manual tests, but overall this looks great so far.

Have a gift of a few questions and nits to for now for your efforts 😅.

Files left to review:

cloud_test.go
handlers.go
metadata_test.go
controller.go
controller_test.go

[AndrewSirenko]

/retest

[torredil]

Fixed CreateSnapshot unit test - turns out, tc.expInput is created in the test function while the actual input passed to CreateSnapshot is created inside the func, even though the contents of the structs are the same they are separate instances with different memory addresses and hence why the equality check failed.

Fixed now + the go sum merge conflict. No issues spotted in manual testing + external e2e tests are all green.

The last remaining action item is RCAing why make verify hangs here in CI environment, but succeeds locally:

$ make verify
go vet $(go list ./...)
INFO Execution took 960.184531ms
./hack/verify-update.sh
All verifications passed!

[torredil]

Upgrading linter for make verify in #1971

[torredil]

Logs captured during manual testing for new middleware handler implementation used to record AWS API metrics:

Handling connection for 3301
# HELP cloudprovider_aws_api_request_duration_seconds [ALPHA] ebs_csi_aws_com metric
# TYPE cloudprovider_aws_api_request_duration_seconds histogram
cloudprovider_aws_api_request_duration_seconds_bucket{request="AttachVolume",le="0.005"} 0
cloudprovider_aws_api_request_duration_seconds_bucket{request="AttachVolume",le="0.01"} 0
cloudprovider_aws_api_request_duration_seconds_bucket{request="AttachVolume",le="0.025"} 0
cloudprovider_aws_api_request_duration_seconds_bucket{request="AttachVolume",le="0.05"} 0
cloudprovider_aws_api_request_duration_seconds_bucket{request="AttachVolume",le="0.1"} 0
cloudprovider_aws_api_request_duration_seconds_bucket{request="AttachVolume",le="0.25"} 0
cloudprovider_aws_api_request_duration_seconds_bucket{request="AttachVolume",le="0.5"} 1
cloudprovider_aws_api_request_duration_seconds_bucket{request="AttachVolume",le="1"} 1
cloudprovider_aws_api_request_duration_seconds_bucket{request="AttachVolume",le="2.5"} 1
cloudprovider_aws_api_request_duration_seconds_bucket{request="AttachVolume",le="5"} 1
cloudprovider_aws_api_request_duration_seconds_bucket{request="AttachVolume",le="10"} 1
cloudprovider_aws_api_request_duration_seconds_bucket{request="AttachVolume",le="+Inf"} 1
cloudprovider_aws_api_request_duration_seconds_sum{request="AttachVolume"} 0.386986709
cloudprovider_aws_api_request_duration_seconds_count{request="AttachVolume"} 1
cloudprovider_aws_api_request_duration_seconds_bucket{request="CreateVolume",le="0.005"} 0
cloudprovider_aws_api_request_duration_seconds_bucket{request="CreateVolume",le="0.01"} 0
cloudprovider_aws_api_request_duration_seconds_bucket{request="CreateVolume",le="0.025"} 0
cloudprovider_aws_api_request_duration_seconds_bucket{request="CreateVolume",le="0.05"} 0
cloudprovider_aws_api_request_duration_seconds_bucket{request="CreateVolume",le="0.1"} 0
cloudprovider_aws_api_request_duration_seconds_bucket{request="CreateVolume",le="0.25"} 1
cloudprovider_aws_api_request_duration_seconds_bucket{request="CreateVolume",le="0.5"} 1
cloudprovider_aws_api_request_duration_seconds_bucket{request="CreateVolume",le="1"} 1
cloudprovider_aws_api_request_duration_seconds_bucket{request="CreateVolume",le="2.5"} 1
cloudprovider_aws_api_request_duration_seconds_bucket{request="CreateVolume",le="5"} 1
cloudprovider_aws_api_request_duration_seconds_bucket{request="CreateVolume",le="10"} 1
cloudprovider_aws_api_request_duration_seconds_bucket{request="CreateVolume",le="+Inf"} 1
cloudprovider_aws_api_request_duration_seconds_sum{request="CreateVolume"} 0.244747201
cloudprovider_aws_api_request_duration_seconds_count{request="CreateVolume"} 1
cloudprovider_aws_api_request_duration_seconds_bucket{request="DescribeInstances",le="0.005"} 0
cloudprovider_aws_api_request_duration_seconds_bucket{request="DescribeInstances",le="0.01"} 0
cloudprovider_aws_api_request_duration_seconds_bucket{request="DescribeInstances",le="0.025"} 0
cloudprovider_aws_api_request_duration_seconds_bucket{request="DescribeInstances",le="0.05"} 0
cloudprovider_aws_api_request_duration_seconds_bucket{request="DescribeInstances",le="0.1"} 1
cloudprovider_aws_api_request_duration_seconds_bucket{request="DescribeInstances",le="0.25"} 1
cloudprovider_aws_api_request_duration_seconds_bucket{request="DescribeInstances",le="0.5"} 1
cloudprovider_aws_api_request_duration_seconds_bucket{request="DescribeInstances",le="1"} 1
cloudprovider_aws_api_request_duration_seconds_bucket{request="DescribeInstances",le="2.5"} 1
cloudprovider_aws_api_request_duration_seconds_bucket{request="DescribeInstances",le="5"} 1
cloudprovider_aws_api_request_duration_seconds_bucket{request="DescribeInstances",le="10"} 1
cloudprovider_aws_api_request_duration_seconds_bucket{request="DescribeInstances",le="+Inf"} 1
cloudprovider_aws_api_request_duration_seconds_sum{request="DescribeInstances"} 0.067674695
cloudprovider_aws_api_request_duration_seconds_count{request="DescribeInstances"} 1
cloudprovider_aws_api_request_duration_seconds_bucket{request="DescribeVolumes",le="0.005"} 0
cloudprovider_aws_api_request_duration_seconds_bucket{request="DescribeVolumes",le="0.01"} 0

/unhold

[ConnorJC3]

Initial review - very little manual testing

Intend to do the bulk of manual testing early next week, after which I will publish a final review

(expect very little additional comments in the final review unless I catch a bug with the manual testing)

[ConnorJC3]

/lgtm

I tried my hardest to break it, and I failed.

[AndrewSirenko]

Great work on this tricky migration. Thank you for delivering results.

[torredil]

/approve
cc: @ConnorJC3 @AndrewSirenko

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: torredil

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [torredil]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ConnorJC3]

/lgtm