[FEATURE] Update K8s version support and component/pkg/build dependencies for Longhorn 1.5

[innobead]

What's the task? Please describe

For each feature release, we will revisit if need to bump the minimum support version of K8s and K8s dependent libraries based on different reasons like K8s support version cycle, the minimum versions of K8s used by CSI sidecar components planned to deliver/support, etc.

https://kubernetes.io/releases/patch-releases/
At the moment, 1.26, 1.25, and 1.24 are actively maintained branches, and 1.23 EOL will be 2023-02-28, so the expected min supported version will be bumped to 1.24.

Describe the items of the task (DoD, definition of done) you'd like

• Bump the recent versions of K8s-dependent libraries to the recently newer version given no backward compatibility concerns
• Fix deprecated resources being used/removed from unsupported K8s versions
• Update the min version support of K8s
• Update CSI sidecar versions
• Update Golang version
• Update the K8s version in e2e testing to the updated nearly K8s version @longhorn/qa

[innobead]

@ejweber try this, and cooperate with @PhanLe1010 .

[ejweber]

Given @PhanLe1010's strategy in #3891 (comment).

1. The latest version of Kubernetes available is v1.27. Future API deprecations and removals are documented through v1.29.
2. Longhorn does not use any APIs removed between our current minimum Kubernetes version (v1.21) and v1.29. The exception is PodSecurityPolicy, but it is not being replaced, so there is no new version to upgrade to. Additionally, the v0.27 version of client-go can still perform CRUD operations on PodSecurityPolicy, so backup and restore operations for PodSecurityPolicy can still work in older clusters.
3. Longhorn has not added any new APIs that didn't exist in v1.21.
4. There is no API-related reason to bump our minimum Kubernetes version.

If we want to arbitrarily bump our minimum Kubernetes version to remain in lockstep with the Kubernetes project itself, we can bump it to v1.24 as suggested in the issue description.

Current plan:

• Keep minimum Kubernetes version at v1.21.
• Upgrade client-go dependency to v0.27.1.

WDYT @innobead @PhanLe1010 ?

[ejweber]

Additional TODO:

• Wait for the backing-image-manager PR to be merged.
• Update longhorn-manager PR with new import.

[PhanLe1010]

Since we can still support latest Kubernetes version, agree that we should keep the minimum version as v1.21

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?

• Update the K8s version in e2e testing to the updated nearly K8s version.
• I have already run the core tests on Kubernetes v1.24 and v1.25 with no issues.
• It's probably interesting to verify system backup and restore functionality on v1.24 with a PSP enabled, since this is the area of greatest risk (I have already done so as well).

• Is there a workaround for the issue? If so, where is it documented?

• Does the PR include the explanation for the fix or the feature?

• Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at: Ensure .proto file name is globally unique backing-image-manager#125.
  The PR is at: Bump Kubernetes dependency for Longhorn v1.5.0 longhorn-manager#1910.

• Which areas/issues this PR might have potential impacts on?
  I have done some testing around this already, but client-go v1.27 does not clearly document its support (or lack of support) for PodSecurityPolicies. Since we interact with PodSecurityPolicies in system backup and restore, we should take care here.

• If labeled: require/LEP Has the Longhorn Enhancement Proposal PR submitted?

• If labeled: area/ui Has the UI issue filed or ready to be merged (including backport-needed/*)?

• If labeled: require/doc Has the necessary document PR submitted or merged (including backport-needed/*)?
  The documentation issue/PR is at: Slightly tweak Kubernetes minimum version wording website#706.

• If labeled: require/automation-e2e Has the end-to-end test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue (including backport-needed/*)

• If labeled: require/automation-engine Has the engine integration test been merged (including backport-needed/*)?

• If labeled: require/manual-test-plan Has the manual test plan been documented?

• If the fix introduces the code for backward compatibility Has a separate issue been filed with the label release/obsolete-compatibility?

[innobead]

Since we can still support latest Kubernetes version, agree that we should keep the minimum version as v1.21

Agreed. If nothing changes, there is no strong reasons to change the min version. Let's do this least breaking change.

[yangchiu]

• Update the K8s version in e2e testing to the updated nearly K8s version:
  • Already updated to the latest version v1.27 in ci: update k8s version to v1.27 infra#124