[TASK] Bump K8s min version support, CSI sidecar versions, dependent lib version

[innobead]

What's the task? Please describe

For each feature release, we should revisit if need to bump the minimum support version of K8s and K8s dependent libraries based on different reasons like K8s support version cycle, the minimum versions of K8s used by CSI sidecar components planned to deliver/support (like recent snapshotter), etc.

Describe the items of the task (DoD, definition of done) you'd like

Please use a task list for items on a separate line with a clickable checkbox https://docs.github.com/en/issues/tracking-your-work-with-issues/about-task-lists

• Bump to the recent versions of K8s dependent libraries if needed
• Update the min version support of K8s if needed
• Update CSI sidecar versions if needed

Additional context

Add any other context or screenshots about the task request here.

[joshimoo]

This is a related issue for this task: #3720

[innobead]

@PhanLe1010 will help with this one w/ @joshimoo help.

[PhanLe1010]

Steps to bump K8s, client-go, CSI sidecars, Golang version

1. Check the Deprecated API Migration Guide to identify the deprecated/removed API versions that are used by Longhorn. Using the info on this document, we should be able to determine the minimum supported Kubernetes version by following the steps:
   1. Find the latest Kubernetes version Longhorn is targeting to support. Let's name it k8s-latest. E.g, Kubernetes v1.25.0
   2. Find all the removed API versions in k8s-latest that is being used by Longhorn. E.g., batch/v1beta1 is removed in Kubernetes v1.25.0
   3. Find the newer versions for the above removed API versions in k8s_latest. E.g., batch/v1 is the newer version for batch/v1beta1 in Kubernetes v1.25.0
   4. The minimum Kubernetes version that Longhorn can support is the Kuberenetes version that supports all of the newer API versions in the above step. Let's call this version k8s-min-supported. E.g., batch/v1 API version, available since v1.21 thus the minimum Kubernetes version that Longhorn can support is >= v1.21
2. Upgrade the K8s, client-go dependency library to k8s-min-supported
3. Modify the code base to fix incompatibilities issues and use the newer API versions found in step 1.iv
4. Findout the Golang version that is used to build k8s-min-supported. Upgrade Golang version to this version
5. CSI sidecars:
   1. From the k8s-min-supported, we can find the minimum CSI Spec version go.mod (Let's name it min-csi-spec) of the k8s-min-supported git repo
   2. Using the min-csi-spec, k8s-min-supported, and the information at the document https://kubernetes-csi.github.io/docs/sidecar-containers.html, we can determine the appropriate CSI external sidecar images.
   3. NOTE We need to read release notes for every newer release of the new CSI external sidecar images to see if there is any breaking changes

[PhanLe1010]

Test steps:

For each of the Kubernetes version: [1.18, 1.19, 1.20, 1.21, 1.22, 1.23, 1.24], do:

• Test new install Longhorn master-head
  • Create some volumes (RWO/RWX), workloads, backups job, snapshot jobs
  • Test basic Longhorn functionality
  • Or you can just run our e2e tests
• Test upgrade from older version
  • Install an older version
    • Create some volumes (RWO/RWX), workloads, backups job, snapshot jobs
    • Test basic Longhorn functionality
  • Upgrade to master-head
    • Create some volumes (RWO/RWX), workloads, backups job, snapshot jobs
    • Test basic Longhorn functionality
    • Or you can just run our e2e tests
• Test monitoring metrics
• Test uninstallation
• Test chart installation and uninstallation

[PhanLe1010]

TODO:

• Wait for share manager PR to be merged and released a new tag version
• Update Longhorn manager deployment with new sharemanager
• PR for longhorn/longhorn chart
• PSP:
  • Test PSP on cluster that enable PSP
  • Why do we need PSP

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at:

• Is there a workaround for the issue? If so, where is it documented?
  The workaround is at:

• Does the PR include the explanation for the fix or the feature?

• Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
  The PR for the YAML change is at: Bump CSI images and share manager image tags #4026
  The PR for the chart change is at:

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at

• Bump K8s, client-go, CSI sidecars, Golang version longhorn-manager#1342

• Bump Kubernetes dependent library to v1.23 longhorn-share-manager#30

• Update longhorn share manager image to v1_20220523 longhorn-manager#1345

• Which areas/issues this PR might have potential impacts on?
  Area Upgrade
  Issues

• If labeled: require/LEP Has the Longhorn Enhancement Proposal PR submitted?
  The LEP PR is at

• If labeled: area/ui Has the UI issue filed or ready to be merged (including backport-needed/*)?
  The UI issue/PR is at

• If labeled: require/doc Has the necessary document PR submitted or merged (including backport-needed/*)?
  The documentation issue/PR is at Add note about supported Kubernetes version in Longhorn v1.3.0 website#501

• If labeled: require/automation-e2e Has the end-to-end test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue (including backport-needed/*)
  The automation skeleton PR is at
  The automation test case PR is at
  The issue of automation test case implementation is at (please create by the template)

• If labeled: require/automation-engine Has the engine integration test been merged (including backport-needed/*)?
  The engine automation PR is at

• If labeled: require/manual-test-plan Has the manual test plan been documented?
  The updated manual test plan is at

• If the fix introduces the code for backward compatibility Has a separate issue been filed with the label release/obsolete-compatibility?
  The compatibility issue is filed at

[innobead]

I think probably we can just leverage e2e to see if any outstanding issues later.

@PhanLe1010 has provided the test steps which we can leverage our e2e core testing to verify. cc @yangchiu

[PhanLe1010]

@innobead Could you help to release a new version for longhorn-manager so I can finish the TODO work?

[innobead]

@innobead Could you help to release a new version for longhorn-manager so I can finish the TODO work?

v1_20220523 has been released.

[yangchiu]

For each of the Kubernetes version: [1.18, 1.19, 1.20, 1.21, 1.22, 1.23, 1.24], do:

• Test new install Longhorn master-head, create some volumes (RWO/RWX), workloads, backups job, snapshot jobs, test basic Longhorn functionality

  • 1.18: verified passed on v1.18.20+k3s1. (Tested in local environment since the kubectl has different output format which would make e2e test pipeline stuck)
  • 1.19: verified passed on v1.19.16+k3s1
  • 1.20: verified passed on v1.20.13+k3s1
  • 1.21: verified passed on v1.21.7+k3s1
  • 1.22: verified passed on v1.22.9+k3s1
  • 1.23: verified passed on v1.23.1+k3s2
  • 1.24: verified passed on v1.24.0-rc1+k3s1

• Test upgrade from older version

  • 1.18: verified passed on v1.18.20+k3s1. (Tested in local environment since the kubectl has different output format which would make e2e test pipeline stuck)
  • 1.19: verified passed on v1.19.16+k3s1
  • 1.20: verified passed on v1.20.13+k3s1
  • 1.21: verified passed on v1.21.7+k3s1
  • 1.22: verified passed on v1.22.9+k3s1
  • 1.23: verified passed on v1.23.1+k3s2
  • 1.24: verified passed on v1.24.0-rc1+k3s1

• Test monitoring metrics:

  • Verified by checking http://LONGHORN_MANAGER_IP:PORT/metrics for each k8s version: [ v1.18.20+k3s1, v1.19.16+k3s1, v1.20.13+k3s1, v1.21.7+k3s1, v1.22.9+k3s1, v1.23.1+k3s2, v1.24.0-rc1+k3s1 ]

• Test uninstallation:

  • Verified by following these steps for each k8s version: [ v1.18.20+k3s1, v1.19.16+k3s1, v1.20.13+k3s1, v1.21.7+k3s1, v1.22.9+k3s1, v1.23.1+k3s2, v1.24.0-rc1+k3s1 ]

• Test chart installation and uninstallation:

  • Verified by following [the installation and uninstallation steps for each k8s version: [ v1.18.20+k3s1, v1.19.16+k3s1, v1.20.13+k3s1, v1.21.7+k3s1, v1.22.9+k3s1, v1.23.1+k3s2, v1.24.0-rc1+k3s1 ]