[BUG] Upgrade from v1.1.0 to v1.1.1 stuck in ErrApplied with harvester-longhorn StorageClass

[guangbochen]

Describe the bug
Upgrade from v1.1.0 to v1.1.1 stuck in ErrApplied with harvester-longhorn StorageClass

bundleState: ErrApplied
      message: 'cannot patch "harvester-longhorn" with kind StorageClass: admission
        webhook "validator.harvesterhci.io" denied the request: default storage class
        %!s(MISSING) already exists, please reset it first'

To Reproduce
Steps to reproduce the behavior:

1. Create a new storageClass via the Advanced tab and set it as the default one.
2. Upgrade existing v1.1.0 cluster to v1.1.1.
3. The upgrade is stuck in Upgrading System Service with 50% status.
4. Check the upgrade yaml via $ks get upgrades.harvesterhci.io hvst-upgrade-d54pm -oyaml and it was stuck with Current version: 1.1.1, Current state: ErrApplied, Current generation: 9
5. Check the Harvester management chart $ks get managedcharts -n fleet-local harvester -oyaml | less and it was stuck with the above StorageClass webhook denied error.

Expected behavior
The upgrade should be continued successfully with other configured default storage class.

Support bundle

Environment

• Harvester ISO version: v1.1.0
• Underlying Infrastructure (e.g. Baremetal with Dell PowerEdge R630): Baremental

Additional context
Add any other context about the problem here.

[guangbochen]

Workaround:

1. go to the Advanced > Storage Classes page
2. set the harvester-longhorn as the default storage class first.

[harvesterhci-io-github-bot]

Pre Ready-For-Testing Checklist

* [ ] If labeled: require/HEP Has the Harvester Enhancement Proposal PR submitted?
The HEP PR is at:

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at: [BUG] Upgrade from v1.1.0 to v1.1.1 stuck in ErrApplied with harvester-longhorn StorageClass #3221 (comment)

• Is there a workaround for the issue? If so, where is it documented?
  The workaround is at: [BUG] Upgrade from v1.1.0 to v1.1.1 stuck in ErrApplied with harvester-longhorn StorageClass #3221 (comment)

* [ ] Have the backend code been merged (harvester, harvester-installer, etc) (including backport-needed/*)?
The PR is at:

• Does the PR include the explanation for the fix or the feature? fix storage class issue when upgrading #3225

* [ ] Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
The PR for the YAML change is at:
The PR for the chart change is at:

* [ ] If labeled: area/ui Has the UI issue filed or ready to be merged?
The UI issue/PR is at:

* [ ] If labeled: require/doc, require/knowledge-base Has the necessary document PR submitted or merged?
The documentation/KB PR is at:

* [ ] If NOT labeled: not-require/test-plan Has the e2e test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue?
- The automation skeleton PR is at:
- The automation test case PR is at:

* [ ] If the fix introduces the code for backward compatibility Has a separate issue been filed with the label release/obsolete-compatibility?
The compatibility issue is filed at:

[harvesterhci-io-github-bot]

Automation e2e test issue: harvester/tests#642

[bk201]

Prefer to test this after harvester/harvester-installer#387 is merged. Thanks!

[bk201]

Test plans:

case 1

• Upgrade from v1.0.3 to master and check storage class:

$ kubectl get sc
NAME                           PROVISIONER          RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
harvester-longhorn (default)   driver.longhorn.io   Delete          Immediate           true                   37s
longhorn                       driver.longhorn.io   Delete          Immediate           true                   37s

A new harvester-longhorn storage class is created as the default storage class, and the longhorn storage class still exists.

case 2

• Create a new v1.1.0 cluster.
• Create a new storage class. Say test and set it as the default storage class.
• Upgrade to master.
• Storage class test should still be the default storage class.

case 3

• Create a new v1.1.0 cluster.
• (Optional) Create a new storage class. Say test ( not set it as the default storage class).
• Upgrade to master.
• Storage class harvester-longhorn should still be the default storage class.

[lanfon72]

Verified this bug has been fixed.

Test Information

• Environment: baremetals DL360G9 3 nodes, qemu/KVM single node, qemu/KVM single node
• Harvester Version: v1.0.3/v1.1.0 to v1.1.1-37ff3072-head
• ui-source Option: Auto

Verify Steps:

v1.0.3 upgrade to v1.1.1-head, baremetals 3 nodes

1. Perform upgrade¹
2. Upgrade should complete successfully
3. Navigate to Advanced/Storage Classes
4. StorageClass harvester-longhorn and longhorn should exist
5. StorageClass harvester-longhorn should be the default

v1.1.0 upgrade to v1.1.1-head, qemu/KVM single node

1. Login to Dashboard, navigate to Advanced/Storage Classes
2. Create StorageClass test then set as default
3. Create an Image img1 with StorageClass test
4. Perform upgrade¹
5. Upgrade should complete successfully
6. Create VM vm1 base on the image img1
7. VM vm1 should started successfully

v1.1.0 upgrade to v1.1.1-head, qemu/KVM single node

1. Login to Dashboard, navigate to Advanced/Storage Classes
2. Create StorageClass test
3. Create an Image img1 with StorageClass test
4. Perform upgrade¹
5. Upgrade should complete successfully
6. Create VM vm1 base on the image img1
7. VM vm1 should started successfully

Footnotes

1. https://docs.harvesterhci.io/v1.1/upgrade/automatic#prepare-an-air-gapped-upgrade ↩ ↩² ↩³