[BUG] No PCI device display after upgrade to v1.2.0 master and enabled the pcidevices-controller addons

[TachunLin]

Describe the bug

After we upgrade from v1.1.2 to v1.2.0 master-b49e264e-head and enable the pcidevices-controller.

Open the PCI Devices page, there is no available pcidevices display.

vokoscreenNG-2023-05-24_13-54-52.mp4

To Reproduce
Steps to reproduce the behavior:

1. Upgrade from v1.1.2 to v1.2.0 master
2. After upgrade complete, open Addons page
3. Enable the pcidevices-controller addon
4. Check the addon in Enabled state
5. Open the PCI Devices page
6. Check the content

Expected behavior

After upgrade from v1.1.2 to v1.2.0 master and enable the pcidevices-controller.
We should be able to see the availabe PCI devices and use this feature

Support bundle

supportbundle_ebea348e-b561-4725-8608-26223fe6f392_2023-05-24T05-51-02Z.zip

Environment

• Harvester ISO version: master-b49e264e-head (05/22)
• Underlying Infrastructure (e.g. Baremetal with Dell PowerEdge R630): 3 nodes bare machines

[bk201]

@TachunLin need to clarify, did you enable the PCI devices add-on before or after the upgrade? Thanks.
The recording has the add-on enabled already.

[TachunLin]

Before upgrade the PCI devices addon did not enabled, just as we fresh install v1.1.2 Harvester cluster.
After upgrade to v1.2.0 master, I try to enable the addon and found no PCI devices list on the page.

In comparison, I fresh install a single node v1.2.0-rc1 cluster and enable the PCI device addon on kvm machine

The pci devices can display correctly after enabled the addon

[ibrokethecloud]

the upgrade shows that addon version being deployed is v0.2.4

apiVersion: harvesterhci.io/v1beta1
kind: Addon
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"harvesterhci.io/v1beta1","kind":"Addon","metadata":{"annotations":{},"labels":{"addon.harvesterhci.io/experimental":"true"},"name":"pcidevices-controller","namespace":"harvester-system"},"spec":{"chart":"harvester-pcidevices-controller","enabled":false,"repo":"http://harvester-cluster-repo.cattle-system.svc/charts","valuesContent":"image:\n  tag: v0.2.4\nfullnameOverride: harvester-pcidevices-controller\n","version":"0.2.4"}}
    sim.harvesterhci.io/creationTimestamp: "2023-05-23T08:01:55Z"
  creationTimestamp: "2023-05-23T08:01:55Z"
  generation: 1
  labels:
    addon.harvesterhci.io/experimental: "true"
  name: pcidevices-controller
  namespace: harvester-system
  resourceVersion: "1951"
  uid: dd759db6-9410-4f58-91ff-e5d5429f667a
spec:
  chart: harvester-pcidevices-controller
  enabled: true
  repo: http://harvester-cluster-repo.cattle-system.svc/charts
  valuesContent: |
    image:
      tag: v0.2.4
    fullnameOverride: harvester-pcidevices-controller
  version: 0.2.4
status:
  status: AddonEnabled

This should be 0.2.5-rc1 post upgrade. The actual addon never gets deployed since the job is unable to find chart for v0.2.4

2023-05-24T05:50:06.107680081Z + echo 'Installing helm_v3 chart'
2023-05-24T05:50:06.107950742Z + helm_v3 install --repo http://harvester-cluster-repo.cattle-system.svc/charts --version 0.2.4 pcidevices-controller harvester-pcidevices-controller --values /config/values-01_HelmChart.yaml
2023-05-24T05:50:06.159640337Z Error: INSTALLATION FAILED: chart "harvester-pcidevices-controller" version "0.2.4" not found in http://harvester-cluster-repo.cattle-system.svc/charts repository

[TachunLin]

Attach upgrade log for the reference
hvst-upgrade-rk756-upgradelog-archive-2023-05-24T07-41-32Z.zip

[bk201]

It looks like the culprit is this check: https://github.com/harvester/harvester/blob/5e4e3ffeb0b53b86165cd0dab834446de1862d0c/package/upgrade/upgrade_manifests.sh#LL627C83-L627C83
Add-ons are not upgraded.

[harvesterhci-io-github-bot]

Pre Ready-For-Testing Checklist

• If labeled: require/HEP Has the Harvester Enhancement Proposal PR submitted?
  The HEP PR is at:

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at: fix(upgrade): remove version check for addon upgrades #3987

• Is there a workaround for the issue? If so, where is it documented?
  The workaround is at:

• Have the backend code been merged (harvester, harvester-installer, etc) (including backport-needed/*)?
  The PR is at: fix(upgrade): remove version check for addon upgrades #3987

  • Does the PR include the explanation for the fix or the feature?

  • Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
    The PR for the YAML change is at:
    The PR for the chart change is at:

• If labeled: area/ui Has the UI issue filed or ready to be merged?
  The UI issue/PR is at:

• If labeled: require/doc, require/knowledge-base Has the necessary document PR submitted or merged?
  The documentation/KB PR is at:

• ~~If NOT labeled: not-require/test-plan Has the e2e test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue?

  • The automation skeleton PR is at:
  • The automation test case PR is at:~~

• If the fix introduces the code for backward compatibility Has a separate issue been filed with the label release/obsolete-compatibility?
  The compatibility issue is filed at:

[harvesterhci-io-github-bot]

Automation e2e test issue: harvester/tests#831

[harvesterhci-io-github-bot]

added backport-needed/1.1.3 issue: #3994.

[TachunLin]

Verified fixed on master-7d7f68ff-head (5/30). Close this issue.

Result

After upgrade from v1.1.2 to v1.2.0 master,

• Check the harvester-pcidevices-controller have been upgraded to 0.2.5-rc1

...
spec:
  chart: harvester-pcidevices-controller
  enabled: false
  repo: http://harvester-cluster-repo.cattle-system.svc/charts
  valuesContent: |
    image:
      tag: v0.2.5-rc1
    fullnameOverride: harvester-pcidevices-controller
  version: 0.2.5-rc1

• After we enable the pcidevices-controller addon and wait for while. The PCI devices list can display correctly
  

Test Information

• Test Environment: 3 nodes harvester on bare machines
• Harvester version: master-7d7f68ff-head (5/30)

Verify Steps

1. Upgrade from v1.1.2 to v1.2.0 master
2. After upgrade complete, open Addons page
3. Enable the pcidevices-controller addon
4. Check the addon in Enabled state
5. Open the PCI Devices page
6. Check the content