[FEATURE] Introduce `upgradeVersionCheck` to decide version upgrade enforcement

[innobead]

What's the task? Please describe

Requesting to allow Harvester users to skip version 1.5 and upgrade directly from version 1.4 to 1.6. Upgrade version enforcement requires a new path to be created. Besides, having this new mechanism to bypass version performance in some specific user cases.

Describe the sub-tasks

• Add back 1.4 -> 1.5 upgrade path migration code
• Disable pre-upgrade check for Longhorn manager
• Change the values.yaml by adding upgradeVersionCheck value to decide to do the upgrade version check in the Longhorn manager.

helmPreUpgradeCheckerJob:
  enabled: true

preUpgradeChecker:
  jobEnabled: true
  upgradeVersionCheck: true

When setting upgradeVersionCheck true, we will have a warning log and event to inform users this is not a recommended behavior. For Harvester, they can take this as a path for 1.4 -> 1.6.

Additional context

We need to maintain the upgrade paths of previous versions within the system, but this is a disadvantage.

cc @longhorn/qa

[mantissahz]

When setting upgradeVersionCheck true, we will have a warning log and event to inform users this is not a recommended behavior. For Harvester, they can take this as a path for 1.4 -> 1.6.

@innobead

Can it be that when setting upgradeVersionCheck false, we will skip the upgrade path check and have a warning log and event to inform users this is not a recommended behavior?

Because setting preUpgradeChecker.jobEnabled false does not exactly skip the upgrade path.

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at:

1. Install 1.4.x Longhorn by Helm
2. Upgrade Longhorn to 1.6.0-dev (master-head) with preUpgradeChecker.jobEnabled and preUpgradeChecker.upgradeVersionCheck both true
3. Upgrade failed because of the pre-upgrade job.
4. Upgrade Longhorn to 1.6.0-dev (master-head) with preUpgradeChecker.jobEnabledP: false and preUpgradeChecker.upgradeVersionCheck: true
5. Upgrade failed with the longhorn-manager in CrashLoopBackOff loop because upgrade path is not supported (check logs of the longhorn-manager).
6. Upgrade Longhorn to 1.6.0-dev (master-head) with preUpgradeChecker.jobEnabled and preUpgradeChecker.upgradeVersionCheck both false
7. Upgrade successfully and workloads work well.
8. Fresh install 1.6.0-dev Longhorn successfully and workloads work well.

• Does the PR include the explanation for the fix or the feature?

• Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
  The PR for the YAML change is at:
  The PR for the chart change is at:
  fix(upgrade): skip upgrade path check #7586
  fix(upgrade): skip to check engine upgrade path #7613

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at
  fix(upgrade): skip upgrade path check longhorn-manager#2435
  fix(upgrade): skip to check engine upgrade path longhorn-manager#2438

• Which areas/issues this PR might have potential impacts on?
  Area
  Issues

• If labeled: require/doc Has the necessary document PR submitted or merged (including backport-needed/*)?
  The documentation issue/PR is at
  Apply new wording convention to descriptions in helm-values.md website#835

• If labeled: require/manual-test-plan Has the manual test plan been documented?
  The updated manual test plan is at
  test: new field Incompatible of EngineImage longhorn-tests#1677