Rancher fleetchart updates and pins version

[manno]

SURE-6673

When upgrading Rancher from one minor version to another, e.g. 2.6 to 2.7, the chart-default-branch changes to release-v2.7. This leads to an update of the catalog resource and a git update. Afterwards (at the same time?) the fleetchart controller triggers, ensuring the latest fleet version from the rancher/charts repo is installed. It will install at least the version of fleet that is greater CATTLE_FLEET_MIN_VERSION.

Besides triggering an upgrade due to a minor version upgrade of Rancher, there are reports of other situations, which will lead to a repo update and fleet upgrade.

This upgrade process ignores the chart annotation catalog.cattle.io/rancher-version, which is used for filtering UI dropbox entries only:

there is no need to add the annotation to CRD charts or internal charts (like fleet).

Which versions are available to install? Besides the charts in the repo, there is also a fleet version in the bundled repo (container image) and the container's cache.

In practice this means, there is no direct relation between the Rancher version and the Fleet version.
Additionally, when updating from one patch version to another (x.y.Z), the chart-default-branch does not change and often no automatic update of fleet is triggered:

• installing 2.7.0 will install fleet 0.7
• updating an old 2.7.0 will keep fleet 0.6

Acceptance Criteria

We want to pin the fleet version to the Rancher release. When updating, installing X, we should always pull in fleet Y.

• It should still be possible to change the version manually.
• For out of band releases, don't update automatically
• Trigger on new setting fleet_version

See rancher/rancher#40159 for a possible solution.

[weyfonk]

Update: we have agreed to restrict pinning as follows: Rancher shall refrain from updating a manually set Fleet version only if that version is more recent than the pinned Fleet version for the considered Rancher release.
This relies on Fleet versions being backwards-compatible and prevents downgrades.

[weyfonk]

• Update CI workflow Release Fleet against rancher/rancher to update CATTLE_FLEET_VERSION instead of CATTLE_FLEET_MIN_VERSION

[kkaempf]

See rancher/rancher#42304 (comment) for additional test cases

[kkaempf]

@weyfonk what is this issue waiting for ?

[weyfonk]

What we need at this point is a review of #1880 to be able to properly release Fleet against Rancher, as per my previous comment.

[sbulage]

QA TEST PLAN

Scenarios

Scenario	Test Case
1	Test Rancher installs fleet version via CATTLE_FLEET_VERSION environment variable.
2	Test Rancher installs fleet version via CATTLE_FLEET_MIN_VERSION(deprecated) environment variable.
3	Test Rancher didn't upgrade fleet version version to latest when CATTLE_FLEET_VERSION was pointed to lower version (less than latest version).
4	Test Rancher installs fleet version via CATTLE_FLEET_VERSION environment variable and fleet version didn't upgrade when rancher pod restarted
5	Test Rancher installs fleet version via CATTLE_FLEET_MIN_VERSION(deprecated) environment variable and fleet version didn't upgrade when rancher pod restarted.
6	Test Rancher UI shows same installed fleet version in the UI when newer/latest version passed via CATTLE_FLEET_VERSION environment variable while installing Rancher.
7	Test Rancher UI shows newer version of fleet in the UI when older version passed via CATTLE_FLEET_VERSION environment variable while installing Rancher.
8	Test fleet version didn't get change after successful rancher backup restore happened.
9	Test Rancher didn't install fleet version when invalid CATTLE_FLEET_MIN_VERSION environment variable is passed.
10	Test Rancher didn't install fleet version when invalid CATTLE_FLEET_VERSION environment variable is passed.

---

Note: Referring some test scenario's from this comment: rancher/rancher#42304 (comment)

[sbulage]

TEST RESULT

Scenario	Test Case	Results
1	Test Rancher installs fleet version via CATTLE_FLEET_VERSION environment variable.	✔️
2	Test Rancher installs fleet version via CATTLE_FLEET_MIN_VERSION(deprecated) environment variable.	✔️
3	Test Rancher didn't upgrade fleet version version to latest when CATTLE_FLEET_VERSION was pointed to lower version (less than latest version).	✔️
4	Test Rancher installs fleet version via CATTLE_FLEET_VERSION environment variable and fleet version didn't upgrade when rancher pod restarted.	✔️
5	Test Rancher installs fleet version via CATTLE_FLEET_MIN_VERSION(deprecated) environment variable and fleet version didn't upgrade when rancher pod restarted.	✔️
6	Test Rancher UI shows same installed fleet version in the UI when newer/latest version passed via CATTLE_FLEET_VERSION environment variable while installing Rancher.	✔️
7	Test Rancher UI shows newer version of fleet in the UI when older version passed via CATTLE_FLEET_VERSION environment variable while installing Rancher.	✔️
8	Test fleet version didn't get change after successful rancher backup restore happened.	✔️
9	Test Rancher didn't install fleet version when invalid CATTLE_FLEET_MIN_VERSION environment variable is passed.
10	Test Rancher didn't install fleet version when invalid CATTLE_FLEET_VERSION environment variable is passed.

REPRO STEPS
Prerequisites:

1. Prepare Kubernetes cluster.
2. Add Rancher helm repository rancher/rancher-alpha

---

Scenario 1

1. In this scenario used the newly introduced environment variable CATTLE_FLEET_VERSION.
2. Setting value less than the available version.
3. Below command is executed.

   helm install rancher rancher-alpha/rancher \
   --namespace cattle-system \
   --set hostname=<IP/HOSTNAME.sslip.io \
   --set replicas=1 \
   --set bootstrapPassword=<PASSWORD> \
   --set global.cattle.psp.enabled=false \
   --set rancherImageTag=v2.8-head \
   --devel \
   --set 'extraEnv[0].name=CATTLE_FLEET_VERSION' \
   --set 'extraEnv[0].value=102.1.0+up0.7.0'
   

4. Wait for the Rancher to create resources and UI running.
5. Verified that Fleet version 0.7.0 is installed.

---

Scenario 2

1. In this scenario used the newly introduced environment variable CATTLE_FLEET_MIN_VERSION.
2. Setting value less than the available version.
3. Below command is executed.

   helm install rancher rancher-alpha/rancher \
   --namespace cattle-system \
   --set hostname=<IP/HOSTNAME.sslip.io \
   --set replicas=1 \
   --set bootstrapPassword=<PASSWORD> \
   --set global.cattle.psp.enabled=false \
   --set rancherImageTag=v2.8-head \
   --devel \
   --set 'extraEnv[0].name=CATTLE_FLEET_MIN_VERSION' \
   --set 'extraEnv[0].value=102.1.1+up0.7.1'
   

4. Wait for the Rancher to create resources and UI running.
5. Verified that Fleet version 0.7.1 is installed.

---

Scenario 3

1. In this scenario used the newly introduced environment variable CATTLE_FLEET_VERSION.
2. Setting value less than the available version.
3. Below command is executed.

   helm install rancher rancher-alpha/rancher \
   --namespace cattle-system \
   --set hostname=<IP/HOSTNAME.sslip.io \
   --set replicas=1 \
   --set bootstrapPassword=<PASSWORD> \
   --set global.cattle.psp.enabled=false \
   --set rancherImageTag=v2.8-head \
   --devel \
   --set 'extraEnv[0].name=CATTLE_FLEET_VERSION' \
   --set 'extraEnv[0].value=102.1.0+up0.7.0'
   

4. Wait for the Rancher to create resources and UI running.
5. Verified that Fleet version 0.7.0 is installed.
6. Verified that version is not changed/upgraded to the available version.

---

Scenario 4

1. In this scenario used the newly introduced environment variable CATTLE_FLEET_MIN_VERSION.
2. Setting value less than the available version.
3. Below command is executed.

   helm install rancher rancher-alpha/rancher \
   --namespace cattle-system \
   --set hostname=<IP/HOSTNAME.sslip.io \
   --set replicas=1 \
   --set bootstrapPassword=<PASSWORD> \
   --set global.cattle.psp.enabled=false \
   --set rancherImageTag=v2.8-head \
   --devel \
   --set 'extraEnv[0].name=CATTLE_FLEET_MIN_VERSION' \
   --set 'extraEnv[0].value=102.2.0+up0.8.0'
   

4. Wait for the Rancher to create resources and UI running.
5. Verified that Fleet version 0.8.0 is installed.
6. Verified that version is not changed/upgraded when Rancher pod.

---

Scenario 5

1. In this scenario used the newly introduced environment variable CATTLE_FLEET_MIN_VERSION.
2. Setting value less than the available version.
3. Below command is executed.

   helm install rancher rancher-alpha/rancher \
   --namespace cattle-system \
   --set hostname=<IP/HOSTNAME.sslip.io \
   --set replicas=1 \
   --set bootstrapPassword=<PASSWORD> \
   --set global.cattle.psp.enabled=false \
   --set rancherImageTag=v2.8-head \
   --devel \
   --set 'extraEnv[0].name=CATTLE_FLEET_MIN_VERSION' \
   --set 'extraEnv[0].value=102.1.0+up0.7.0'
   

4. Wait for the Rancher to create resources and UI running.
5. Verified that Fleet version 0.7.0 is installed.
6. Verified that version is not changed/upgraded when Rancher pod.

---

Scenario 6

1. In this scenario used the newly introduced environment variable CATTLE_FLEET_VERSION.
2. Setting value less than the available version.
3. Below command is executed.

   helm install rancher rancher-alpha/rancher \
   --namespace cattle-system \
   --set hostname=<IP/HOSTNAME.sslip.io \
   --set replicas=1 \
   --set bootstrapPassword=<PASSWORD> \
   --set global.cattle.psp.enabled=false \
   --set rancherImageTag=v2.8-head \
   --devel \
   --set 'extraEnv[0].name=CATTLE_FLEET_VERSION' \
   --set 'extraEnv[0].value=103.1.0+up0.9.0-rc.5'
   

4. Wait for the Rancher to create resources and UI running.
5. Verified that Fleet version 0.9.0-rc.5 is installed.
6. Navigate to the Apps --> Installed Charts --> Fleet (Edit/Upgrade).
7. Verified that same fleet version is selected which is latest/newer version specified for CATTLE_FLEET_VERSION environment variable.

---

Scenario 7

1. In this scenario used the newly introduced environment variable CATTLE_FLEET_VERSION.
2. Setting value less than the available version.
3. Below command is executed.

   helm install rancher rancher-alpha/rancher \
   --namespace cattle-system \
   --set hostname=<IP/HOSTNAME.sslip.io \
   --set replicas=1 \
   --set bootstrapPassword=<PASSWORD> \
   --set global.cattle.psp.enabled=false \
   --set rancherImageTag=v2.8-head \
   --devel \
   --set 'extraEnv[0].name=CATTLE_FLEET_VERSION' \
   --set 'extraEnv[0].value=103.1.0+up0.9.0-rc.5'
   

4. Wait for the Rancher to create resources and UI running.
5. Verified that Fleet version 0.8.0 is installed.
6. Navigate to the Apps --> Installed Charts --> Fleet (Edit/Upgrade).
7. Verified that same fleet version is selected which is latest/newer version specified for CATTLE_FLEET_VERSION environment variable.

---

Scenario 8

1. Deployed K8s in a VM/instance on AWS.
2. In this scenario used the newly introduced environment variable CATTLE_FLEET_VERSION.
3. Setting value less than the available version.
4. Below command is executed.

   helm install rancher rancher-alpha/rancher \
   --namespace cattle-system \
   --set hostname=<IP/HOSTNAME.sslip.io \
   --set replicas=1 \
   --set bootstrapPassword=<PASSWORD> \
   --set global.cattle.psp.enabled=false \
   --set rancherImageTag=v2.8-head \
   --devel \
   --set 'extraEnv[0].name=CATTLE_FLEET_VERSION' \
   --set 'extraEnv[0].value=103.1.0+up0.9.0-rc.5'
   

5. Wait for the Rancher to create resources and UI running.
6. Verified that Fleet version 0.8.0 is installed.
7. Setup Backup/Restore operator using S3 endpoints.
8. Created backup of current Rancher setup.
9. Deployed same k8s in a different VM/instance on AWS.
10. Restored the rancher backup created from previous step.
11. Wait for a restore to complete and Rancher UI comes up.
12. Verified that same fleet version is installed which was mentioned in the CATTLE_FLEET_VERSION before backup i.e. 0.8.0.

---

[sbulage]

Last 2 scenario's will be covered in a separate issue. PR is already created.

Scenarios

Scenario	Test Case
9	Test Rancher didn't install fleet version when invalid CATTLE_FLEET_MIN_VERSION environment variable is passed.
10	Test Rancher didn't install fleet version when invalid CATTLE_FLEET_VERSION environment variable is passed.