[rancher-backup] resourceSet is left in the old version if the rancher-backup app is already the latest in a upgrade-rollback-upgrade scenario

[sowmyav27]

Steps:

1. run rancher 2.5.9
2. install the rancher-backup 1.0.400
3. make a backup, name it backup-1
4. upgrade rancher to 2.6.0
5. upgrade the rancher-backup to 2.0.0
6. restore the backup
7. upgrade rancher to 2.6.0 again

Results:
The resourceSet in the cluster is the one designed for 2.5.9

Expected Results:
The resourceSet in the cluster should be the one for 2.6.0

Reasons:
Since the default resourceSet named rancher-resource-set is in the backup file, so it is restored to the version for Rancher 2.5.9 at step 6. But since the rancher-backup app is already in its latest version, rancher-resource-set is untouched even we upgrade rancher to Rancher 2.6.0 again at step 7.

As the result, we are running the rancher-backup app 2.0.0 with an old version of the resourceSet.
If we make new backups now, the backups will NOT contain any new resources that are defined in new resourceSet in rancher-backup 2.0.0

Workaround:
Edit and upgrade the rancher-backup app to trigger a correction on the resourceSet

SURE-5348

[jiaqiluo]

It is expected to restore the resouceSet as part of the rancher-restoring process because the resoureSet needs to match the current status of Rancher and other apps in the cluster. For example, we restore the backup of rancher 2.5.9, the resourceSet should also be the one designed for rancher 2.5.9, not the one for 2.6.0, although we are running a newer version of the rancher-backup app.

---

UPDATE:
The above explanation is still valid.
But The issue is edited to reflect another issue that is found in this scenario.

[MKlimuszka]

There was a design meeting in October with a work-around discussed that Eli was supposed to check on with QA.

Otherwise, this needs to be reproduced in 2023-Q1-2.6x and 2023-Q1-2.7x

[eliyamlevy]

@MKlimuszka Tried to reproduce this issue by upgrading from rancher v2.6.0 and rancher-backups v2.0.0 to rancher v2.7-head and rancher-backups v3.0.0 and the issue did not occur.

The resourceSet was updated after upgrading the backups charts on the upgraded rancher. Since rancher v2.5 is EOL I believe we can cose the backport issue as well.