[FEATURE] Backup/Restore Longhorn System

[yasker]

Rancher has introduced ETCD snapshot/restore, but we found that it doesn't work well with Longhorn. More importantly, there is currently no recommended way to backup the Longhorn system itself. It can be useful for the following scenarios:

1. Rollback in the case of an upgrade failure.
2. Recreate/replicate the cluster

In general, we can dump the related CRDs and rollback to them if something went wrong.

It requires some designs to offer the best experience to the end user.

REF: #1648

[innobead]

This is what we have now for backup/restore longhorn.

• https://longhorn.io/docs/1.3.0/advanced-resources/cluster-restore/restore-to-a-cluster-contains-data-using-rancher-snapshot/
• https://longhorn.io/docs/1.3.0/advanced-resources/cluster-restore/restore-to-a-new-cluster-using-velero/

[c3y1huang]

Note: @FrankYang0529 is currently working on backing image backup and restore. AFAIK the implementation will have:

1. New backup cmd to the backing image manager to upload to the backup targets.
2. New fields ex:lastBackingImage... to the related CR

For this issue, we can leverage the field lastBackingImage for rollout restore.

Relate issue #4165, harvester/harvester#2237

cc @innobead @shuo-wu

[c3y1huang]

Backing image backup/restore, ref #4385.

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at:

• Is there a workaround for the issue? If so, where is it documented?
  The workaround is at:

  • https://longhorn.io/docs/1.3.0/advanced-resources/cluster-restore/restore-to-a-new-cluster-using-velero/
  • https://longhorn.io/docs/1.3.0/advanced-resources/cluster-restore/restore-to-a-cluster-contains-data-using-rancher-snapshot/

• Does the PR include the explanation for the fix or the feature?

• Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
  The PR for the YAML change is at:

  • feat(system-backup/restore): chart, YAML update #4916
    The PR for the chart change is at:
  • feat(system-backup/restore): chart, YAML update #4916

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at:

  • feat(system-backup-restore): implement longhorn-manager#1489
  • feat(system-backup/restore): new commands backupstore#90
  • feat(system-backup/restore): new commands longhorn-engine#767

• Which areas/issues this PR might have potential impacts on?
  Area *
  Issues

• If labeled: require/LEP Has the Longhorn Enhancement Proposal PR submitted?
  The LEP PR is at:

  • feat(system-backup/restore): add LEP #4388

• If labeled: area/ui Has the UI issue filed or ready to be merged (including backport-needed/*)?
  The UI issue/PR is at:

  • [IMPROVEMENT] Longhorn system backup/restore UI #4855

• If labeled: require/doc Has the necessary document PR submitted or merged (including backport-needed/*)?
  The documentation issue/PR is at:

  • doc(system-backup/restore): add feature doc website#621

• If labeled: require/automation-e2e Has the end-to-end test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue (including backport-needed/*)
  The automation skeleton PR is at:

  • test(auto/manual): system backup/restore longhorn-tests#1191
    The automation test case PR is at:
  • test(auto/manual): system backup/restore longhorn-tests#1191
    The issue of automation test case implementation is at:
  • [TEST][FEATURE] Backup/Restore Longhorn System #4923

• If labeled: require/automation-engine Has the engine integration test been merged (including backport-needed/*)?
  The engine automation PR is at

• If labeled: require/manual-test-plan Has the manual test plan been documented?
  The updated manual test plan is at:

  • test(auto/manual): system backup/restore longhorn-tests#1191

• If the fix introduces the code for backward compatibility Has a separate issue been filed with the label release/obsolete-compatibility?
  The compatibility issue is filed at

[khushboo-rancher]

Validation - PASS

Verified below scenarios and tests from longhorn/longhorn-tests#1191 on v1.4.x-head

1. Backup creation and deletion - PASS
2. Restore creation and deletion - PASS
3. Empty backup target handling - PASS
4. Backup and restore with RWO volume (Created using sc, LH GUI, vch template - PASS
5. Backup and restore of RWX volume - PASS
6. Backup and Restore of volume with backing image - FAIL - [IMPROVEMENT] System restore unable to restore volume with backing image #5085
7. Restore in same cluster and new cluster - PASS
8. Backup & Restore in upgraded set up - PASS
9. Backup & Restore of volumes with setting changes - PASS
10. Backup & restore of 50 volumes - PASS