[TASK] Restore to a brand new cluster that does not contain anything

[shuo-wu]

What's the task? Please describe

Provide the doc that instructs how we can do restore for an empty cluster via Velero

Additional context

Refer to the discussion in https://cloud-native.slack.com/archives/CNVPEL9U3/p1638530150134100?thread_ts=1638467093.129100&cid=CNVPEL9U3

[tom-mayer]

We wanted to have a very generic solution of how to backup/restore a cluster from scratch that uses longhorn as its storage provider. For etcd backup we use Velero.

1. We backup the cluster state and exclude (note that we are also excluding PVs and PVCs):
   - persistentvolumes
   - persistentvolumeclaims
   - volumes.longhorn.io,
   - backups.longhorn.io
   - backupvolumes.longhorn.io
   - nodes.longhorn.io
   - replicas.longhorn.io
   - engines.longhorn.io
   - backingimagedatasources.longhorn.io
   - backingimagemanagers.longhorn.io,
   - backingimages.longhorn.io
   - sharemanagers.longhorn.io
   - instancemanagers.longhorn.io
   - engineimages.longhorn.io
2. On the new cluster which is blank, we install Velero via their CLI. We restore the backup we took before with Velero which will make longhorn spin up with the old settings. The longhorn deployment/pods were in the backup, that why it is not necessary to reinstall it. Which was one of our goals. The settings/backupTarget get restored by the restore of the CRDs through Velero.
3. (All pods that do require longhorn Volumes are now waiting because they don't have their PVCs (which we didn't backup for that exact reason)). When the longhorn ui is ready we click on the "Backup" tab which will load all the backups from the restored backupTarget. We can then batch restore all the Volumes from there
4. On the Volumes tab when the volumes are ready, we recreate the PVs/PVCs form the UI (Actions -> Create PVC). It is crucial to tick the "Use previous PVC" checkbox in the dialog.
5. Claims get auto-attached to the waiting pods

This works since we restored etcd together with longhorn all the original namespaces are there and the PVCs will also get recreated via longhorn in their original namespaces, making the pods heal after they are there.

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

* [ ] Where is the reproduce steps/test steps documented?
The reproduce steps/test steps are at:

* [ ] Is there a workaround for the issue? If so, where is it documented?
The workaround is at:

• Does the PR include the explanation for the fix or the feature?

* [ ] Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
The PR for the YAML change is at:
The PR for the chart change is at:

* [ ] Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
The PR is at

* [ ] Which areas/issues this PR might have potential impacts on?
Area
Issues

* [ ] If labeled: require/LEP Has the Longhorn Enhancement Proposal PR submitted?
The LEP PR is at

* [ ] If labeled: area/ui Has the UI issue filed or ready to be merged (including backport-needed/*)?
The UI issue/PR is at

• If labeled: require/doc Has the necessary document PR submitted or merged (including backport-needed/*)?
  The documentation issue/PR is at
  Add a doc that instructs how users can restore Longhorn to a new cluster  website#440
  [Backport][v1.2.3]Cluster restore doc website#442

* [ ] If labeled: require/automation-e2e Has the end-to-end test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue (including backport-needed/*)
The automation skeleton PR is at
The automation test case PR is at
The issue of automation test case implementation is at (please create by the template)

* [ ] If labeled: require/automation-engine Has the engine integration test been merged (including backport-needed/*)?
The engine automation PR is at

• If labeled: require/manual-test-plan Has the manual test plan been documented?
  The updated manual test plan is at Add a test for the new cluster restore instruction  longhorn-tests#836

* [ ] If the fix introduces the code for backward compatibility Has a separate issue been filed with the label release/obsolete-compatibility?
The compatibility issue is filed at

[khushboo-rancher]

Verified with master-head 02/16/2022

Validation - pass

Follow the instructions from longhorn/website#440 to restore in the new cluster. It worked as expected.