[FEATURE] v2 volume supports volume backup/restore

[derekbit]

Is your feature request related to a problem? Please describe (👍 if you like this request)

SPDK volumes support volume incremental/delta backup, and full/incremental (DR) restore.

Sub-tasks

• LEP
  • lep: add Support Backup and Full Restore for Volumes with V2 Data Engine #6480
• SPDK bdev_lvol_get_fragmap JSON-SPDK method
  • Implement bdev_lvol_get_fragmap spdk#17
• SPDK lvol get and set xattr operations and snapshot lvol creation timestamp
  • [TASK] Implement xattr get and set operations on SPDK logical volumes (lvol) #6604
• Move common functions for backup to backupstore lib
  • [TASK] Move common functions for backup to backupstore lib #6514
• Abstract the disk/lvol file operations in backupstore
  • [REFACTOR] Abstract the disk/lvol file operations in backupstore #6576
• Backup implementation
• Restore implementation
  • Full restore
  • [ ] Incremental restore (not in the scope of the ticket)
• UI improvement
  • [IMPROVEMENT] UI backup restoration supports v1 and v2 Data Engine #6597

Describe the solution you'd like

A clear and concise description of what you want to happen

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

[innobead]

This probably needs to work with @shuo-wu, @DamiaSan or @james-munson .

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at:

Use the UI image (derekbit/longhorn-ui:backup-1) for the following tests

1. Backup: Test the backup of v2 volumes as v1 volumes
2. Restore
   • Restore v2 volume's backup to v2 volume
   • Restore v2 volume's backup to v1 volume
   • Restore v1 volume's backup to v1 volume
   • Restore v1 volume's backup to v2 volume

• Does the PR include the explanation for the fix or the feature?

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at

longhorn/spdk#17
longhorn/spdk#18

longhorn/go-spdk-helper#45
longhorn/go-spdk-helper#46

longhorn/longhorn-spdk-engine#61
longhorn/longhorn-spdk-engine#85
longhorn/longhorn-instance-manager#282
longhorn/longhorn-manager#2137

• Which areas/issues this PR might have potential impacts on?
  Area: v2 volume backup and full restore
  Issues

• If labeled: require/LEP Has the Longhorn Enhancement Proposal PR submitted?
  The LEP PR is at

#6480

• If labeled: area/ui Has the UI issue filed or ready to be merged (including backport-needed/*)?
  The UI issue/PR is at

• If labeled: require/doc Has the necessary document PR submitted or merged (including backport-needed/*)?
  The documentation issue/PR is at

[yangchiu]

Verified passed on master-head (longhorn-manager 970ba44) following the test plan.

[khushboo-rancher]

@yangchiu Have we tested the below scenarios:
Backup:

1. Interrupting the v2 backup by deleting the replica responsible for backup when backup is in progress.
2. Taking two v2 volumes's backups together.
3. Triggering two v2 backups from two snapshots of same volume together.
4. v2 Backup creation and backup deletion at same time.
5. V2 Backup on S3 and NFS backupstore.
6. Corrupting the backups on backupstore and verifying the state in Longhorn.
7. Taking Backup after a failed backup creation.
8. Some similar edge cases as above.

Restore:

1. Restoring v1 volume backup crated in 1.5.x to v2 volume.
2. Interrupt v2 restore by crashing the IM or rebooting the attached node.
3. Restoring of v2 volume and deleting the same backup at same time.
4. Consequently restoring after the step3.
5. Restoring backup on another cluster.
6. Randomly delete some blocks on backup in backupstore and try to restore the corrupted backup and restoring of subsequent backup.

[yangchiu]

Backup:

1. Interrupting the v2 backup by deleting the replica responsible for backup when backup is in progress.

#7573 #7574 #7575

2. Taking two v2 volumes's backups together.

Pass

3. Triggering two v2 backups from two snapshots of same volume together.

Pass

4. v2 Backup creation and backup deletion at same time.

Pass

5. V2 Backup on S3 and NFS backupstore.

Pass

6. Corrupting the backups on backupstore and verifying the state in Longhorn.

#7583

7. Taking Backup after a failed backup creation.

Pass

8. Some similar edge cases as above.

Restore:

1. Restoring v1 volume backup crated in 1.5.x to v2 volume.

Pass

2. Interrupt v2 restore by crashing the IM or rebooting the attached node.

#7579 #7581

3. Restoring of v2 volume and deleting the same backup at same time.
4. Consequently restoring after the step3.

#7584

5. Restoring backup on another cluster.

Pass

6. Randomly delete some blocks on backup in backupstore and try to restore the corrupted backup and restoring of subsequent backup.

#7583

[innobead]

@yangchiu For the above tests, if not covered in automation yet, create a ticket for that.

[yangchiu]

Related issues tracked by #6138 (comment). Close this for now.