[FEATURE] New setting to limit the concurrent volume restoring from backup

[c3y1huang]

Is your feature request related to a problem? Please describe

We like to support to rolling out the volume with data from backup/backing-image. However, Longhorn has no limitation on the concurrent volume restoring from backup.

Longhorn should have another setting similar to concurrent-replica-rebuild-per-node-limit.

Describe the solution you'd like

Introduce a new setting to limit the concurrent volume creating from the backup.

Describe alternatives you've considered

Handle in rollout-load controller.

Additional context

#4388 (comment)

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at:

• Is there a workaround for the issue? If so, where is it documented?
  The workaround is at:

• Does the PR include the explanation for the fix or the feature?

• Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
  The PR for the YAML change is at: N/A
  The PR for the chart change is at:

  • feat(concurrent-backup-restore): deploy update #4992

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at:

  • feat(concurrent-backup-restore): implement longhorn-manager#1583

• Which areas/issues this PR might have potential impacts on?
  Area manager, backup
  Issues

• If labeled: require/LEP Has the Longhorn Enhancement Proposal PR submitted?
  The LEP PR is at:

  • feat(lep): concurrent backup restore design #4991

• If labeled: area/ui Has the UI issue filed or ready to be merged (including backport-needed/*)?
  The UI issue/PR is at

• If labeled: require/doc Has the necessary document PR submitted or merged (including backport-needed/*)?
  The documentation issue/PR is at:

  • doc(concurrent-backup-restore): add setting website#625

• If labeled: require/automation-e2e Has the end-to-end test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue (including backport-needed/*)
  The automation skeleton PR is at:

  • test(auto): setting concurrent-backup-restore-limit longhorn-tests#1216
    The automation test case PR is at:
  • test(auto): setting concurrent-backup-restore-limit longhorn-tests#1216
    The issue of automation test case implementation is at:
  • [TEST][FEATURE] New setting to limit the concurrent volume restoring from backup. #4993

• If labeled: require/automation-engine Has the engine integration test been merged (including backport-needed/*)?
  The engine automation PR is at

• If labeled: require/manual-test-plan Has the manual test plan been documented?
  The updated manual test plan is at

• If the fix introduces the code for backward compatibility Has a separate issue been filed with the label release/obsolete-compatibility?
  The compatibility issue is filed at

[chriscchien]

Hi @c3y1huang

I have a 3 nodes cluster, 8 backups from different volumes and Longhorn set Concurrent Volume Backup Restore Per Node Limit to 1
After I select all backups in backup page then click Restore latest Backup
The concurrent restore number somehow will greater then 3 as below picture shown.

In addition, I had a simple test script to calculate rebuilding numbers from backend API, when restoring, somehow it will give me number greater than 3 which reflected the situation as UI displayed.

def test_count_restore_limit(client):

    for i in range(1000):
        try:
            restoring = 0
            volumes = client.list_volume()
            for v in volumes:
                if v.restoreStatus and v.restoreStatus[0].progress != 0 and v.restoreStatus[0].progress != 100:
                    restoring = restoring + 1
                else:
                    pass
        except:
            print("x")
            pass

        print(restoring)
        time.sleep(1)

[chriscchien]

Did several times of restoring volume with 1 replica and Concurrent Volume Backup Restore Per Node Limit set to 1

At begging the restore number = 3 matched the setting. But there still have chance that the restoring volume number surpassed Concurrent Volume Backup Restore Per Node Limit value, not every time, but not very hard to reproduce ( Can reproduce every time when restoring volume with 3 replicas)

In addition, last success build of longhorn-engine was 6 days ago, not sure if this impacted

[chriscchien]

Verified in longhorn master aa3998 with steps
Result Pass

• Environment:
  • 3 nodes cluster
  • 8 backups from different volume, each size were 2Gi

1. The setting won't effect DR volumes, no matter the setting is 0 or not, all DR volumes were restoring at the same time.
2. Set concurrent-volume-backup-restore-per-node-limit to 0, then restore all volumes, can see volume attach to nodes in maintenance mode and restoring never started.
3. Set concurrent-volume-backup-restore-per-node-limit to 1, can observe volume start restoring and each node's restoring number not exceed the setting.
4. Also tested concurrent-volume-backup-restore-per-node-limit = 2 , restore volume with 1 and 3 replicas, all worked well, the restoring number not exceed setting per node.