[BUG] Add Pagination support to the S3 client (currently we only list the first 1000 results)

[joshimoo]

We need to add pagination support to the S3 client, currently we only list the first 1000 results.
We use the list calls for checking the backups/volumes/blocks currently we get lucky that we
use partitions for volumes and blocks.

But even so it is still possible to have more then 1000 backup.cfs as well as blocks/volumes in the same partition.
Which would end up not showing up on the longhorn-ui or any logic that relies on this content.

To Reproduce

1. Open browser developer tool.
2. Create one snapshot by clicks LH GUI (It'll calls snapshotCreate and snapshotBackup APIs).
3. Copy the snapshotBackup API call, right click Copy -> Copy as cURL.
4. Run the curl command over 1k times in the Shell.
5. On LH GUI, Clicks the Backup -> PVC Name to display total PVC backups, you could see that only 100 pages (10 backups/page) are available even we take over 1k backups.

[joshimoo]

Maybe cause of #1667

[timmy59100]

Is this also the cause of this error in the backup UI?
error listing backups: error listing backup volumes: Timeout executing: /var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-v1.0.0/longhorn [backup ls --volume-only s3://PATH/], output , stderr, , error <nil>
Seems only the case if there's lots of data in the bucket, currently 2TB.

[yasker]

@timmy59100 Yes it's possible. This command needs to go to each level of the backup directory for iteration, so if there are lots of volumes, it's possible didn't get the response in time. Another potential issue is the S3 vendor might respond late. Which S3 vendor you're using?

[timmy59100]

@yasker we use Exoscale

Is there a way to increase the timeout?

[yasker]

@timmy59100 Not currently. Can you file another issue regarding the timeout? Also, how many volumes you have? We want to see what we can do to address it in the v1.1.0 release.

[longhorn-io-github-bot]

Pre-merged Checklist

• Is the reproduce steps/test steps documented?

• Is there a workaround for the issue? If so, is it documented?

• Does the PR include the explanation for the fix or the feature?

• Does the PR include deployment change (YAML/Chart)? If so, have both YAML file and Chart been updated in the PR?

• Is the backend code merged (Manager, Engine, Instance Manager, BackupStore etc)?
  The PR is at

• Support s3 pagination backupstore#61
• Downgrade to use v1 ListObjects backupstore#62
• Support s3 pagination longhorn-engine#559
• Support s3 pagination longhorn-manager#821

• Which areas/issues this PR might have potential impacts on?
  Area: GUI -> List Backups
  Issues: List Backups timeout

• If labeled: require/LEP Has the Longhorn Enhancement Proposal PR submitted?
  The LEP PR is at

• If labeled: area/ui Has the UI issue filed or ready to be merged?
  The UI issue/PR is at

• If labeled: require/doc Has the necessary document PR submitted or merged?
  The Doc issue/PR is at

• If labeled: require/automation-e2e Has the end-to-end test plan been merged? Have QAs agreed on the automation test case?
  The automation skeleton PR is at
  The automation test case PR is at

• If labeled: require/automation-engine Has the engine integration test been merged?
  The engine automation PR is at

• If labeled: require/manual-test-plan Has the manual test plan been documented?
  The updated manual test plan is at

• If the fix introduces the code for backward compatibility Has an separate issue been filed with the label release/obsolete-compatibility?
  The compatibility issue is filed at

[khushboo-rancher]

Verified with the Longhorn Master - 02/18/2021

Validation - Pass

Validated with S3 backupstore, more than 1000 backups are getting listed. Also, verified with NFS backupstore, there is no impact.

The timeout issue during backup is being tracked and will be reproduced and validated as a part of #2218