Limiting mc mirror parallel downloads

[kevinlul]

Expected behavior

Judging from previous issues on this subject, setting GOMAXPROCS=1 should limit the number of parallel downloads mc mirror attempts from the same S3 server.

Actual behavior

mc mirror continues to attempt to download several objects in parallel even with GOMAXPROCS=1 and the number of connections used varies wildly between one and over a dozen. This slows down the download for all the objects it is trying to download when the bottleneck is bandwidth, making it slower than an equivalent rsync, and increases the load on the server unnecessarily.

Steps to reproduce the behavior

1. Create a MinIO server with a bunch of large objects.
2. GOMAXPROCS=1 mc mirror --watch --overwrite --remove --quiet NEWSERVER/bucket localdisk
3. Observe from logs that mc attempts to download several objects at once.

mc --version

mc version RELEASE.2022-05-09T04-08-26Z

System information

CentOS 7

[dormanze]

You can use MC_UPLOAD_MULTIPART_THREADS(default 4) parameter to limit the number of threads , and use MC_UPLOAD_MULTIPART_SIZE(default unlimited) parameter to limit the upload size.

[harshavardhana]

^^ these values are "per" object

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.