Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extremely slow S3 downloads of large files #11833

Closed
cyberduck opened this issue Oct 2, 2021 · 1 comment
Closed

Extremely slow S3 downloads of large files #11833

cyberduck opened this issue Oct 2, 2021 · 1 comment

Comments

@cyberduck
Copy link
Collaborator

@cyberduck cyberduck commented Oct 2, 2021

4d59c66 created the issue

For the second time in a few weeks I've faced massive delays in download large files from an S3 bucket.

In the latest case it's a single 220GB video file. In another case it was a number of larger video clips totaling 150GB.

The actual download proceeds at network speed (Gigabit Internet), however after the last segment has been fetched Cyberduck sits for hours at 100% while it's assembling the segments into a single file. Despite having a very fast RAID (800MB/s transfer rates).

It seems it shouldn't take that long if the code were to simply read each segment and concatenate them through file I/O. I'm assuming there must be some inefficiency as the only reason why this exponentially slows down with file size. Maybe the target file gets written over and over again? Or the copy buffer size is very small which penalizes spinning disk with lots of seeks as it's copying from/to the same disk (segments are located in temp subfolder).

In this last download it created 104 2GB segments. The last segment (104) completed download at 10:37AM. It's now 12:19PM and the new combined file still is only 109GB (about 50% of the total).

That practically makes Cyberduck unusable for files like this.

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Oct 15, 2021

@dkocher commented

Duplicate for #10961. Uncheck the use of segmented downloads in Preferences.

@iterate-ch iterate-ch locked as resolved and limited conversation to collaborators Nov 27, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

1 participant