Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compose segments of download very slow after downloading is slow #12996

Closed
yardik opened this issue Mar 21, 2022 · 3 comments · Fixed by #13000
Closed

Compose segments of download very slow after downloading is slow #12996

yardik opened this issue Mar 21, 2022 · 3 comments · Fixed by #13000
Milestone

Comments

@yardik
Copy link

@yardik yardik commented Mar 21, 2022

After downloading a multi-segment download of large size (200GB etc.) the recompositing of the file is extremely slow, like hours, longer than the download. I see this was closed previously with a 'don't use segments' workaround, but doing so trashes your ability to

  • get maximum download speed
  • effectively resume download, by resuming missing segments

I don't think the workaround solves the problem.

I'd be happy to contribute a fix for this, if someone can point me towards the area of the codebase that functionally recreates the downloaded file from the multiple chunks.

Thanks!

@dkocher
Copy link
Contributor

@dkocher dkocher commented Mar 21, 2022

Duplicate for #10961.

@dkocher
Copy link
Contributor

@dkocher dkocher commented Mar 21, 2022

You can find the code in

if(status.isComplete()) {
if(status.isSegmented()) {
// Obtain ordered list of segments to reassemble
final List<TransferStatus> segments = status.getSegments();
if(log.isInfoEnabled()) {
log.info(String.format("Compile %d segments to file %s", segments.size(), local));
}
if(local.exists()) {
local.delete();
}
for(Iterator<TransferStatus> iterator = segments.iterator(); iterator.hasNext(); ) {
final TransferStatus segmentStatus = iterator.next();
// Segment
final Local segmentFile = segmentStatus.getRename().local;
if(log.isInfoEnabled()) {
log.info(String.format("Append segment %s to %s", segmentFile, local));
}
segmentFile.copy(local, new Local.CopyOptions().append(true));
if(log.isInfoEnabled()) {
log.info(String.format("Delete segment %s", segmentFile));
}
segmentFile.delete();
if(!iterator.hasNext()) {
final Local folder = segmentFile.getParent();
if(log.isInfoEnabled()) {
log.info(String.format("Remove segment folder %s", folder));
}
folder.delete();
}
}
}

If you can find a better performing implementation that would certainly be welcome.

@dkocher dkocher closed this as completed Mar 21, 2022
@yardik
Copy link
Author

@yardik yardik commented Mar 21, 2022

@dkocher Awesome thanks! Channels and/or buffered writer instead of byte stream are both quite a bit faster. I'm trying to get cyberduck to build locally, once I have, I'll put something together.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants