Compose segments of download very slow after downloading is slow

[yardik]

After downloading a multi-segment download of large size (200GB etc.) the recompositing of the file is extremely slow, like hours, longer than the download. I see this was closed previously with a 'don't use segments' workaround, but doing so trashes your ability to

• get maximum download speed
• effectively resume download, by resuming missing segments

I don't think the workaround solves the problem.

I'd be happy to contribute a fix for this, if someone can point me towards the area of the codebase that functionally recreates the downloaded file from the multiple chunks.

Thanks!

[dkocher]

Duplicate for #10961.

[dkocher]

You can find the code in

cyberduck/core/src/main/java/ch/cyberduck/core/transfer/download/AbstractDownloadFilter.java

Lines 270 to 300 in 2b1b543

	if(status.isComplete()) {
	if(status.isSegmented()) {
	// Obtain ordered list of segments to reassemble
	final List<TransferStatus> segments = status.getSegments();
	if(log.isInfoEnabled()) {
	log.info(String.format("Compile %d segments to file %s", segments.size(), local));
	}
	if(local.exists()) {
	local.delete();
	}
	for(Iterator<TransferStatus> iterator = segments.iterator(); iterator.hasNext(); ) {
	final TransferStatus segmentStatus = iterator.next();
	// Segment
	final Local segmentFile = segmentStatus.getRename().local;
	if(log.isInfoEnabled()) {
	log.info(String.format("Append segment %s to %s", segmentFile, local));
	}
	segmentFile.copy(local, new Local.CopyOptions().append(true));
	if(log.isInfoEnabled()) {
	log.info(String.format("Delete segment %s", segmentFile));
	}
	segmentFile.delete();
	if(!iterator.hasNext()) {
	final Local folder = segmentFile.getParent();
	if(log.isInfoEnabled()) {
	log.info(String.format("Remove segment folder %s", folder));
	}
	folder.delete();
	}
	}
	}

If you can find a better performing implementation that would certainly be welcome.

[yardik]

@dkocher Awesome thanks! Channels and/or buffered writer instead of byte stream are both quite a bit faster. I'm trying to get cyberduck to build locally, once I have, I'll put something together.