Large uploads requiring checksum of parts fail with network timeout

[cyberduck]

e8c8a40 created the issue

Hello We have had this problem, since the release of 6.8.0 (it was fine on 6.7.x) and 6.8.1 or 6.8.2 updates did not fix it.

When we try to upload a large file to S3 Share, the Checksum gets computer at the begining - that takes a really long time. (but that is how its always been) now when the actuall transfer begins, all the CPU's on the computer are pegged out at 100% and then the upload eventually fails.

The interesting thing is that it happens on Macs and PC's , i tried to test this on multiple computers. (same or similar issue is happening)

If I can help in any way to try to send some logs, i could do that.

Thank you very much for looking at this issue.

-Marek

---

Attachments

• Cyberduck_2018-10-23-114719_MBABALA-MAC.cpu_resource.diag (6.7 KiB)
• Screen Shot 2018-10-30 at 3.18.41 PM.png (177.7 KiB)
• Screen Shot 2018-10-30 at 3.19.46 PM.png (208.2 KiB)

[cyberduck]

@dkocher commented

Can you please attach the error message displayed.

[cyberduck]

e8c8a40 commented

I forgot to add that the transfer speeds are slowing down from about 45 MB/sec down to 5-6.5 MB/sec and then it fails. This is very easy reproducable, the files 2GB or less will transfer okay , (but still with visible slow down toward the end of the transfer ) ..... anything larger than 2GB, its pretty upsetting how it slows down and eventually slow down.

Thank you for looking into this,

Marek

[cyberduck]

@dkocher commented

[cyberduck]

@dkocher commented

For the issue that MD5 checksum calculation is taking a long time and could be skipped for multipart uploads refer to #10278.

[cyberduck]

@dkocher commented

Regression from 1ab744c which decreases performance significantly when skipping from the file input stream which is used extensively when calculating checksums for parts of a multipart upload.

[cyberduck]

@dkocher commented

In 4f64fbe.

[cyberduck]

@dkocher commented

#10491 closed as duplicate.

[cyberduck]

@dkocher commented

#10492 closed as duplicate.

[cyberduck]

@dkocher commented

#10465 closed as duplicate.

[cyberduck]

@dkocher commented

#10485 closed as duplicate.

[cyberduck]

e8c8a40 commented

David,

Again, thanks for taking a look at this. I noticed that you closed this ticket, as it did resurface previously (large data fail upload to S3) ... Is there something you would like me to do? Test it once you guys release 6.8.3? or test it with a nightly update? Not rushing or anything. Just offering to do more testing.

Thank you,
Marek

[cyberduck]

@dkocher commented

#10429 closed as duplicate.

[cyberduck]

@dkocher commented

Replying to [comment:15 Chicago_Marek]:

David,

Again, thanks for taking a look at this. I noticed that you closed this ticket, as it did resurface previously (large data fail upload to S3) ... Is there something you would like me to do? Test it once you guys release 6.8.3? or test it with a nightly update? Not rushing or anything. Just offering to do more testing.

Thank you,
Marek
Please update to the latest beta build available. Version 6.8.3 will be released tomorrow.