Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Large uploads requiring checksum of parts fail with network timeout #10516

Closed
cyberduck opened this issue Oct 23, 2018 · 13 comments
Closed

Large uploads requiring checksum of parts fail with network timeout #10516

cyberduck opened this issue Oct 23, 2018 · 13 comments

Comments

@cyberduck
Copy link
Collaborator

@cyberduck cyberduck commented Oct 23, 2018

e8c8a40 created the issue

Hello We have had this problem, since the release of 6.8.0 (it was fine on 6.7.x) and 6.8.1 or 6.8.2 updates did not fix it.

When we try to upload a large file to S3 Share, the Checksum gets computer at the begining - that takes a really long time. (but that is how its always been) now when the actuall transfer begins, all the CPU's on the computer are pegged out at 100% and then the upload eventually fails.

The interesting thing is that it happens on Macs and PC's , i tried to test this on multiple computers. (same or similar issue is happening)

If I can help in any way to try to send some logs, i could do that.

Thank you very much for looking at this issue.

-Marek


Attachments

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Oct 26, 2018

@dkocher commented

Can you please attach the error message displayed.

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Oct 30, 2018

e8c8a40 commented

I forgot to add that the transfer speeds are slowing down from about 45 MB/sec down to 5-6.5 MB/sec and then it fails. This is very easy reproducable, the files 2GB or less will transfer okay , (but still with visible slow down toward the end of the transfer ) ..... anything larger than 2GB, its pretty upsetting how it slows down and eventually slow down.

Thank you for looking into this,

Marek

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Oct 31, 2018

@dkocher commented

Screen Shot 2018-10-30 at 3.18.41 PM.png

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Oct 31, 2018

@dkocher commented

For the issue that MD5 checksum calculation is taking a long time and could be skipped for multipart uploads refer to #10278.

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Oct 31, 2018

@dkocher commented

Regression from 1ab744c which decreases performance significantly when skipping from the file input stream which is used extensively when calculating checksums for parts of a multipart upload.

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Oct 31, 2018

@dkocher commented

In 4f64fbe.

@cyberduck cyberduck closed this Oct 31, 2018
@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Oct 31, 2018

@dkocher commented

#10491 closed as duplicate.

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Oct 31, 2018

@dkocher commented

#10492 closed as duplicate.

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Oct 31, 2018

@dkocher commented

#10465 closed as duplicate.

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Oct 31, 2018

@dkocher commented

#10485 closed as duplicate.

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Oct 31, 2018

e8c8a40 commented

David,

Again, thanks for taking a look at this. I noticed that you closed this ticket, as it did resurface previously (large data fail upload to S3) ... Is there something you would like me to do? Test it once you guys release 6.8.3? or test it with a nightly update? Not rushing or anything. Just offering to do more testing.

Thank you,
Marek

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Oct 31, 2018

@dkocher commented

#10429 closed as duplicate.

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Oct 31, 2018

@dkocher commented

Replying to [comment:15 Chicago_Marek]:

David,

Again, thanks for taking a look at this. I noticed that you closed this ticket, as it did resurface previously (large data fail upload to S3) ... Is there something you would like me to do? Test it once you guys release 6.8.3? or test it with a nightly update? Not rushing or anything. Just offering to do more testing.

Thank you,
Marek
Please update to the latest beta build available. Version 6.8.3 will be released tomorrow.

@iterate-ch iterate-ch locked as resolved and limited conversation to collaborators Nov 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants