Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Uploading to OneDrive Cryptomator vault is extremely slow if there are many files #10849

Closed
cyberduck opened this issue Oct 19, 2019 · 18 comments
Assignees
Labels
enhancement fixed onedrive OneDrive Protocol Implementation
Milestone

Comments

@cyberduck
Copy link
Collaborator

cyberduck commented Oct 19, 2019

ce6d08c created the issue

Hi,

there seems to be a bug in Cyberduck when lots of fairly small files are uploaded to a Cryptomator vault located in OneDrive. I've attached an example. Just unpack it and drag and drop it to the vault. It will be extremely slow.


Attachments

@cyberduck
Copy link
Collaborator Author

cyberduck commented Mar 17, 2020

ce6d08c commented

Hi,

what's the status on this? It still takes a very long time to browse through directories. Please give an update.

Thanks and best regards
rootie

@cyberduck
Copy link
Collaborator Author

cyberduck commented Jul 17, 2020

ce6d08c commented

Hi,

is there any plan in improving this? This is clearly a bug and can most likely fixed fairly easily.

Please give an update.

Thanks and best regards
rootie

@cyberduck
Copy link
Collaborator Author

cyberduck commented Apr 2, 2021

f108a4e commented

Can confirm, super slow, using Windows 10 latest ver Cyberduck 7.8.3 and Mountain duck 4.5.0, I suspect there also is something to do with Ticket [[11582]] because you get rate limited very often uploading. Tried uploading 20,000 small files and it took around 6 hours for even 2000 files to be uploaded! It's quite frustrating to be honest. I hope there is a fix soon. It doesn't look good that this has been open for over 2 years now with no resolution especially after i went out and just purchased this software for basically this reason :/

@cyberduck
Copy link
Collaborator Author

cyberduck commented Apr 2, 2021

ce6d08c commented

Yes unfortunately it seems that the developers are not interested in such things and this is very disappointing. The most disturbing thing is that there's not even a reaction - nothing.

As they call it 'enhancement' it's not declared a bug and hence won't be fixed anytime soon.

@cyberduck
Copy link
Collaborator Author

cyberduck commented Apr 2, 2021

f108a4e commented

Yep, this is absolutely a defect, not a "enhancement". The software is unusable at this moment with OneDrive and basically over 1000 smallish files. Just went back and used the original OneDrive desktop client and Cryptomator desktop and it got my 20,000 files synced within an hour or two while Mountainduck and Cyberduck get stuck uploading for several more hours even if you have as few as 500 files! It's insane, absolutely is an issue with OneDrive syncing. I don't know whether to blame how slow it is to transfer many small files into a cryptomator vault on my awful computer or on Mountain Duck/Cyberduck though, Cryptomator desktop app seems to be blazing fast though. It's so weird. I was trialing this software for 2 weeks, it seemed alright with smaller things but it just can't handle a photo library, and I still went out and bought it today. I really didn't expect it to fail so hard today :/

@cyberduck
Copy link
Collaborator Author

cyberduck commented Apr 8, 2021

@dkocher commented

Related to #10990.

@cyberduck
Copy link
Collaborator Author

cyberduck commented Apr 9, 2021

@dkocher commented

Fix in 5980dd7 with missing cache.

@cyberduck
Copy link
Collaborator Author

cyberduck commented Apr 9, 2021

@dkocher commented

Optimization in 78b0f34 to add newly created directories to cache for later lookup of parent directory id when uploading containing files.

@cyberduck
Copy link
Collaborator Author

cyberduck commented Apr 12, 2021

@dkocher commented

Allow reuse of cached file id for previously created directory in 89770cd.

@cyberduck
Copy link
Collaborator Author

cyberduck commented Apr 12, 2021

@dkocher commented

New Auto default allows up to 20 concurrent transfers for an upload in 013a4b9 to embrace parallelism when uploading many small files. The auto setting for FTP is at 1 connection (in 9a37e24) because FTP is stateful and the number of allowed connections is usually limited by the server.

@cyberduck
Copy link
Collaborator Author

cyberduck commented Apr 12, 2021

@dkocher commented

The same issue affects all protocols where we have a file id to filename mapping.

@cyberduck
Copy link
Collaborator Author

cyberduck commented Apr 29, 2021

@dkocher commented

Optimization to reuse known file id when setting timestamp in 487a309.

@cyberduck
Copy link
Collaborator Author

cyberduck commented Apr 29, 2021

@dkocher commented

Fix mapping of file id in decrypted file reference in c43add0.

@cyberduck
Copy link
Collaborator Author

cyberduck commented Apr 29, 2021

@dkocher commented

Add internal cache in file id providers in d07148a.

@cyberduck
Copy link
Collaborator Author

cyberduck commented Apr 29, 2021

@dkocher commented

Refactored in 77a74af and a18313f.

@cyberduck
Copy link
Collaborator Author

cyberduck commented Apr 29, 2021

@dkocher commented

In b258a73.

@cyberduck
Copy link
Collaborator Author

cyberduck commented Apr 29, 2021

@dkocher commented

Milestone renamed

@cyberduck
Copy link
Collaborator Author

cyberduck commented Apr 29, 2021

@dkocher commented

Replying to [comment:7 panrag227]:

Please try the current snapshot build by updating from within Cyberduck in Preferences → Update → Automatically check for updates in → Snapshot Builds.

@iterate-ch iterate-ch locked as resolved and limited conversation to collaborators Nov 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement fixed onedrive OneDrive Protocol Implementation
Projects
None yet
Development

No branches or pull requests

2 participants