Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support parallel uploads for a backend [$381] #2026

Open
wvc opened this Issue Oct 18, 2016 · 6 comments

Comments

Projects
None yet
8 participants
@wvc
Copy link

wvc commented Oct 18, 2016

Some of the cloud backends (e.g. Dropbox) seem bandwidth limited by the cloud provider for individual connections (upload/download). One way to achieve higher throughput would be to upload backup volumes in parallel. Obviously this only works if there is more than 1 backup volume to be uploaded.
The same approach would also be possible for achieving higher throughput on downloads.

However, an important implication for the backend implementations is that they should be thread-safe , i.e. calling PUT should be possible in parallel. Perhaps, an additional interface could be defined, IThreadSafeBackend or something, that IBackends can implement for indicating that they support parallel upload/download operations.

--- There is a **[$50 open bounty](https://www.bountysource.com/issues/38460805-support-parallel-uploads-for-a-backend?utm_campaign=plugin&utm_content=tracker%2F4870652&utm_medium=issues&utm_source=github)** on this issue. Add to the bounty at [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F4870652&utm_medium=issues&utm_source=github).

@kenkendk kenkendk changed the title Support parallel uploads for a backend Support parallel uploads for a backend [$50] Nov 3, 2016

@kenkendk kenkendk added the bounty label Nov 3, 2016

@hofesh

This comment has been minimized.

Copy link

hofesh commented Sep 2, 2017

any progress on parallel uploads? using a single thread is really slow for Backblaze B2

@Scarfmonster

This comment has been minimized.

Copy link

Scarfmonster commented Mar 13, 2018

Yes, this would be a much needed feature, I want to switch to B2 but currently Duplicati is so slow there's no point. If I take the last time Duplicati was compacting into account the total run time for me would go from 4 hours to over 12 to download and reupload everything. And 4 hours still is still over 2 times longer than it should take on my connection.

Also, the bounty on this currently is $165, it may be worth updating the title to reflect that @kenkendk

@JonMikelV JonMikelV changed the title Support parallel uploads for a backend [$50] Support parallel uploads for a backend [$165] Mar 14, 2018

@studersi

This comment has been minimized.

Copy link

studersi commented Mar 28, 2018

+1

Bounty is at $196 now btw.

@wvc

This comment has been minimized.

Copy link
Author

wvc commented Mar 28, 2018

Original reporter of this issue/feature request here . I took a brief look at handling this issue. My conclusion: it is not an easy extension, I think it requires a complete rewrite of the BackendManager and that possibly has repercussions on the rest of Duplicati. Not sure if anyone is up for such a daunting task.

@JonMikelV JonMikelV changed the title Support parallel uploads for a backend [$165] Support parallel uploads for a backend [$231] Apr 21, 2018

@djtm

This comment has been minimized.

@mikaelmello mikaelmello changed the title Support parallel uploads for a backend [$231] Support parallel uploads for a backend [$381] Nov 12, 2018

@Marv2190

This comment has been minimized.

Copy link

Marv2190 commented Jan 26, 2019

Someone with multi WANs here. Also the parallel uploading would nicely split on my WANs (Load balancing ) and it would drastically improve my uploads speed!
As Duplicati is spliting or combine my files into 100MB Blocks and Uploads then, it would be nice if in the time a 100MB Block is uploading the next is combine and also send. (In the preferences you can choose how many Blocks you want to upload at the same time).

@seantempleton seantempleton referenced a pull request that will close this issue Mar 6, 2019

Open

Implement parallel uploads to a single backend #3684

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.