-
Notifications
You must be signed in to change notification settings - Fork 873
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add note about concurrency setting to throttle-upload setting #5137
base: master
Are you sure you want to change the base?
Add note about concurrency setting to throttle-upload setting #5137
Conversation
Can you provide details of your testing? It looks like the intention was to scale the throttle down by the concurrency number: duplicati/Duplicati/Library/Main/Operation/Backup/BackendUploader.cs Lines 136 to 137 in b0e852d
duplicati/Duplicati/Library/Main/Operation/Backup/BackendUploader.cs Lines 490 to 495 in b0e852d
although a developer should probably render an opinion on whether the code is correct, and whatever subtle things it's doing. |
Confirming that something isn't working as the above code chunks seem to intend. I backed up a 300 MB source file to local folder with throttle-upload=1MB/sec. At asynchronous-concurrent-upload-limit=1, total time was 310 seconds (seems right), however at asynchronous-concurrent-upload-limit=10, time was 73 seconds. If that's a bug (need a developer), maybe a code fix will be better. |
It still looks like a bug to me. Thanks for finding it. Would you like to file an issue saying how you tested? If not, I suppose I could. I was watching file writes with Sysinternals Process Monitor. At the higher concurrencies it seems to just blast out a whole dblock. When throttling actually works, you can see it pausing between writes. This is all it can do -- it can't finely control the output rate. These tests have been done with a 10 MB remote volume size. One can also study a log using a tag filter or a profiling log to see:
type lines which track individual files, and are certainly far off from 100 Kbits/second. |
I think the intention is that the throttle upload should be observed across all threads. However, the implementation only looks at a single stream and limits this to a fraction of the total possible uploads. A better solution would be to implement a shared throttle mechanism that has some kind of fair distribution of the allowed bandwidth. Looking at the measurements done by @ts678 it seems that the throttling does work for some settings. I think there might be some lower-case limit that we hit with divided that causes unexpected throttle values to be effected. |
In my opinion it's a bug, although I hope I didn't just test it badly. You can see if you can reproduce the odd results that I got, possibly testing over some network destination. Possibly the workaround still helps, but the help doesn't usually cover those... |
I certainly got similar results as yours. Setting If you think the addition to the documentation is still useful we can keep this issue. I'd still create a separate issue for the problem with |
@bjoernmartin I agree. It appears the documentation is as intended, but the implementation is not doing a great job observing the limit. |
I had trouble getting the upload throttle to work. The forum post https://forum.duplicati.com/t/upload-throttle-not-working/8955/7 helped me. So I thought that information could also go into the application's documentation.
If this is accepted / acceptable I can contribute this change to the documentation as well.