speed-limit and Current Speed problems #1556
This pull-request is not complete, but I'm not sure how you want me to share my work exactly,
I've worked out how to make a test in your system, and I've added 2 tests that begin to demonstrate the problems with the speed-limit calculation.
Daniel has started work on improving the algorithm, but I believe the problem runs deeper, in particular the current-speed calculation and the speeder array.
I have a long-winded email draft that I'm putting my thoughts into, but its not finished, so I thought I'd submit these 2 tests to the public to start the conversation rolling.
To run them, clone the repo and checkout this test branch
And to see whats going on, in one terminal:
And in another terminal, run the test that will pass.
Note that the "Current" speed can go above 200 bytes/sec, so there is a problem in the math there.
And in another terminal, run the test that will pass. It has a long long command line due to the long post-form-data that is delivered first.
In this case, the Current speed drops to zero because the initial upload data biases the current speed calculation, and the speeder array updates will ignore all the downloaded data progress until the total downloaded bytes is more than the total uploaded bytes. Not a good approach.
Note that I don't like the speeder array approach much, I think there are problems with the fixed 6 slot history, and the fact that the speeder array can be updated on occasions (by Curl_pgrsUpdate) when the Curl_speedcheck() function is not subsequently called.
But thats part of the long winded email, I'm going to try to propose a better approach.
Here is the long command for test 3000:
Nah, that's really stupid. I think the "current speed" should be the maximum speed in either direction for this to work properly. Possibly it should be both directions?
I think we can fix this pretty neatly by moving the call to
I haven't checked what you've done yet, but I wanted to fire off an email while you are online.
I was thinking about the approach to solve, and I was thinking...
Allow Curl_speedcheck to remember the last total transferred bytes (up+down) and the last "passed threshold time". I don't see why we should use any of the update's transfer speeds at all.
WHEN TO CALL SPEEDCHECK
Curl_pgrsUpdate is called in about 20 different places, but Curl_speedcheck is called in only < 8 places.
Instead of speeder, perhaps we should just be adding the dl and ul speeds together?
Its not perfect, I need to improve it a bit,
Then we would have a "total dl speed" (the Curl dlspeed as its implemented now)
Rather than watching the computed current average speed, which is not as specific as a simple total over a time window.
I've updated this with my suggested implementation.
I'll have to leave that to you to determine what is the best course of action in that regard.
At the moment, I figure it'll "be ok" since it'll "probably" get called as often as required, but I can't be sure.
Regarding the definition, the "R bytes/sec over T time" is a bit confusing.
If I had a time machine, I'd suggest just using "N bytes over T time".
At the moment, the minimum rate I can specify over a 60 second window is 1 byte/sec.
Or 100 bytes / 2 minutes ? Minimum rate over 2 minutes possible is 120 bytes (1bps)
You can see someone had the same confusion here (I'm not involved in this project, I just found it via search):
The reason I'm using the low-speed limit is actually to avoid using timeouts in longpoll connections, while at the same time watching for broken connections that Curl doesn't detect (by design, it seems?).
Is the above the correct design, from Curl's point of view?
Sorry if this is polluting this discussion...
Another question... see here:
In short, they are streaming video. They pause the video (or the buffer becomes full) so they throttle the network speed.
In this scenario, shouldn't the Curl user adjust the low-speed-limit to match the throttled speed desired?
And if so, CAN the Curl user adjust the low-speed-limit as the code currently stands?
Bug #1556 Reported-by: Paul Harris
First, I think we should focus on getting the existing functionality done right. I don't think it is valuable to scan a lot of random misunderstandings or others repeating our bugs before then. Once we believe we have ironed out our current problems, then I think we can start to see what we can do for the remaining problems (if possible).
The low speed timeout was never meant to be the all-covering solution for everyone. It is just one way for applications to kill almost idle connections. Applications can and should still implement their own logic if they want more fancy algorithms.
Those first 20 seconds are easy - but what happens on the 21st second? Do you restart the counter at second 20 and wait another 20 seconds so it won't abort this connection until second 40?
By checking the speed every second, the situation where 2000 bytes come the first second and is then followed by 19 totally stalled seconds is detected at second 21 where it actually reaches 20 seconds in a row at less than 100 bytes/sec...
It does as you say, If in the first 1 second, 2000 bytes is received, then it is reset at that 1st second, and then at the 21st second it is deemed too slow. If it receives 1999 bytes in the first second, and no more for 19 more seconds, then it too slow at the 20th second. If it receives 1999 bytes in the 1st second, and 1 byte in the 20th second, it is reset there. Then it takes 20 more seconds to be too slow. That is bad. Would be better to be deemed too slow in the 21st second. A speeder array would work, with a slot for every second in the window. If the user requests a huge window, then will require a huge array. But not that huge. You could store the log of number of bytes in each cell (1 char per cell for 1e255 Max bytes for example), so it can inaccurately but reasonably track the data volume even if we get a huge chunk in one second. 1 byte per second window. Alternatively, Not sure if the maths I suggested would work perfectly for this situation. But could probably do it this way. Hmm.. Other ideas?…
On 9 Jun 2017 5:25 PM, "Daniel Stenberg" ***@***.***> wrote: First, I think we should focus on getting the existing functionality done right. I don't think it is valuable to scan a lot of random misunderstandings or others repeating our bugs before then. Once we believe we have ironed out our current problems, *then* I think we can start to see what we can do for the remaining problems (if possible). The low speed timeout was never meant to be the all-covering solution for everyone. It is just one way for applications to kill almost idle connections. Applications can and *should* still implement their own logic if they want more fancy algorithms. So, if LIMIT=100 bytes/sec, TIME=20 sec, then THRESHOLD=100x20=2000 bytes must be transferred within a 20 sec window. Those first 20 seconds are easy - but what happens on the 21st second? Do you restart the counter at second 20 and wait another 20 seconds so it won't abort this connection until second 40? By checking the speed every second, the situation where 2000 bytes come the first second and is then followed by 19 totally stalled seconds is detected at second 21 where it actually reaches 20 seconds in a row at less than 100 bytes/sec... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1556 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABkgS_eoqtId3Q4v4mq69Owqr5K_KnOiks5sCQ-QgaJpZM4NzxdY> .
I'm not sure what you're suggesting. If we ignore the problem #1559 is trying to fix, what is this new code bringing to libcurl that the existing code doesn't already handle?
I think the current code has several advantages compared to your version, including that the "current speed" is based on the last 5 seconds only. Limiting the speed math to the last N seconds is hard to do without using an array like this.
In short, I think your changes in #1559 are ok,
N = average num bytes/sec lower limit
The current description implies (to me):
Over T seconds, the connection should be operating with an average speed of N bytes/sec.
The description should be changed to something more like...
The connection is defined as too slow if its total speed is lower than N bytes/sec averaged over a 5 second window.
Alternatively: The connection will be aborted (due to low speed) if it T seconds pass without 5*N bytes uploaded or downloaded within a 5 second window.
Ok, so the current speed is based on the total transferred in the last 5 seconds (divided by 5).
Then, Curl_speedcheck() should be called at least once a second (right?)
You seem to be hung up and not agree that we need more than one second to get an accurate "current speed". I don't think the fact that it uses the last 5 seconds to get that information is something we need to hammer the user with, it is just an implementation detail for how it gets the current speed (I'm concerned that using less will get very unstable number and thus instead cause other issues). And it uses less than 5 seconds the first 5 seconds of a transfer. Therefore, i don't think your suggested updated descriptions improve matters very much.
it should. But it is a fairly new requirement so I won't be surprised if there's some case where it doesn't happen. We should probably add a debug-build condition that checks and warns if that happens. Or have you figured out any situation where you know it happens?
I agree we need more than one second, that's logical.
My confusion (and I think others) comes from the wording of what an "average speed" is. Current description implies that the average speed over the whole time window should be above the limit.
That isn't the case.
The case is that there only needs to be one second in the time window where the "current speed" is above the limit.
Regarding the question regarding whether speedcheck is always called every second, I was thinking of adding some checks, ie when prgsupdate is called AND the speeder is updated, has speedcheck been called since the last speeder update?
#1567 replaces it, in terms of the commit.
By the way, you are doing a great job with such a complex library.
Due to the slight shift in focus from this PR, I think that's better taken to a new PR/issue. Makes it easier for everyone to keep track of the exact topic.