Using the CURLOPT_MAX_RECV_SPEED_LARGE option with either libevent or libuv cause downloads to be interrupted

[r-cyr]

I'm working on a project involving libuv + libcurl to do HTTP downloads and it works really well until I try to enable download throttling using the CURLOPT_MAX_RECV_SPEED_LARGE option. When I do that, depending on the speed I set, the download will either never start or stall in the middle.

I thought something was wrong with my code until I downloaded the hiperfifo.c (which uses libevent) and the multi-uv.c (libuv) examples and was able to reproduce the same problem by adding the CURLOPT_MAX_RECV_SPEED_LARGE option.

I was able to reproduce the issue with libcurl 7.48 on Linux and OSX.

[jay]

make sure you are passing a curl_off_t type
curl_easy_setopt(curl, CURLOPT_MAX_RECV_SPEED_LARGE, (curl_off_t)123);

[r-cyr]

Adding the following line to the multi-uv.c example makes it download of small chunk of the target resource then it gracefully exits without error or anything:
curl_easy_setopt(handle, CURLOPT_MAX_RECV_SPEED_LARGE, (curl_off_t)500 * 1024);

[bagder]

We don't special-case that option for the multi socket API so it works the same as otherwise. And it should just remove the socket as one to monitor for actions and then put it back again when it is below the threshold again.

What does libcurl tell your socket callback for the socket you have rate limited like this?

[r-cyr]

My socket handler gets called with a CURL_POLL_REMOVE action then it never gets called again unless I call curl_multi_socket_action(handle_, CURL_SOCKET_TIMEOUT, 0, &running_handles); when it happens. Is that normal?

[bagder]

Surely there's also a timeout (call to the timer function) done before or after that so you will libcurl again at least then, right?

[r-cyr]

Ok, I managed to fix my problem, here is what I did to make it work:

1. I call uv_timer_stop in curl_perform() only when no running handles remain instead of everytime.
2. I added the following two options to my easy handles

• curl_easy_setopt(easy_handle, CURLOPT_LOW_SPEED_TIME, 2L);
• curl_easy_setopt(easy_handle, CURLOPT_LOW_SPEED_LIMIT, 10000L);

Does that even make sense?

Thanks for you patience and your help :)

[zjx20]

Thanks @r-cyr! I am using libuv and encounter the exact problem you described. After applying the step 1 of your solution, the problem seems to be gone. So is the step 2 still necessary?

[r-cyr]

If step 1 alone makes it work then no need for the step 2. I found that these settings worked well for me by testing the bandwidth throttling at different speed, initially using a high-speed local server and then with dummy files from speed tests.

[zjx20]

After testing more cases, I found the solution doesn't always work well. For example, if I limit the speed to 400KB/s, the test case passes. But it fails if I raise the limitation to 700KB/s.
I am also using a high-speed local server for the test.

[zjx20]

I think there is something wrong in either the multi-uv.c example or libcurl itself, but I don't have the ability to figure it out. @jay @bagder could you please take a look?

Steps to reproduce:

1. Save the code https://gist.github.com/zjx20/fda1da4b71f206cd6cadf234e105c55e as multi-uv.c
2. Prepare libcurl 7.50.0 and libuv 1.9.1 (brew install libuv)
3. Compile the code: gcc -lcurl -luv multi-uv.c -o multi-uv
4. Create a dummy file and run http server locally: mkfile -n 5m sparseFile && python -m SimpleHTTPServer
5. Run multi-uv: ./multi-uv http://127.0.0.1:8000/sparseFile
6. The program should exit immediately. The downloaded file should be named "1.download", but the file size is 98304 while 5242880 is expected.

Now, remove the line curl_easy_setopt(handle, CURLOPT_MAX_RECV_SPEED_LARGE, (curl_off_t)500 * 1024); in the code, and try the steps again. You will see at this time the downloaded file is correct.

[bagder]

If you have an issue with a current release of libcurl, then I suggest you post to the curl-library mailing list or create a new issue. Just adding comments to a since-long closed issue will vanish in the general flood of noise.

[zjx20]

Reference #942