When uploading files to an SFTP server, I limited the CURLOPT_MAX_SEND_SPEED_LARGE to 1024, and set the CURLOPT_UPLOAD_BUFFERSIZE to it's minimum of 16kb. When I upload a file of 25855 bytes it finished the first 16384 bytes nearly instantly, then takes some time for the rest. It spends totally 16 seconds uploading something that should take 25855/1024 = 25 seconds give or take. Leaving the CURLOPT_UPLOAD_BUFFERSIZE to the default 64kb makes the entire upload happen instantly.
I also noticed something similar for download as well, if I set CURLOPT_BUFFERSIZE to say 131072 bytes (CURL_MAX_READ_SIZE/4), the entire download happens instantly. If I lowered CURLOPT_BUFFERSIZE to the same as CURLOPT_MAX_RECV_SPEED_LARGE it would take the expected time to download.
I expected the following
I expect that even if the buffer size is larger than the max speed, the max speed would be upheld.
Clearly, when libcurl's been instructed to do the transfer slower (in bytes/sec) than the size of the upload/download buffer, it should rather limit the amount data to handle to the capped amount in order to better acknowledge the setting in the first network read/write.
I think this is just an oversight as it only affects the first network operation and capping network speed has always been about being "roughly right over time", and not an exact science.
I played around with it today, and as far as I can tell, CURL, calls Curl_fillreadbuffer and fills the buffer, then wait for an appropriate about of time so that the limit-rate get correctly averaged, which I suppose work fine in household networks etc.
However, I am dealing with a very limited pipe, of something like 20kb/s on bad days lower, where a 64kb filled buffer stop all communication for 3 seconds. As alluded to in my issue, the temporary fix for this is to set CURLOPT_MAX_SEND_SPEED_LARGE and CURLOPT_UPLOAD_BUFFERSIZE to the same value (For download CURLOPT_MAX_RECV_SPEED_LARGE and CURLOPT_BUFFERSIZE). As I am no network expert, I do not know if this is intended behaviour for buffer to:
Ask the server to fill the buffer, then wait buffer/speed seconds and ask for the next buffer
This seems to be supported when I run the curl binary "Current Speed" stay at 0, for a long time, the jumps up for some few seconds, then back to 0