First buffer upload seem to ignore the send speed

[Togtja]

I did this

When uploading files to an SFTP server, I limited the CURLOPT_MAX_SEND_SPEED_LARGE to 1024, and set the CURLOPT_UPLOAD_BUFFERSIZE to it's minimum of 16kb. When I upload a file of 25855 bytes it finished the first 16384 bytes nearly instantly, then takes some time for the rest. It spends totally 16 seconds uploading something that should take 25855/1024 = 25 seconds give or take. Leaving the CURLOPT_UPLOAD_BUFFERSIZE to the default 64kb makes the entire upload happen instantly.

I also noticed something similar for download as well, if I set CURLOPT_BUFFERSIZE to say 131072 bytes (CURL_MAX_READ_SIZE/4), the entire download happens instantly. If I lowered CURLOPT_BUFFERSIZE to the same as CURLOPT_MAX_RECV_SPEED_LARGE it would take the expected time to download.

I expected the following

I expect that even if the buffer size is larger than the max speed, the max speed would be upheld.

curl/libcurl version

curl 7.71.1 (Windows) libcurl/7.71.1 OpenSSL/1.1.1g libssh2/1.9.0_DEV
Release-Date: 2020-07-01
Protocols: dict file ftp ftps gopher http https imap imaps ldap pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS HTTPS-proxy IPv6 Largefile NTLM SSL

operating system

Windows 10 Enterprise

[bagder]

Clearly, when libcurl's been instructed to do the transfer slower (in bytes/sec) than the size of the upload/download buffer, it should rather limit the amount data to handle to the capped amount in order to better acknowledge the setting in the first network read/write.

I think this is just an oversight as it only affects the first network operation and capping network speed has always been about being "roughly right over time", and not an exact science.

Are you interested in writing a patch for this?

[Togtja]

I am willing to give it a shot, if I can get pointed to the right direction in the source code.

[bagder]

Curl_fillreadbuffer reads data into the upload buffer

The primary point in the code that reads data off the network in a download state is when readwrite_data() calls Curl_read.

[Togtja]

I played around with it today, and as far as I can tell, CURL, calls Curl_fillreadbuffer and fills the buffer, then wait for an appropriate about of time so that the limit-rate get correctly averaged, which I suppose work fine in household networks etc.
However, I am dealing with a very limited pipe, of something like 20kb/s on bad days lower, where a 64kb filled buffer stop all communication for 3 seconds. As alluded to in my issue, the temporary fix for this is to set CURLOPT_MAX_SEND_SPEED_LARGE and CURLOPT_UPLOAD_BUFFERSIZE to the same value (For download CURLOPT_MAX_RECV_SPEED_LARGE and CURLOPT_BUFFERSIZE). As I am no network expert, I do not know if this is intended behaviour for buffer to:
Ask the server to fill the buffer, then wait buffer/speed seconds and ask for the next buffer

This seems to be supported when I run the curl binary "Current Speed" stay at 0, for a long time, the jumps up for some few seconds, then back to 0

[bagder]

Yes that's how it currently works which has the effect on the first recv/send you described. Limiting the buffer sizes is of course a mitigating method.