Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CURLOPT_MAX_RECV_SPEED_LARGE < CURLOPT_BUFFERSIZE breaks SFTP download #13695

Closed
alervd opened this issue May 18, 2024 · 10 comments
Closed

CURLOPT_MAX_RECV_SPEED_LARGE < CURLOPT_BUFFERSIZE breaks SFTP download #13695

alervd opened this issue May 18, 2024 · 10 comments
Labels

Comments

@alervd
Copy link

alervd commented May 18, 2024

I did this

  1. Create and setup an easy handle for SFTP download.
  2. Call this: curl_easy_setopt(easy, CURLOPT_MAX_RECV_SPEED_LARGE, (curl_off_t)1000).
  3. Create multi handle and add easy handle to it.
  4. Start download. Result: it downloads first 1000 bytes and stalls. No further downloading occurs. Subsequent calls to turn the speed limit off has no effect also (curl_easy_setopt(easy, CURLOPT_MAX_RECV_SPEED_LARGE, (curl_off_t)0)).

I expected the following

I expect it to download with the speed limit I set with no limitations and to be able to change this limit at any time.

curl/libcurl version

libcurl 8.7.1.

operating system

Windows 10 22H2.

@alervd alervd changed the title CURLOPT_MAX_RECV_SPEED_LARGE < CURL_MAX_WRITE_SIZE breaks download CURLOPT_MAX_RECV_SPEED_LARGE < 7623 breaks download May 18, 2024
@bagder
Copy link
Member

bagder commented May 18, 2024

I cannot reproduce using the curl tool on Linux. Tested 8.7.1 and git master:

curl --limit-rate 1000 https://curl.se/

@alervd
Copy link
Author

alervd commented May 18, 2024

@bagder Well, it seems it's protocol specific. I can't reproduce it for HTTP protocol too. For me, it's reproducible for SFTP protocol.

@alexdconf
Copy link

Are you using your curl command in Powershell, CMD prompt, WSL -- on your Win10 install -- can you explicitly tell us which?
If you are not using WSL, can you post the output of the command you use to check the version of curl you are using?

On Windows systems, sometimes commands such as curl are aliased and actually implemented, not by the open source project, but some MS implementation -- just want to check that.

@alervd
Copy link
Author

alervd commented May 18, 2024

@alexdconf I'm using libcurl in my own code and have this issue. However, I've just tried to test it with curl from official downloads and it has the same issue for SFTP link. I launched it from CMD, however I'm pretty sure it does not matter at all (cmd, powershell, etc.).

@alervd alervd changed the title CURLOPT_MAX_RECV_SPEED_LARGE < 7623 breaks download CURLOPT_MAX_RECV_SPEED_LARGE < 7623 breaks SFTP download May 18, 2024
@alexdconf
Copy link

@alervd Apologies, I missed that you were working with the library directly and not in a CLI environment — since you cited Windows, I know that curl is aliased in those systems sometimes so I wanted to be sure you were leveraging the actual curl.

I think I managed to reproduce the problem you have.

I’m on a M1 Mac so I used a Docker container with a fedora:latest base image to reproduce this. I also spun up a sftp container to test this containing the file (just some file I had) BayesMask1.png in the /home/ directory.

The test script in the Fedora environment:

#!/bin/bash
curl -V
curl --limit-rate 1000 --max-time 60 -O -k -u "sftp:blank" sftp://10.0.0.5/~/BayesMask1.png
echo "Done."

The --max-time 60 is there because without it, the curl command hung for over fifteen minutes and I wanted to get logs. Without the --limit-rate 1000 option, the file downloads properly, fairly immediately.

The output from the script in the Fedora environment:

2024-05-18 11:46:03 curl 8.6.0 (aarch64-redhat-linux-gnu) libcurl/8.6.0 OpenSSL/3.2.1 zlib/1.3.0.zlib-ng brotli/1.1.0 libidn2/2.3.7 libpsl/0.21.5 libssh/0.10.6/openssl/zlib nghttp2/1.59.0 OpenLDAP/2.6.7
2024-05-18 11:46:03 Release-Date: 2024-01-31
2024-05-18 11:46:03 Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp ws wss
2024-05-18 11:46:03 Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets
2024-05-18 11:46:03   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2024-05-18 11:46:03                                  Dload  Upload   Total   Spent    Left  Speed
  3 25806    3  1000    0     0     16      0  0:26:52  0:00:59  0:25:53     0
  3 25806    3  1000    0     0     16      0  0:26:52  0:00:59  0:25:53    16
2024-05-18 11:47:03 curl: (28) Operation timed out after 59999 milliseconds with 1000 out of 25806 bytes received
2024-05-18 11:47:03 Done.

@bagder
Copy link
Member

bagder commented May 18, 2024

it's reproducible for SFTP protocol

What SSH library and version are you using? (curl -V output is always a good idea...)

@alervd
Copy link
Author

alervd commented May 19, 2024

@bagder

curl 8.7.1 (x86_64-w64-mingw32) libcurl/8.7.1 LibreSSL/3.9.1 zlib/1.3.1 brotli/1.1.0 zstd/1.5.6 WinIDN libpsl/0.21.5 libssh2/1.11.0 nghttp2/1.61.0 ngtcp2/1.4.0 nghttp3/1.2.0
Release-Date: 2024-03-27
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM PSL SPNEGO SSL SSPI threadsafe UnixSockets zstd

@alervd alervd changed the title CURLOPT_MAX_RECV_SPEED_LARGE < 7623 breaks SFTP download CURLOPT_MAX_RECV_SPEED_LARGE < CURLOPT_BUFFERSIZE breaks SFTP download May 29, 2024
@alervd
Copy link
Author

alervd commented May 29, 2024

Any ETA on this issue fix? I've found that libcurl is too slow downloading from SFTP in case CURLOPT_BUFFERSIZE is set to default 16K. Now I have to increase it to 256K and found that my minimum speed limit is increased to this 256K also... :)

@alervd
Copy link
Author

alervd commented May 29, 2024

For now, it seems there is a workaround (return CURL_WRITEFUNC_PAUSE in write func; didn't try it btw). But I would prefer to wait for a fix in case it appears in some reasonable time.

@bagder
Copy link
Member

bagder commented May 29, 2024

Any ETA on this issue fix?

I have not seen anyone working on this. There is no estimate.

bagder added a commit that referenced this issue Jun 13, 2024
When aborting the transfer loop early, like when there is rate limiting
in effect, there might be buffered data already read off the socket so
the socket might not signal reability. Therefore we must set the
CSELECT_IN manually if data_pending_() suggests there might be more data
to get. This is particularly noticeable with SSH when the underlying
library has drained the socket and holds pending data in its buffer.

Reported-by: alervd on github
Fixes #13695
@bagder bagder closed this as completed in cb5c703 Jun 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Development

Successfully merging a pull request may close this issue.

3 participants