SFTP: downloading an empty file on a reused connection results in CURLE_TOO_MANY_REDIRECTS

[blach]

I did this

In my app, I'm using libcurl for getting directory listings and downloading files using SFTP.

When attempting to download an empty file after listing a directory (so the connection is reused), I get the error code CURLE_TOO_MANY_REDIRECTS.

I can reproduce the issue with command line curl using the "--next" parameter to issue two commands to the same server:

In the home directory on my server, I have created the folder "empty_file_test". This contains the 0 byte file "empty.txt".

mkdir empty_file_test
cd empty_file_test
touch empty.txt

On the client, I then issue this curl command (redacted):

curl -vvv sftp://server/~/empty_file_test/ --next sftp://server/~/empty_file_test/empty.txt --output empty.txt

Here's the redacted log file:

== Info: [READ] client_reset, clear readers
== Info: Host [redacted] was resolved.
== Info: IPv6: (none)
== Info: IPv4: [redacted]
== Info: [SETUP] added
== Info:   Trying [redacted]...
== Info: [SETUP] Curl_conn_connect(block=0) -> 0, done=0
== Info: [SETUP] Curl_conn_connect(block=0) -> 0, done=0
== Info: [SETUP] Curl_conn_connect(block=0) -> 0, done=0
== Info: [SETUP] Curl_conn_connect(block=0) -> 0, done=0
== Info: [SETUP] Curl_conn_connect(block=0) -> 0, done=0
== Info: [SETUP] Curl_conn_connect(block=0) -> 0, done=0
== Info: [SETUP] Curl_conn_connect(block=0) -> 0, done=0
== Info: [SETUP] Curl_conn_connect(block=0) -> 0, done=1
== Info: Established connection to [redacted] ([redacted] port 22) from [redacted] port 55451 
== Info: libssh2 cryptography backend: openssl compatible
== Info: User: '[redacted]'
== Info: Initialized password authentication
== Info: Authentication complete
<= Recv data, 66 bytes (0x42)
0000: -rw-r--r--    1 [redacted]   [redacted]          0 Oct 20 17:33 empty.tx
0040: t.
-rw-r--r--    1 [redacted]   [redacted]          0 Oct 20 17:33 empty.txt
== Info: [WRITE] [OUT] wrote 66 body bytes -> 66
== Info: [WRITE] [PAUSE] writing 66/66 bytes of type 1 -> 0
== Info: [WRITE] download_write body(type=1, blen=66) -> 0
== Info: [WRITE] client_write(type=1, len=66) -> 0
<= Recv data, 59 bytes (0x3b)
0000: drwx------   11 [redacted]   [redacted]       4096 Oct 20 16:59 ...
drwx------   11 [redacted]   [redacted]       4096 Oct 20 16:59 ..
== Info: [WRITE] [OUT] wrote 59 body bytes -> 59
== Info: [WRITE] [PAUSE] writing 59/59 bytes of type 1 -> 0
== Info: [WRITE] download_write body(type=1, blen=59) -> 0
== Info: [WRITE] client_write(type=1, len=59) -> 0
<= Recv data, 58 bytes (0x3a)
0000: drwxr-xr-x    2 [redacted]   [redacted]       4096 Oct 20 17:33 ..
drwxr-xr-x    2 [redacted]   [redacted]       4096 Oct 20 17:33 .
Info: [WRITE] [OUT] wrote 58 body bytes -> 58
Info: [WRITE] [PAUSE] writing 58/58 bytes of type 1 -> 0
Info: [WRITE] download_write body(type=1, blen=58) -> 0
Info: [WRITE] client_write(type=1, len=58) -> 0
Info: [WRITE] [OUT] done
Info: [READ] client_reset, clear readers
Info: Connection #0 to host [redacted]:22 left intact
Info: [READ] client_reset, clear readers
Info: Reusing existing SFTP: connection with host [redacted]
Info: abort upload
Recv data, 0 bytes (0x0)
Info: [WRITE] [PAUSE] writing 0/0 bytes of type 81 -> 0
Info: [WRITE] download_write body(type=81, blen=0) -> 0
Info: [WRITE] client_write(type=81, len=0) -> 0
Info: [WRITE] xfer_write_resp(len=0, eos=1) -> 0
Info: Connection died, retrying a fresh connect (retry count: 1)
Info: [WRITE] [OUT] done
Info: [READ] client_reset, will rewind reader
Info: shutting down connection #0
curl: (47) Number of redirects hit maximum amount

As you can see, this ends with "curl: (47) Number of redirects hit maximum amount".

This does not happen when just downloading an empty file without listing the directory first. I guess it only happens when a connection is reused.

I expected the following

There should be no error.

curl/libcurl version

curl 8.16.0 (aarch64-apple-darwin24.4.0) libcurl/8.16.0 OpenSSL/3.6.0 zlib/1.2.12 brotli/1.1.0 zstd/1.5.7 AppleIDN libssh2/1.11.1 nghttp2/1.67.1 ngtcp2/1.17.0 nghttp3/1.12.0 librtmp/2.3
Release-Date: 2025-09-10
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

operating system

Darwin Mac-Studio-von-Alexander.local 24.6.0 Darwin Kernel Version 24.6.0: Mon Aug 11 21:16:34 PDT 2025; root:xnu-11417.140.69.701.11~1/RELEASE_ARM64_T6020 arm64

[blach]

I assume this line is where the problem happens:

Info: Connection died, retrying a fresh connect (retry count: 1)

The connection did not actually die, the file is just empty, so 0 bytes are received from the server.

[blach]

I think in this case (downloading an empty file and connection reused) there is a problem with the logic in Curl_retry_request here:

curl/lib/transfer.c

Lines 665 to 678 in dbff3ee

	if((data->req.bytecount + data->req.headerbytecount == 0) &&
	conn->bits.reuse &&
	(!data->req.no_body || (conn->handler->protocol & PROTO_FAMILY_HTTP))
	#ifndef CURL_DISABLE_RTSP
	&& (data->set.rtspreq != RTSPREQ_RECEIVE)
	#endif
	)
	/* We got no data, we attempted to reuse a connection. For HTTP this
	can be a retry so we try again regardless if we expected a body.
	For other protocols we only try again only if we expected a body.
	This might happen if the connection was left alive when we were
	done using it before, but that was closed when we wanted to read from
	it again. Bad luck. Retry the same request on a fresh connect! */

(I don't know enough about curl internals to suggest a fix though.)

[icing]

Thanks for the report. This looks indeed like some quirky combination of reuse and 0 length that thinks the transfer failed. Investigating...

[icing]

I propose #19189 as a fix for this. I was able to add a test case to reproduce. Would be nice if you could verify. Thanks!

[blach]

Thanks! I'll try to verify later today.

[blach]

I compiled libcurl with the changes from your pull request and I can confirm that this solves the issue for me.

Here is what is now logged by my app with those changes:

20251022 15:41:29.623 == Info: Connection #0 to host [redacted]:22 left intact
20251022 15:41:29.624 == Successfully retrieved file listing with 1 entries.
20251022 15:41:36.137 == Downloading file from /empty_file_test/empty.txt
20251022 15:41:36.137 == Info: Reusing existing SFTP: connection with host [redacted]
20251022 15:41:36.162 == Info: abort upload
20251022 15:41:36.162 <= Recv data, 0000000000 bytes (0x00000000)
20251022 15:41:36.167 == Info: Connection #0 to host [redacted]:22 left intact
20251022 15:41:36.167 == Successfully downloaded file!

(The only strange thing that I still see in the log is "== Info: abort upload" - not sure why this is logged when downloading a file.)

[icing]

abort upload

I amended the PR to handle this situation correctly. Thanks for pointing this out.

[blach]

I've tested your latest PR changes with the curl command line tool and with libcurl in my app and can confirm that the "abort upload" message is gone.

Looking good!