-T . / --upload-file . non-blocking STDIN/STDOUT does not work on Windows

[denandz]

I did this

curl.exe -T . -N https://pipeto.me/oincqoinlqher
Warning: fcntl failed on fd=0: No such file or directory
^C

Same issue with other protocols using -T .. Curl will write any input that comes in via STDIN out to the server, but will not print any server responses to STDOUT until the very end when curl.exe receives a ctrl-c.

I expected the following

Expected non-blocking IO, allowing server read and write at the same time, as per Linux.

curl/libcurl version

curl.exe --version
curl 8.9.1 (Windows) libcurl/8.9.1 Schannel zlib/1.3 WinIDN
Release-Date: 2024-07-31
Protocols: dict file ftp ftps http https imap imaps ipfs ipns mqtt pop3 pop3s smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS HSTS HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM SPNEGO SSL SSPI threadsafe Unicode UnixSockets

operating system

Windows 10/Windows 11

[bagder]

I have the feeling this is not easily done on Windows...

[denandz]

Likely so, I think it's more involved than just an fnctl call. Should be do-able though, netcat and such have the expected behaviour under windows:

nc64.exe -vv -n 192.168.122.1 8000
(UNKNOWN) [192.168.122.1] 8000 (?) open
client -> server
serv <- client

Will do some more digging.

[jay]

I don't know if this is related but on Windows there is no line buffering in the C runtime. Although Windows offers a line buffering flag for compatibility purposes it does not actually do anything, which is misleading. It's the same as full buffering. We disabled all buffering in Windows for the SSL key log file so that other applications would have access to that data immediately because otherwise it didn't work right, the data was delayed.

curl/lib/vtls/keylog.c

Lines 53 to 57 in 1c31498

	#ifdef _WIN32
	if(setvbuf(keylog_file_fp, NULL, _IONBF, 0))
	#else
	if(setvbuf(keylog_file_fp, NULL, _IOLBF, 4096))
	#endif

In contrast here, when -N (nobuffer) is used then curl will call fflush which should work on Windows, but maybe there is some underlying layer that is not flushing?

curl/src/tool_cb_wrt.c

Lines 365 to 370 in 1c31498

	if(config->nobuffer) {
	/* output buffering disabled */
	int res = fflush(outs->stream);
	if(res)
	return CURL_WRITEFUNC_ERROR;
	}

[denandz]

After some more digging, it looks like windows netcat options use kbhit() which likely isn't going to work for us. This might work when stdin is a console, but the second we pipe in from another program it'll go back to the blocking read behaviour. There are other options like WaitForMultipleObjects() or WaitForSingleObject() which can operate on console IO, but that has it's own problems and might need filtering for mouse events. Not sure how this will handle piped data.

nmap have solved this issue in ncat by spawning a seperate thread to read from stdin and write data out to a socket which is then read non-blocking styles by the rest of the code - https://github.com/nmap/nmap/blob/master/nbase/nbase_winunix.c#L164

Could we potentially spawn a thread in check_stdin_upload https://github.com/curl/curl/blob/master/src/tool_operate.c#L1069 if we're on Windows with -T . and replace per->infd with the other end of the socket or pipe ?

[bagder]

Should be do-able though

Of course it is doable, I don't think anyone would suggest anything else. The question is how complicated it is to get it into the existing architecture and APIs since the Windows way of doing things is completely different than the rest of the world.

Could we potentially spawn a thread in check_stdin_upload

That sounds like a rather clever approach! If we can get a "proper" file descriptor for libcurl to wait on, it solves a lot of problems.

[denandz]

I've attempted to get this working and stumbled across a few issues. One of which is Winsock sockets, also, are not "proper" file descriptors and read() won't work, they need a recv() call on Windows instead. I can keep chipping away at that.

Another issue has arisen though. The --upload-file . functionality doesn't work at all on Curl 8.14 (testing under Linux). The input is never read from STDIN. I did some more digging and reading from STDIN with --upload-file . worked correctly under 8.7.1, reading from STDIN became slow from curl 8.8 onwards, and stopped working completely in 8.14. Since this is unrelated to the Windows-specific behavior, I'll go ahead and get some better evidence of this behavior and open another issue.